This note was first taken when I learnt the machine learning course on Coursera.
Lectures in this week: Lecture 6, Lecture 7.

Classification & Representation

Classification

• Variable $y$ has discrete values.

• Other name of : 1 (positive class), 0 (negative class) $\Rightarrow$ binary classification problem

• If y has more than 2 values, it’s called multi classification

• Using linear regression in this case seems not to be very good because there may be some values that effects much more than the others (blue line).

  

• $h_{\theta}$ may take values >1 or <0 but we want $0\le h_{\theta} \le 1$. That’s why we need logistic regression, i.e. $h_{\theta}$ is always between $[0,1]$

• Remember and not confused that logistic regression is just a classification regression in cases of y taking discrete values.

Hypothesis representation

• What is the function we are going to use to represent the hypothesis

• Logistic regression

• They are the same: sigmoid function = logistic function = $g(z)$

• Some propabilities

  $$ \begin{align*}& h_\theta(x) = P(y=1 | x ; \theta) = 1 - P(y=0 | x ; \theta) \newline& P(y = 0 | x;\theta) + P(y = 1 | x ; \theta) = 1\end{align*} $$

Decision Boundary

From the above figure, we see that

We have

The decision boundary is the line that separates the area where y = 0 and where y = 1mark>. It is created by our hypothesis function.

An example,

The training set is not used to determine the decision boundary, but parameter $\theta$. The training set is used only for fit the parameter $\theta$.

Logistic regression model

Cost function

or we can write,

The cost function is rewritten as

Vectorization

Simplified Cost Function and Gradient Descent

In this logistic regression,

Notice that, above equation looks the same with one in linear regression, the different is def of $h_{\theta}$!

Vectorization

Advanced Optimization

“Conjugate gradient”, “BFGS”, and “L-BFGS” are more sophisticated, faster ways to optimize $\theta$ that can be used instead of gradient descent. We suggest that you should not write these more sophisticated algorithms yourself (unless you are an expert in numerical computing) but use the libraries instead, as they’re already tested and highly optimized. Octave/Matlab provides them.

A single function that returns both $J(\theta)$ and $\frac{\partial}{\partial\theta_j}J(\theta)$

function [jVal, gradient] = costFunction(theta)
  jVal = [...code to compute J(theta)...];
  gradient = [...code to compute derivative of J(theta)...];
end

Then we can use octave’s fminunc() optimization algorithm along with the optimset() function that creates an object containing the options we want to send to fminunc().

options = optimset('GradObj', 'on', 'MaxIter', 100);
initialTheta = zeros(2,1);
[optTheta, functionVal, exitFlag] = fminunc(@costFunction, initialTheta, options);

We give to the function fminunc() our cost function, our initial vector of theta values, and the options object that we created beforehand.

info fmincg works similarly to fminunc, but is more more efficient for dealing with a large number of parameters.

Multiclass classification: one-vs-all

$y$ has more values than only two 0 and 1. We keep using binary classification for each group of 2 (consider one and see the others as the other group)

• face Why max?

  We want to choose a $\Theta$ such that for all $j\in \{ 0,\ldots,n \}$,

  $$h_{\theta}^{(j)} = P(y=j\vert x;\theta) \ge 0.5$$

  Don’t forget that, we consider $h_{\theta} \ge 0.5$ as true. Because of that, there is onlty 1 option, that’s max of all $j$.

info See ex 4 for an example in practice.

Solving the problem of overfitting

The problem of overfitting

We have many features, $h_{\theta}$ may fit the training set very well ($J(\theta) \simeq 0$) but fail to generalize.

Cost function

Options to solve:

• Reduce the number of features
  • Manually select which features to keep
  • By algorithm
• Regularization (add penalty terms)
  • Keep all features but reduce magnitude/values of parameters $\theta_j$
  • Works well when we have a lot of features, each of which contributes a bit to predicting $y$

Because we need to find the minimum, we multiply $\theta_3, \theta_4$ by 1000 to make them very big and never be a min, i.e. they look like 0.

If $\lambda$ is too large, the problem of underfitting occurs!

Regularized linear regression

Gradient Descent

Intuitively, reduce $\theta_j$ by some amount on every update, the second term is exactly the same it was before.

Normal equation

Regularized logistic regression

We can regularize this equation by adding a term to the end:

And the gradient descent

The same form with GD regularized linear regression, the difference in this case is only the definition of $h_{\theta}(x)$

Exercice de programmation: Logistic Regression

Logistic Regression

• plotData: Plot from X, y to separate two kind of X

  XPos = X(y==1, :);
  XNeg = X(y==0, :);
  plot(XPos(:,1), XPos(:,2), 'k+', 'LineWidth', 2, 'MarkerSize', 7);
  plot(XNeg(:,1), XNeg(:,2), 'ko', 'MarkerFaceColor', 'y', 'MarkerSize', 7);
  

• sigmoid.m: recall that

  $$\begin{align} h_{\theta}(x) &= g(\theta^Tx) \\ g(z) &= \dfrac{1}{1+e^{-z}} \end{align}$$

  g = 1 ./ (1 + exp(-z));
  

• costFunction.m: recall that, the cost function in logistic regression is

  $$J(\theta) = - \frac{1}{m} \displaystyle \sum_{i=1}^m [y^{(i)}\log (h_\theta (x^{(i)})) + (1 - y^{(i)})\log (1 - h_\theta(x^{(i)}))]$$

  or in vectorization,

  $$\begin{align*} h &= g(X\theta) \\ J(\theta) &= \frac{1}{m} \left(-y^{T}\log(h)-(1-y)^{T}\log(1-h)\right) \end{align*}$$

  and its gradient is

  $$\dfrac{\partial(\theta)}{\partial\theta_j} = \dfrac{1}{m} \sum_{i=1}^m (h_{\theta}(x^{(i)}) - y^{(i)}) x_j^{(i)}, \text{ for } j=0,\ldots,n.$$

  or in vectorization,

  $$ \nabla \theta = \dfrac{1}{m} X^T(g(X\theta) - y). $$

  h = sigmoid(X*theta); % hypothesis
  J = 1/m * ( -y' * log(h) - (1-y)' * log(1-h) );
  grad = 1/m * X' * ( h - y);
  

• fminunc:

  • GradObj option to on, which tells fminunc that our function returns both the cost and the gradient

      options = optimset('GradObj', 'on', 'MaxIter', 400);
    

  • Notice that by using fminunc, you did not have to write any loops yourself, or set a learning rate like you did for gradient descent.

• predict.m: remember that,

  $$\begin{align} h_{\theta}(x) &= g(X\theta), \\ h_{\theta}(x) &\ge 0.5 \to y = 1\\ h_{\theta}(x) &< 0.5 \to y = 0 \end{align}$$

  h = sigmoid(X*theta); % m x 1
  p = (h >= 0.5);
  

Regularized logistic regression

costFunctionReg.m: recall that,

its gradient,

h = sigmoid(X*theta); % hypothesis
J = 1/m * ( -y' * log(h) - (1-y)' * log(1-h) ) + lambda/(2*m) * sum(theta(2:end).^2);

grad(1,1) = 1/m * X(:,1)' * (h-y);
grad(2:end,1) = 1/m * X(:,2:end)' * (h-y) + lambda/m * theta(2:end,1);