Logistic Regression

1 min read

# topic/ai-ml/classical

Logistic regression is a linear classifier that models the probability of class membership:

P(y=1∣x)=σ(w⊤x+b)=11+e−(w⊤x+b)P(y=1|\mathbf{x}) = \sigma(\mathbf{w}^\top\mathbf{x} + b) = \frac{1}{1 + e^{-(\mathbf{w}^\top\mathbf{x} + b)}}

The sigmoid σ\sigma squashes the linear output to [0,1][0, 1].

Training: minimize the binary cross-entropy loss:

L=−1n∑i[yilog⁡y^i+(1−yi)log⁡(1−y^i)]L = -\frac{1}{n}\sum_i \left[y_i \log\hat{y}_i + (1-y_i)\log(1-\hat{y}_i)\right]

This is the negative log-likelihood of the Bernoulli distribution → MLE.

Key properties:

  • Loss is convex → unique global minimum (Convexity)
  • No closed-form solution — must use gradient descent
  • Decision boundary is linear: w⊤x+b=0\mathbf{w}^\top\mathbf{x} + b = 0 (a hyperplane)
  • Outputs calibrated probabilities (unlike SVMs)

Multiclass: replace sigmoid with softmax, cross-entropy loss over KK classes.

The building block of neural networks: a single neuron with sigmoid activation is exactly logistic regression. A neural network stacks these with nonlinearities to learn non-linear decision boundaries.

See also: Linear Regression, The Artificial Neuron, Loss Functions

Linked from