1 · The linear model and least squares
We observe $n$ trips. Trip $i$ has features $x_i \in \mathbb{R}^p$ (peak-hour flag, rain flag, traffic speed, drivers available) and an outcome $y_i \in \mathbb{R}$, the surge charged in rupees. We posit
$$ y_i = x_i^\top w + \varepsilon_i, \qquad i = 1,\dots,n. $$Stack the rows into the design matrix $X \in \mathbb{R}^{n \times p}$ and the outcomes into $y \in \mathbb{R}^n$, so that $y = Xw + \varepsilon$. Least squares chooses $w$ to minimise the residual sum of squares
$$ \mathrm{RSS}(w) = \lVert y - Xw \rVert_2^2 = \sum_{i=1}^n \left(y_i - x_i^\top w\right)^2 .$$The Hessian $2X^\top X \succeq 0$, so this is a global minimum. Geometrically $X\hat w = X(X^\top X)^{-1}X^\top y$ is the orthogonal projection of $y$ onto the column space of $X$, and the residual is perpendicular to every column.
1.1 · Where it breaks: collinearity
Everything above hinges on inverting $X^\top X$. Write the singular value decomposition $X = U D V^\top$ with $D = \mathrm{diag}(d_1 \ge \dots \ge d_p)$. Then
$$ \hat w_{\text{OLS}} = V D^{-1} U^\top y, \qquad \operatorname{Var}(\hat w_{\text{OLS}}) = \sigma^2 (X^\top X)^{-1} = \sigma^2 V D^{-2} V^\top .$$A small singular value $d_j$ enters as $d_j^{-2}$. In the Mumbai data is_peak,
is_rain and traffic_speed move together: rain is likelier in the evening
peak, and both slow traffic. Their pairwise correlations run from $-0.75$ to $0.66$ (see the
correlation matrix). $X^\top X$ is then nearly singular, and least
squares can shift weight between the rain and peak coefficients with almost no change in fit.
Fit OLS on is_rain and traffic_speed using only the first six trips.
Those two columns are correlated at $-0.94$ there, and you get
$\hat w = (10.8,\, -34.2)$. Add one more trip and it becomes $(21.7,\, -23.8)$ — a swing of
₹11 per standard deviation from a single observation. This is not a bug in your code; it is
$D^{-2}$ doing exactly what the algebra above says it will.
Worse is available. Take is_rain and is_bad_weather over the first ten
trips: in that stretch every rainy trip also had traffic below 25 km/h, so the two columns are
identical, $\det(X^\top X) = 0$, and the maximum likelihood estimate does not exist at
all. Not "is unstable" — does not exist. Ridge, as we are about to see, returns a perfectly
sensible answer on the same data.
is_bad_weather is added to the eight base features.2 · Maximum likelihood
So far least squares was an aesthetic choice. Make it a statistical one: assume the noise is independent Gaussian, $\varepsilon_i \sim \mathcal{N}(0, \sigma^2)$. Then
$$ p(y \mid X, w, \sigma^2) = \prod_{i=1}^{n} \frac{1}{\sqrt{2\pi\sigma^2}} \exp\!\left(-\frac{(y_i - x_i^\top w)^2}{2\sigma^2}\right). $$Take logs — the likelihood is a product, and products are miserable to differentiate:
$$ \ell(w,\sigma^2) = -\frac{n}{2}\log(2\pi\sigma^2) \;-\; \frac{1}{2\sigma^2}\sum_{i=1}^n (y_i - x_i^\top w)^2 . $$Only the last term involves $w$, and it is $-\mathrm{RSS}(w)/2\sigma^2$. Maximising over $w$ is therefore identical to minimising RSS:
$$ \hat w_{\text{MLE}} = \argmin_w \; \lVert y - Xw\rVert_2^2 = \hat w_{\text{OLS}}. $$Least squares is not an alternative to maximum likelihood; under Gaussian noise it is maximum likelihood. Change the noise model and the loss changes with it. Laplace noise gives least absolute deviations; Bernoulli outcomes give cross-entropy.
Differentiating $\ell$ with respect to $\sigma^2$ and setting to zero gives $\hat\sigma^2_{\text{MLE}} = \mathrm{RSS}(\hat w)/n$, which is biased downward; the unbiased estimator divides by $n - p$ instead. In the assignment we simply fix $\sigma = 10$ ₹, because the data generating process is known.
MLE is consistent, asymptotically efficient and invariant to reparameterisation. Its weakness matters here: when $p$ approaches $n$, or columns are collinear, maximising fit to the data in hand is the wrong instruction.
3 · Bayes, and the MAP estimator
Treat $w$ as a random variable with a prior $p(w)$ encoding what you believed before seeing data. Bayes' theorem gives the posterior
$$ p(w \mid y, X) = \frac{p(y \mid X, w)\,p(w)}{p(y\mid X)} \;\propto\; p(y\mid X,w)\,p(w). $$The maximum a posteriori estimate is the posterior mode. Taking logs turns the product into a sum, and the normalising constant drops out because it does not depend on $w$:
$$ \hat w_{\text{MAP}} = \argmax_w \Big[\log p(y\mid X,w) + \log p(w)\Big] = \argmin_w \Big[\underbrace{\lVert y - Xw\rVert_2^2}_{\text{data fit}} \;-\; \underbrace{2\sigma^2 \log p(w)}_{\text{penalty}}\Big]. $$$-\log(\text{prior})$ is the penalty. Choose a prior, take its negative log, and you have a regularizer. Every method below follows from this equation; only the density on $w$ changes.
Walkthrough 2 runs this construction step by step: it starts from the likelihood, multiplies in a prior, negates, and watches $\lambda = \sigma^2/\tau^2$ appear. It also lets you dial $\tau$ and see the estimate travel from the MLE to zero.
4 · Ridge regression — the Gaussian prior
Put an isotropic Gaussian prior on the coefficients, $w_j \sim \mathcal{N}(0,\tau^2)$ independently. Then
$$ \log p(w) = -\frac{1}{2\tau^2}\sum_{j=1}^p w_j^2 + \text{const}, $$and substituting into the MAP objective:
$$ \hat w_{\text{ridge}} = \argmin_w \; \lVert y - Xw\rVert_2^2 + \lambda \lVert w \rVert_2^2 , \qquad \boxed{\lambda = \frac{\sigma^2}{\tau^2}} $$The penalty weight is the ratio of noise variance to prior variance, not a free parameter. A confident prior (small $\tau^2$) or noisy data (large $\sigma^2$) both increase shrinkage. In the assignment $\sigma = 10$ and $\tau = 1$, so $\lambda = 100$.
Since $X^\top X \succeq 0$, adding $\lambda I$ with $\lambda > 0$ raises every eigenvalue to at least $\lambda$, so the inverse always exists, even when $p > n$ and OLS is undefined.
4.1 · What shrinkage does, in the SVD basis
Using $X = UDV^\top$, the fitted values become
$$ X\hat w_{\text{ridge}} = \sum_{j=1}^{p} u_j \,\frac{d_j^2}{d_j^2 + \lambda}\, u_j^\top y .$$Each principal direction is multiplied by $d_j^2/(d_j^2+\lambda) \in (0,1)$. Directions with large $d_j$, where the data genuinely varies, are barely touched; directions with tiny $d_j$, the ill-conditioned combinations, are crushed. Ridge shrinks the unreliable directions hardest.
4.2 · Why correlated features share weight
Suppose two columns are identical, $x_1 = x_2$, and the best total effect is $c$. Any split $(a, c-a)$ fits equally well, so OLS is indifferent — the solution is not unique. Ridge breaks the tie by minimising $a^2 + (c-a)^2$, which is smallest at $a = c/2$. Equal split.
This is what Task 2 shows: at $\lambda = 0$ the peak and rain coefficients differ sharply, and as $\lambda$ grows they converge. Ridge attributes a little to each rather than a lot to one.
Two adjacent road segments in a travel-time model are nearly the same feature. Attributing all the delay to one produces misleading segment-level diagnostics and a different answer after the next retrain. Splitting it is more stable.
5 · Lasso — the Laplace prior
Now use a Laplace (double-exponential) prior, $p(w_j) = \frac{1}{2b}\exp(-|w_j|/b)$:
$$ \log p(w) = -\frac{1}{b}\sum_{j=1}^p |w_j| + \text{const} \quad\Longrightarrow\quad \hat w_{\text{lasso}} = \argmin_w \lVert y-Xw\rVert_2^2 + \lambda\lVert w\rVert_1, \quad \lambda = \frac{2\sigma^2}{b}. $$The objective is convex but not differentiable at $w_j = 0$, so there is no closed form. That non-differentiability is the mechanism that produces exact zeros.
5.1 · Deriving the soft-threshold by hand
Take one coordinate at a time. Hold $w_k$ fixed for $k \neq j$ and define the partial residual $r^{(j)} = y - \sum_{k\neq j} x_k w_k$ and $\rho_j = x_j^\top r^{(j)}$. The objective in $w_j$ alone is
$$ f(w_j) = \lVert r^{(j)} - x_j w_j\rVert_2^2 + \lambda|w_j| = \text{const} - 2\rho_j w_j + \lVert x_j\rVert^2 w_j^2 + \lambda|w_j|. $$Cycling this update over $j = 1,\dots,p$ until convergence is coordinate descent:
about a dozen lines of code, and what scikit-learn runs.
5.2 · The geometry: why a diamond has corners
In constrained form, Lasso solves $\min \lVert y - Xw\rVert^2$ subject to $\lVert w\rVert_1 \le t$, and Ridge subject to $\lVert w\rVert_2^2 \le t$. The RSS contours are ellipses centred on $\hat w_{\text{OLS}}$. The solution is where the growing ellipse first touches the constraint set.
The $\ell_2$ ball is smooth, so first contact almost surely happens where all coordinates are non-zero. The $\ell_1$ ball is a diamond with vertices on the axes, and an expanding ellipse is far more likely to meet a corner than an edge. A corner means a coordinate is exactly zero: sparsity follows from the geometry.
5.3 · Three caveats worth stating out loud
- Among correlated features the choice is arbitrary. If
is_rainandtraffic_speedcarry the same information, Lasso keeps roughly one, and which one can change under resampling. The survivor is not evidence of a cause. - It saturates when $p > n$. Lasso selects at most $n$ variables, an awkward ceiling in genomics where $p$ is 20,000 and $n$ is 200.
- The estimates are biased. Soft-thresholding shrinks the survivors too. If you need honest effect sizes, refit OLS on the selected set (the "relaxed" or debiased Lasso).
6 · Elastic Net — both priors at once
Multiply the two priors together — Gaussian times Laplace — and the log turns the product into a sum of penalties:
$$ \hat w_{\text{EN}} = \argmin_w \;\lVert y - Xw\rVert_2^2 \;+\; \lambda_1\lVert w\rVert_1 \;+\; \lambda_2\lVert w\rVert_2^2 . $$The usual parameterisation writes $\lambda_1 = \alpha\rho$ and $\lambda_2 = \alpha(1-\rho)/2$: $\alpha$ sets the strength, and the mixing ratio $\rho \in [0,1]$ sets the character, with $\rho = 1$ pure Lasso and $\rho = 0$ pure Ridge.
The coordinate update inherits the soft-threshold, with the ridge term appearing in the denominator — one extra symbol relative to Lasso:
$$ \hat w_j \;\leftarrow\; \frac{S\!\left(\rho_j,\;\lambda_1\right)}{\lVert x_j\rVert^2 + \lambda_2}. $$6.1 · The grouping effect
Zou & Hastie proved a bound worth carrying around. For standardized inputs and two features with sample correlation $r$, the Elastic Net solution satisfies
$$ \frac{|\hat w_i - \hat w_j|}{\lVert y\rVert_1} \;\le\; \frac{\sqrt{2(1-r)}}{\lambda_2}. $$As $r \to 1$ the bound goes to zero, so perfectly correlated features receive identical coefficients. Lasso has no such guarantee. This is the formal version of "Lasso picks one, Elastic Net keeps the group".
You will build is_bad_weather = is_rain AND traffic_speed < 25, which is
correlated with both of its parents, then show that Lasso keeps roughly one of the three,
Ridge keeps all three with similar weight, and Elastic Net keeps the group while still zeroing
is_weekend. That last clause is the part Ridge cannot do.
6.2 · The double-shrinkage correction
Applying both penalties shrinks twice, once by $\ell_1$ and again by $\ell_2$, biasing the
naïve solution toward zero. Zou & Hastie rescale by $(1+\lambda_2)$ to undo the second
shrinkage; scikit-learn absorbs this into its parameterisation. If your coordinate
descent disagrees with ElasticNet in the third decimal, the objective is the usual
culprit.
7 · Past the point estimate
MAP returns the posterior mode and throws away the rest of the distribution. With a Gaussian prior and Gaussian likelihood the entire posterior is available in closed form:
$$ p(w \mid y, X) = \mathcal{N}(\mu, \Sigma), \qquad \Sigma = \left(\frac{X^\top X}{\sigma^2} + \frac{I}{\tau^2}\right)^{-1}, \qquad \mu = \frac{1}{\sigma^2}\,\Sigma\, X^\top y . $$The mean $\mu$ equals the Ridge solution at $\lambda = \sigma^2/\tau^2$, because a Gaussian's mode is its mean. Ridge is the posterior with the error bars discarded; the diagonal of $\Sigma$ returns them as $\hat w_j \pm 1.96\sqrt{\Sigma_{jj}}$.
For a new trip $x_\star$ the posterior predictive adds observation noise back:
$$ p(y_\star \mid x_\star, y) = \mathcal{N}\!\left(x_\star^\top \mu,\; \underbrace{x_\star^\top \Sigma x_\star}_{\text{uncertainty in } w} + \underbrace{\sigma^2}_{\text{irreducible}}\right). $$The first term shrinks as you collect data; the second does not. That is the difference between uncertainty about the rain coefficient and the inherent noise in surges, and the reason a quote of ₹50 ± ₹10 is more honest than ₹50.
8 · Logistic regression: a different likelihood, the same priors
Everything so far modelled a number. Now the outcome is binary — does this cell need a surge alert or not — so the Gaussian likelihood is the wrong model: it puts probability mass on $y = 1.4$ and $y = -0.3$, which do not exist.
Replace it with a Bernoulli likelihood, with the success probability written as a sigmoid of the same linear score:
$$ p_i = \sigma(x_i^\top w + b) = \frac{1}{1 + e^{-(x_i^\top w + b)}}, \qquad y_i \sim \text{Bernoulli}(p_i). $$The log-likelihood is then
$$ \ell(w,b) = \sum_{i=1}^{n}\Big[ y_i \log p_i + (1-y_i)\log(1-p_i) \Big], $$and maximising it is minimising the average log-loss, also called binary cross-entropy. Exactly as in §2, the loss was not chosen — it fell out of the noise model.
8.1 · MAP, unchanged
Now apply §3 verbatim. Multiply by a prior, take logs, negate:
$$ \hat w = \argmin_{w,b} \; \underbrace{\frac{1}{n}\sum_i -\big[y_i\log p_i + (1-y_i)\log(1-p_i)\big]}_{\text{log-loss}} \;+\; \underbrace{\alpha\rho\lVert w\rVert_1 + \tfrac{\alpha(1-\rho)}{2}\lVert w\rVert_2^2}_{\text{same penalty as before}} $$A Gaussian prior still gives an $\ell_2$ penalty and a Laplace prior still gives an $\ell_1$ penalty, because the penalty comes from $-\log p(w)$ and $p(w)$ never mentioned the likelihood. This is the dividend from deriving MAP once, generally, in Walkthrough 2 — swapping the likelihood costs you nothing.
The intercept $b$ is left unpenalized. Shrinking it would bias the predicted base rate toward one half, which is a statement about the data nobody intended to make.
8.2 · Two things that do change
There is no closed form. The gradient of the log-loss is
$$ \nabla_w = \frac{1}{n}X^\top(p - y) + \alpha(1-\rho)\,w, $$which is non-linear in $w$ through $p$, so there is no normal equation to solve. The standard answers are Newton's method — iteratively reweighted least squares, which solves a weighted ridge problem at each step — or, when there is an $\ell_1$ term, proximal gradient descent: a gradient step followed by the same soft-threshold from §5.1. The lab uses the latter.
The error measure changes. RMSE is meaningless for a 0/1 outcome. Use log-loss for fitting and model selection, AUC for a threshold-free summary, and report accuracy only alongside the base rate — on a 70/30 split, predicting "never" scores 70%.
It outputs a probability. Turning that into a yes/no needs a threshold, and the threshold is a statement about the relative cost of a missed alert and a false alarm — a policy decision, not a model output. Keep it in the policy layer, next to the price cap in the case study.
8.3 · And a warning about the label
Our binary target is manufactured: high_surge = surge > ₹100. That is a real
operational trigger, but dichotomising a continuous outcome discards information — a ₹101 trip and
a ₹150 trip become the same observation. Before building a classifier on a threshold of something
you can already model continuously, check whether predicting the number and thresholding the
prediction does just as well. The lab measures this,
and on this dataset the answer is uncomfortable.
Work through it in the logistic section of the lab.
9 · One page to revise from
| Estimator | Prior | Objective | Solution | Zeros? |
|---|---|---|---|---|
| OLS / MLE | flat | $\lVert y-Xw\rVert^2$ | $(X^\top X)^{-1}X^\top y$ | no |
| Ridge | $\mathcal{N}(0,\tau^2)$ | $\lVert y-Xw\rVert^2+\lambda\lVert w\rVert_2^2$ | $(X^\top X+\lambda I)^{-1}X^\top y$ | no |
| Lasso | $\mathrm{Laplace}(0,b)$ | $\lVert y-Xw\rVert^2+\lambda\lVert w\rVert_1$ | coordinate descent, $S(\rho,\lambda/2)/\lVert x_j\rVert^2$ | yes |
| Elastic Net | Gaussian × Laplace | $\lVert y-Xw\rVert^2+\lambda_1\lVert w\rVert_1+\lambda_2\lVert w\rVert_2^2$ | $S(\rho_j,\lambda_1)/(\lVert x_j\rVert^2+\lambda_2)$ | yes |
| Bayesian | $\mathcal{N}(0,\tau^2)$ | — (full posterior) | $\mathcal{N}(\mu,\Sigma)$, $\mu = \hat w_{\text{ridge}}$ | no, but intervals |
| Logistic + ℓ₁/ℓ₂ | same priors | log-loss $+\;\alpha\rho\lVert w\rVert_1+\frac{\alpha(1-\rho)}{2}\lVert w\rVert_2^2$ | no closed form; IRLS or proximal gradient | yes, with ℓ₁ |
Ridge shrinks, Lasso selects, Elastic Net does both. The penalty weight is a noise-to-prior variance ratio. And a coefficient whose credible interval covers zero is not a finding.
Next: see all of this on the real data in the interactive lab, or read how it is deployed in the Uber surge case study.