OLS, Ridge, Lasso and Elastic Net on the same nine features: fit them, read their coefficient paths, choose a penalty honestly, and compare all four on held-out trips.
1 · Model workbench
Choose features, an estimator and a penalty, then watch the coefficients and, more importantly, the held-out error.
Coefficients (standardized units)
₹ change per one standard deviation of the feature.
Predicted vs actual — held-out trips
Points on the dashed line are perfect predictions.
2 · Regularization paths
Each line is one coefficient as the penalty sweeps from negligible to overwhelming. The vertical line marks the α chosen by 5-fold cross-validation.
3 · Choosing the penalty honestly
Three curves. Training error only worsens as α grows, so it is not evidence. Choose on the cross-validated curve; report the held-out one.
With all 150 training trips and 8 well-behaved features, the CV curve is flat and its minimum sits at the smallest α on the grid. Regularization buys nothing here, and the honest conclusion is that this problem does not need it. Drag the sample size down to 25 or 15 trips and a proper U appears: regularization answers a shortage of data relative to parameters.
Over a wide range of α the CV curve often barely moves, so α quoted to four significant figures is mostly noise. Take the simplest model within one standard error of the minimum, and say that you did.
4 · All four estimators, side by side
Each row is tuned by 5-fold cross-validation on the training split, then scored once on the held-out 50 trips. This is the table Deliverable 2 asks for.
| Estimator | α | Train RMSE | Test RMSE | Test R² | Non-zero | Dropped |
|---|
Why the ℓ₁ penalty exists: least squares collapsing under collinearity, and selection when there are twice as many features as trips.
5 · Watch OLS fall over
The Task 1 failure, live. Pick two correlated features, fit on the first n trips, and watch the least-squares coefficients thrash as n grows one trip at a time. Ridge, on the same data, traces a smooth line. Each column is standardized within the subsample, so the correlation shown is the one the estimator faces.
OLS coefficients as trips arrive
Ridge coefficients, same data
6 · New city launch: 120 features, 60 trips
The $p \gg n$ regime from Task 3. Of 120 candidate features, six drive surge, and each has a correlated decoy planted beside it. OLS cannot be computed. See how many of the six Lasso recovers, and what it keeps instead.
Everything on this page can be reproduced in the notebooks.
The outcome becomes a yes/no. Swapping the Gaussian likelihood for a Bernoulli one turns least squares into logistic regression and squared error into log-loss — while the priors, and therefore the ℓ₁ and ℓ₂ penalties, do not change at all. Derivation in Theory §8.
1 · The label, and what making it cost us
There is no binary column in the dataset, so we build one:
high_surge = surge_additive_inr > ₹100. That is an honest operational target — above
roughly ₹100 the dispatch team wants an alert — but it is also a deliberate loss of
information, and the lab measures that cost in §6 rather than hiding it.
Where the cut falls in the surge distribution
Everything to the right of the line becomes a 1. A trip at ₹101 and a trip at ₹150 are now the same observation.
Class balance
A model that predicts "no alert" for every cell scores — accuracy. Any accuracy you report has to beat that number before it means anything, and at extreme thresholds it becomes almost impossible to beat. This is why §5 uses ROC and log-loss instead.
2 · Why a straight line will not do
Fit ordinary least squares to a 0/1 label and it will happily predict −0.3 and 1.4. Logistic regression puts the linear score through a sigmoid, so the output is always a probability:
$$ P(y=1 \mid x) = \sigma(x^\top w + b) = \frac{1}{1 + e^{-(x^\top w + b)}} $$3 · The decision boundary, and what the penalty does to it
With two predictors the model is a straight line in feature space: the set where $x^\top w + b = 0$, i.e. where the predicted probability is exactly one half. Shading shows the predicted probability, so you can see confidence as well as the boundary itself.
Fitted model
4 · Coefficient paths, exactly as before
This is the same figure as Lab §2, with log-loss in place of squared error. The behaviour is identical: ℓ₂ shrinks everything smoothly, ℓ₁ drives coefficients to exactly zero, and the first feature to die should be the one with no real effect.
5 · The model gives a probability; the threshold is your decision
A fitted classifier does not output a class. It outputs a probability, and someone has to choose where to cut. That choice is a business trade-off between missed alerts and false alarms, and it belongs in the policy layer — exactly like the price cap in the case study.
ROC curve
Each point is one threshold. The diagonal is random guessing. AUC is threshold-free, which is why it is the honest headline number.
At this threshold
6 · Four penalties, and the cost of dichotomising
Each row is tuned by 5-fold cross-validation on log-loss over the training split, then scored once on the held-out 50. The last row is the interesting one: it ignores the classifier entirely, predicts the surge in rupees with the linear model from Lab §4, and thresholds that prediction at the same cut.
| Model | α | Log-loss | AUC | Accuracy | Non-zero | Dropped |
|---|
The penalties behave exactly as they did for linear regression, because nothing about the prior changed — only the likelihood. That is the whole point of having derived MAP once, generally: swap in a Bernoulli likelihood and you get penalized logistic regression for free, with the same λ = σ²/τ² reasoning about where the penalty weight comes from.
The second lesson is about the label. If thresholding the regression matches the purpose-built classifier, then collapsing a rich continuous outcome into two buckets bought you nothing and cost you resolution. Check the last row of the table before reaching for a classifier next time.
The penalties behave exactly as they did for linear regression, because nothing about the prior changed — only the likelihood. That is the dividend from deriving MAP once, generally.
The second lesson is about the label. If thresholding the regression matches the purpose-built classifier, then collapsing a continuous outcome into two buckets bought you nothing and cost you resolution. Check the last row of the table above before reaching for a classifier next time.
The other sections pick λ by cross-validation and move on. This one opens that up: four criteria that disagree, the degrees-of-freedom count they are built on, every quantity that moves with λ, the prior-to-posterior update, the matrices themselves, and whether the solver converged.
Nine features, time-based split, scaler fitted on the training rows only — the same setup as the lab, so numbers are directly comparable.
1 · Four ways to choose λ, and why they disagree
Cross-validation resamples. AIC and BIC instead penalize model complexity analytically, using the effective degrees of freedom from §2:
$$ \mathrm{AIC} = n\log\frac{\mathrm{RSS}}{n} + 2\,\mathrm{df}(\lambda), \qquad \mathrm{BIC} = n\log\frac{\mathrm{RSS}}{n} + \log(n)\,\mathrm{df}(\lambda) $$Because $\log n > 2$ for any $n > 7$, BIC charges more per degree of freedom and will always choose at least as much regularization as AIC. Held-out error is shown too — greyed, because selecting on it would spend the test set.
All four curves are min–max rescaled onto [0, 1] so they fit one axis; the dashed verticals mark each criterion's own minimum. Absolute values are in the table.
| Criterion | chosen α | df there | value | test RMSE there | What it is doing |
|---|
2 · Effective degrees of freedom
An unpenalized model with nine features uses nine degrees of freedom. A penalized one uses fewer — but not a whole number. For ridge, writing $X = UDV^\top$,
$$ \mathrm{df}(\lambda) = \operatorname{tr}\!\big(X(X^\top X + \lambda I)^{-1}X^\top\big) = \sum_{j} \frac{d_j^2}{d_j^2 + \lambda}, $$which is the sum of the shrinkage factors from Theory §4.1. For Lasso, the count of non-zero coefficients is an unbiased estimate of df (Zou, Hastie & Tibshirani, 2007), so df is an integer there and drops in steps.
df against λ
Where it comes from: the singular values
Each bar is one direction's shrinkage factor $d_j^2/(d_j^2+\lambda)$ at the current α. Directions with small $d_j$ — the ill-conditioned ones — are the first to be given up.
3 · Everything that moves with λ
Six quantities on one shared axis. Reading them together is how you tell shrinkage from selection, and how you see what the penalty is actually buying.
Error
Coefficient norms
Sparsity — coefficients still alive
Objective, split into its two parts
4 · One coefficient, from prior to posterior
The Bayesian walkthrough shows this in two dimensions as contours. Here it is in one, for a coefficient of your choosing: the noise the model assumes, the prior you brought, and the posterior the data leaves you with.
Assumed noise
$\varepsilon \sim \mathcal{N}(0,\sigma^2)$ — the likelihood's assumption, in rupees.
Prior on this coefficient
$w_j \sim \mathcal{N}(0,\tau^2)$ — what you believed before the data.
Posterior
Prior × likelihood, normalised. Its mode is the ridge estimate.
5 · The matrices, at the current α
What the closed form is actually made of, including the one output most reports omit: the standard errors, from $\operatorname{Var}(\hat w) = \sigma^2 (X^\top X + \lambda I)^{-1} X^\top X (X^\top X + \lambda I)^{-1}$.
XᵀX / n (the correlation matrix)
+ αI on the diagonal
Coefficients with standard errors
6 · Did the solver actually converge?
Ridge has a closed form, so there is nothing to check. Lasso and Elastic Net are iterative, and a tolerance set too loosely will silently hand you a coefficient that should have been zero.
Objective per sweep
Coordinate descent is a descent method: this can never rise. If yours does, the residual update is wrong.
Largest coefficient change per sweep
This is what the stopping rule watches, plotted as log₁₀. Coordinate descent converges geometrically, so a straight line down is the expected shape; a plateau means the tolerance has been reached.
Task 2 asks you to choose λ and justify it. Quoting one CV minimum is the weakest defensible answer. Quoting CV, AIC and BIC, noting that they disagree, showing that held-out error barely moves across the whole region they disagree over, and then taking the simplest model in that region — that is the answer the rubric is looking for.