Learning outcomes
By the end of the module a student should be able to:
- Derive the OLS estimator from the Gaussian likelihood and state which assumption each step depends on.
- Convert a prior on the coefficients into a penalty term, in both directions — given a penalty, name the prior it corresponds to.
- Implement Ridge in closed form and Lasso by coordinate descent, without a library.
- Diagnose collinearity from the correlation matrix and condition number, and predict before fitting how each estimator will respond.
- Select a penalty by cross-validation, and explain why the training RMSE is not evidence.
- Choose between the four estimators for a stated business constraint (interpretability, stability, latency, regulatory review) and defend the choice.
- Report a coefficient with a credible interval and say what the interval does and does not mean.
Prerequisites
Linear algebra
Matrix multiplication, inverses, rank, eigenvalues. The SVD is used but re-introduced in Module 4.
Probability
Joint and conditional densities, Gaussian and Laplace distributions, Bayes' theorem. No measure theory.
Python
NumPy array manipulation. Pandas and Matplotlib are used but can be picked up from the starter notebooks.
Weekly schedule
The linear model, least squares, and the likelihood behind it
Design matrices and the normal equations. Projection geometry. Gaussian likelihood ⇒ least squares. Estimating $\sigma^2$. First look at the Mumbai dataset and its correlation structure.
Lab: compute $X^\top X$ and $X^\top y$ by hand for five trips; invert the $2\times 2$ system; watch the coefficients blow up. Reading: Theory §1–2. Due: Task 1.
Priors, MAP, and Ridge regression
Bayes' theorem for parameters. $-\log p(w)$ as a penalty. Gaussian prior ⇒ $\ell_2$. $\lambda = \sigma^2/\tau^2$. Shrinkage in the SVD basis, effective degrees of freedom, and the equal-split argument for correlated columns.
Lab: Ridge coefficient paths; find where
is_peak and is_rain converge.
Reading: Theory §3–4.
Due: Task 2.
Sparsity: the Laplace prior and coordinate descent
Soft-thresholding derived from the subdifferential. Coordinate descent implemented line by line. The diamond-corner geometry. Selection instability among correlated features, the $p > n$ saturation bound, and the bias of the survivors.
Lab: Lasso on the 120-feature "new city launch" data where $p \gg n$; recover the six real drivers. Reading: Theory §5. Due: Task 3.
Elastic Net and the grouping effect
Both penalties at once. The Zou–Hastie grouping bound. Double shrinkage and its correction. Choosing $\alpha$ and $\rho$ by two-dimensional cross-validation, and why the CV curve is flat-bottomed more often than students expect.
Lab: construct is_bad_weather and show the
three estimators disagree exactly as theory predicts.
Reading: Theory §6.
Due: Task 4.
The surge engine: from coefficients to a price
Demand and supply forecasts as separate regressions. The multiplicative-to-additive redesign and why it is incentive-compatible. Price caps, the no-surge rule, censored targets, and what a cap does to your estimates. Latency and retraining cadence.
Lab: ship the decision rule; simulate 7 pm in BKC in the rain. Reading: Case study. Due: Task 5.
Uncertainty, ethics, and model choice
The full Gaussian posterior and credible intervals. Posterior predictive variance. Empirical Bayes for $\lambda$. Surge caps as a fairness constraint; what a model trained on censored prices learns. Seminar: where each estimator actually ships.
Lab: quote a surge with an honest error bar. Reading: Theory §7, Applications. Due: Bonus task + final write-up.
Classification: a different likelihood, the same priors
Bernoulli likelihood and log-loss. Why least squares on a 0/1 label is wrong. Penalized logistic regression, solved by IRLS or proximal gradient. ROC, AUC and the base rate. The threshold as a policy decision. And the cost of dichotomising an outcome you could already model continuously.
Lab: the logistic regression lab — sigmoid, decision boundary, coefficient paths, ROC. Reading: Theory §8. Due: Task 6.
Assessment
| Component | Weight | What is actually marked |
|---|---|---|
| Task 1 — MLE / OLS by hand | 10% | Correct hand computation of $X^\top X$, and an account of why the coefficients are unstable rather than just that they are. |
| Task 2 — Ridge | 18% | Closed-form implementation, the coefficient path plot, and identification of the λ that minimises held-out error — with the selection method stated. |
| Task 3 — Lasso | 18% | Working coordinate descent matching sklearn to 3 decimals, plus the
feature-selection argument on the $p \gg n$ data. |
| Task 4 — Elastic Net | 18% | The three-way comparison on the correlated group, and an explanation of the grouping effect that does not restate the result. |
| Task 5 — Surge engine | 18% | End-to-end decision rule with cap and threshold, the BKC simulation, and a deployment recommendation with a defensible reason. |
| Task 6 — Logistic regression | 18% | From-scratch penalized logistic regression matching sklearn, log-loss
used for model selection rather than accuracy, a justified threshold, and a defensible
answer to whether the binary label was worth creating. |
| Bonus — Bayesian posterior | +10% | Credible intervals, and a correct statement of what they mean. |
A model that fits the training set better is not a better model. Reporting training RMSE as evidence of quality loses marks in every task. Report held-out or cross-validated error, and say which.
Software
Python 3.10+, NumPy, pandas, Matplotlib and scikit-learn. Everything runs in Colab without
installation. Implement each estimator from scratch first; sklearn is permitted only as
a correctness check.
Collaboration policy
Discuss freely; write alone. Comparing numerical answers with a classmate is fine — if your Ridge path and theirs disagree, one of you has a bug and finding it is the exercise. Submitted code and prose must be your own. If you use an AI assistant, say so in one line at the top of the notebook and state what you asked it.