Build a surge pricing engine

Assumptions you are given

Fixed for every task. Several are choices a real team would have to justify, and questioning them is worth marks.

Task 1 · Maximum likelihood, by hand 10 marks · 30 min

Goal: derive the OLS estimator numerically, then watch it fail twice: once by becoming unstable, once by ceasing to exist.

1a · An ill-conditioned pair

Use is_rain and traffic_speed_kmph on the first six trips, standardized using those six rows only.

1. Compute $X^\top X$ (a $2\times2$ matrix) and $X^\top y$ by hand — show the arithmetic, not just the NumPy output.
2. Report the sample correlation between the two columns and the determinant of $X^\top X$.
3. Invert analytically using $\begin{pmatrix}a&b\\c&d\end{pmatrix}^{-1} = \frac{1}{ad-bc}\begin{pmatrix}d&-b\\-c&a\end{pmatrix}$ and obtain $\hat w_{\text{MLE}}$.
4. Predict the surge for trip 2 (19:00, peak and rain, 6 drivers) and compare with the ₹150 actually charged.
5. Now add trips one at a time. Refit on the first $n = 6, 7, \dots, 15$ trips and tabulate, for each $n$: the correlation, the determinant, both coefficients, and how far each coefficient moved from the previous row.

1b · A pair where OLS does not exist

Now use is_rain and the derived is_bad_weather = (is_rain == 1) & (traffic_speed_kmph < 25) on the first ten trips.

1. Print the two columns side by side. What do you notice?
2. Compute $X^\top X$ and its determinant. Try to invert it and report exactly what happens.
3. State, in one sentence, what "the maximum likelihood estimate does not exist" means here — not "NumPy raised an error" but what is true about the likelihood surface.
4. Find the smallest $n$ at which the problem becomes solvable, and say what changed in the data at that row.
5. Would Ridge return an answer on the first ten trips? Explain why, in terms of eigenvalues. (You do not need to compute it yet — Task 2 will.)

1c · The real thing

1. Fit all eight base features on the full 150-trip training set. Report the coefficients, the condition number of $X^\top X$, and train and test RMSE.
2. Compare that condition number with the ones from 1a. Is the full problem well conditioned? If so, say plainly that this dataset does not need regularization at this sample size — and keep that conclusion in mind for Task 2, where cross-validation will agree with you.

Check yourself: Walkthrough 1 runs these exact computations step by step, and Lab §5 plots the coefficient thrashing as $n$ grows. Use them to confirm your hand arithmetic, not to replace it.

Task 2 · Ridge as MAP 18 marks · 45 min

Goal: implement the Gaussian-prior MAP estimator and read a coefficient path. Walkthrough 2 derives the objective you are implementing, and Walkthrough 3 solves it.

1. Implement $\hat w = (X^\top X + \alpha I)^{-1} X^\top y$ with np.linalg.solve (not inv — say why in one line).
2. Fit for $\alpha \in \{0.1, 1, 10, 100, 1000\}$ on the eight base features. Tabulate all eight coefficients at each α.
3. Plot the coefficient path: α on a log x-axis, coefficients on y, one line per feature. This is Deliverable 1.
4. Identify the α at which is_peak and is_rain come closest together, and explain the convergence using the equal-split argument.
5. Compute training and test RMSE at each α. Then run 5-fold cross-validation on the training split only over a log grid and report the CV-optimal α.
6. Assumption 4 says $\lambda = \sigma^2/\tau^2 = 100$. Does CV agree? If not, which would you trust, and what does the disagreement tell you about the prior?
7. Verify against sklearn.linear_model.Ridge. Note that its alpha uses the same convention as your closed form; state the match to 3 decimals.

Task 3 · Lasso and coordinate descent 18 marks · 50 min

Goal: implement soft-thresholding, then use it on a real selection problem.

3a · The algorithm

1. Implement soft_threshold(rho, gamma) in three lines.
2. Implement cyclic coordinate descent for the objective $\frac{1}{2n}\lVert y - Xw\rVert^2 + \alpha\lVert w\rVert_1$, updating $w_j \leftarrow S(\rho_j, \alpha) / (\lVert x_j\rVert^2/n)$ and stopping when the largest coefficient change falls below $10^{-7}$.
3. Verify against sklearn.linear_model.Lasso(alpha=...) to 3 decimal places at three different α. If they disagree, your objective is scaled differently — find the factor rather than adjusting until it matches.
4. Plot the Lasso path over a log α grid and record the order of elimination.

3b · The new-city launch

Load new_city_features.csv: 60 trips, 120 candidate features. Six are real drivers, and each real driver has a decoy planted beside it, correlated at $r \approx 0.9$–$0.96$. Ground truth is in new_city_truth.csv — do not open it until step 3.

1. Explain in one sentence why OLS cannot be computed here at all.
2. Run your Lasso across $\alpha \in \{1, 2, 4, 6, 8, 10, 12, 15, 20, 25\}$. For each, record how many features survive.
3. Now open the truth file. For each α report recall (how many of the six you found) and precision (what fraction of your selections are real). Plot both against α. Which α would you hand to the city-ops team, and why is it not the α with the best precision?
4. The interesting part. Somewhere in that sweep, Lasso drops a real driver while keeping a feature that is not real. Identify which real driver goes first, which feature survives in its place, and the correlation between them. Then explain why this is the expected behaviour of an $\ell_1$ penalty rather than a failure of your code.
5. Repeat with Elastic Net at $\rho = 0.5$ and at $\rho = 0.9$. Does recall improve? Does precision? Compare the number of features selected — and if Elastic Net turns out to be a worse tool for this particular job, say so and explain what the $\ell_2$ term is doing to sparsity.
6. State the theoretical bound on how many features Lasso can select when $n = 60$, and say whether you ever came near it.

3c · The same penalty, a different question

You meet the ℓ₁ penalty again in Task 6, on a classification target. Set the comparison up now, in one cell, and keep the output:

1. Record the set of features your Lasso keeps at $\alpha = 2$ on the nine-feature regression.
2. Write down, before fitting anything, which features you expect an ℓ₁-penalized logistic model to keep once the target becomes high_surge = surge > ₹100. Task 6c checks your prediction.

Check yourself: Lab §6 runs this live.

Task 4 · Elastic Net and the grouping effect 18 marks · 60 min

Goal: show that the three estimators handle a correlated group in three distinguishable ways.

1. Construct the derived feature is_bad_weather = (is_rain == 1) & (traffic_speed_kmph < 25). Report its correlation with each of its two parents.
2. Fit all three estimators on the nine features (eight base plus the new one), each tuned by 5-fold CV. Report the coefficients on is_rain, traffic_speed and is_bad_weather in a single table.
3. Verify the three predicted behaviours:
   • Ridge keeps all three with comparable magnitudes;
   • Lasso concentrates on roughly one and shrinks the others hard;
   • Elastic Net keeps the group together while still zeroing is_weekend.
   If any of the three does not behave as predicted, say so and investigate why. An honest negative result is worth more than a forced positive one.
4. Sweep $\rho \in \{0.1, 0.3, 0.5, 0.7, 0.9, 1.0\}$ at fixed α and plot the three group coefficients against ρ. At what ρ does the group start to break apart?
5. Test the stability claim directly: bootstrap the training set 200 times, refit Lasso and Elastic Net each time, and report how often each of the three weather features is selected. This is the empirical version of the grouping bound.

Task 5 · The surge engine 18 marks · 30 min

Goal: assemble a system rather than a model.

1. Wrap your best estimator in a function price(hour, is_rain, traffic, drivers, requests, distance_km, ...) → dict returning the raw model output, the surge after policy, the base fare, and the final fare.
2. Implement the policy layer outside the model:

   ratio = requests / drivers
   if ratio < 1.5:        surge = 0
   surge = min(max(surge, 0), 150)
   final = base + surge

3. Simulate: 19:00, BKC, raining, 5 drivers, 25 open requests, 8 km trip. Report every intermediate quantity. Should a driver 3 km away in Andheri reposition? Use the rule "yes if surge > ₹50" and state what that threshold is implicitly assuming about the driver's cost.
4. Sweep the drivers-available count from 1 to 40 with everything else fixed, and plot the resulting surge. Mark where the cap binds and where the threshold switches surge off. Comment on the discontinuity at $R/S = 1.5$ — is a step function the right policy, and what would a rider experience just either side of it?
5. Compute the price elasticity your model implies: if surge rises by ₹50, by how much would demand have to fall for $R/S$ to return to 1? Explain why your data cannot answer this and what experiment would.

Check yourself: Case study §7 is a working reference implementation of steps 1–3.

Task 6 · Logistic regression 18 marks · 60 min

Goal: change the likelihood, keep the priors, and find that the penalties behave exactly as before — then ask whether the binary target was worth creating.

6a · Why not least squares

Build the label high_surge = (surge_additive_inr > 100) on the nine features.

1. Report the number of positives in train and test, and the base rate — the accuracy of always predicting "no alert". Every accuracy you quote from here on must be shown next to it.
2. Fit ordinary least squares to the 0/1 label. Count how many of the 150 fitted values fall outside $[0,1]$, and say in one sentence why that disqualifies it as a probability model.
3. Write down the Bernoulli log-likelihood and show that maximising it is minimising the average log-loss. Two lines; it is the same manoeuvre as Task 1 with a different density.

6b · Implement it

1. Implement penalized logistic regression from scratch, minimising $\frac{1}{n}\sum_i\big[-y_i\log p_i-(1-y_i)\log(1-p_i)\big] + \alpha\rho\lVert w\rVert_1 + \frac{\alpha(1-\rho)}{2}\lVert w\rVert_2^2$ with $p_i = \sigma(x_i^\top w + b)$ and the intercept unpenalized. Either iteratively reweighted least squares (for $\rho = 0$) or proximal gradient descent (any $\rho$) is acceptable; state which you chose and why.
2. Verify against sklearn.linear_model.LogisticRegression at $C = 1/(n\alpha)$, to 3 decimal places.

6c · The penalties, again

1. Plot coefficient paths over a log α grid at $\rho = 0$ and $\rho = 1$. Confirm that ℓ₂ never reaches zero and ℓ₁ does.
2. Record the ℓ₁ elimination order and compare it with your Task 3c prediction. It will probably be wrong. Explain why — which features go first here, and what does that tell you about what an ℓ₁ ordering actually measures?
3. Tune all four models (no penalty, ℓ₂, ℓ₁, elastic net) by 5-fold cross-validation on log-loss, not accuracy. Report held-out log-loss, AUC, accuracy and non-zero count for each.

6d · The threshold is not the model's job

1. Plot the ROC curve for your best model and report AUC.
2. Give the confusion matrix at thresholds 0.3, 0.5 and 0.7, stating how many real alerts are missed and how many false alarms raised at each.
3. Dispatch tells you a missed alert costs roughly four times a false alarm. Choose the threshold that minimises expected cost, show the arithmetic, and say which layer of the system that number belongs in.

6e · Was the label worth making?

The binary target was manufactured by cutting a continuous one. Measure what that cost.

1. Take your Ridge model from Task 2, predict the surge in rupees on the held-out trips, and threshold the prediction at ₹100 to get a yes/no.
2. Compare its accuracy, and the AUC implied by ranking its rupee predictions, against your best purpose-built classifier.
3. Conclude. If thresholding the regression matches or beats the classifier, say so plainly and explain what dichotomising cost you. If it does not, explain what the classifier is exploiting that the regression is not.

Check yourself: the logistic regression lab runs every part of this interactively, and Theory §8 has the derivation.

Bonus · The full posterior +10 marks

1. Compute the Gaussian posterior $\Sigma = (X^\top X/\sigma^2 + I/\tau^2)^{-1}$, $\mu = \Sigma X^\top y/\sigma^2$ with $\sigma = 10$, $\tau = 1$.
2. Confirm numerically that $\mu$ equals your Ridge solution at $\lambda = 100$. Explain why in one sentence.
3. Report each coefficient as $\hat w_j \pm 1.96\sqrt{\Sigma_{jj}}$. Which intervals contain zero? What does that tell you, and how does it compare with what Lasso decided about the same features?
4. For the BKC scenario, compute the posterior predictive variance $x_\star^\top \Sigma x_\star + \sigma^2$ and quote the surge as "₹X ± ₹Y".
5. Which of the two terms in that variance dominates? What would you have to do to reduce each of them?
6. Write two sentences a product manager could put in front of a rider. Then write the one sentence you would refuse to let them write, and say why.

What to hand in

Marking rubric