The problem every walkthrough uses
Two features, so the whole parameter space fits on one plot. Pick the pair and the sample size; every walkthrough below re-runs on your choice.
Standardized design matrix (first 6 rows)
Each column has mean 0 and standard deviation 1
within the rows actually used, and y is centred. No intercept appears below
because it would always be zero.
What the two columns look like together
1 · Least squares, entry by entry MLE
The closed form $\hat w = (X^\top X)^{-1}X^\top y$ hides four separate pieces of arithmetic. Here they are in order, with the loss surface drawn as soon as it is determined.
2 · MAP: turning a prior into a penalty MLE → MAP
The other walkthroughs start from a penalized objective and solve it. This one builds that objective: take the likelihood, multiply by a prior, take logs, negate, and a penalty appears with $\lambda = \sigma^2/\tau^2$ falling out of the algebra rather than being asserted. Then dial τ and watch the estimate travel from the MLE to zero.
Data fit against penalty, along the road to the mode
Where τ takes each coefficient
3 · Ridge: one line changes MAP · Gaussian prior
The same four steps, with $\alpha I$ added to the Gram matrix. Watch the effect on the determinant, the eigenvalues and the solution.
4 · Lasso by coordinate descent MAP · Laplace prior
There is no closed form, so we descend one coordinate at a time. Each step freezes the other coefficients, finds the exact optimum along one axis and soft-thresholds it. The trajectory moves in right-angled segments, and coefficients reach exactly zero and stay there.
The soft-threshold being applied right now
Objective value at each step
Coordinate descent is a descent method: this curve can never go up. If yours does, you have a bug in the residual update.
5 · Elastic Net: the same loop, one extra symbol both priors
The update becomes $\hat w_j \leftarrow S(\rho_j,\ \alpha\rho)\ /\ (\lVert x_j\rVert^2/n + \alpha(1-\rho))$: the numerator is Lasso's threshold, the denominator Ridge's shrinkage. Slide ρ from 1 to 0 to turn one into the other.
Threshold and shrinkage, separated
Where all three estimators land at this α
6 · Bayesian updating: the prior losing the argument full posterior
Ridge returns the posterior mode and discards the rest. Here is the whole distribution: start from the prior, add the data a few trips at a time, and watch the posterior contract onto the least-squares answer. The posterior mean is the ridge solution at $\lambda = \sigma^2/\tau^2$.