🧮 Brain Teaser
← Archive
ML/Stats
The Gaussian That Knows Its Own Variance
2026-09-13
✏︎

The Gaussian That Knows Its Own Variance

Suppose you observe a single data point xx drawn from a Gaussian with known mean μ=0\mu = 0 but unknown variance σ2\sigma^2. You want to estimate σ2\sigma^2.

The MLE gives σ^MLE2=x2\hat{\sigma}^2_{\text{MLE}} = x^2.

Now consider the family of estimators σ^c2=c⋅x2\hat{\sigma}^2_c = c \cdot x^2 for some constant c>0c > 0.

Question: Find the value of c∗c^* that minimizes the mean squared error MSE(c)=E[(cx2−σ2)2]\text{MSE}(c) = \mathbb{E}\left[(c x^2 - \sigma^2)^2\right].

Is c∗=1c^* = 1 (the MLE)? If not, which direction does the optimal c∗c^* move, and why does this make intuitive sense?

bias-variance tradeoffMLEMSEchi-squaredshrinkageadmissibility

Answer: The Gaussian That Knows Its Own Variance

Key Idea / Intuition

The MLE is unbiased but can have large variance. A slightly shrunken estimator — choosing c<1c < 1 — trades a little bias for a big reduction in variance, and the net effect is a lower MSE. This is the bias-variance tradeoff made completely explicit in a one-parameter family: the optimal c∗c^* is strictly less than 1, meaning the MLE is inadmissible in the MSE sense even for this simple problem.

Formal Proof / Solution

Setup. Since x∼N(0,σ2)x \sim \mathcal{N}(0, \sigma^2), we have x2/σ2∼χ12x^2 / \sigma^2 \sim \chi^2_1, so:

E[x2]=σ2,E[x4]=3σ4\mathbb{E}[x^2] = \sigma^2, \qquad \mathbb{E}[x^4] = 3\sigma^4

(using the fact that for Z∼χ12Z \sim \chi^2_1, E[Z2]=3\mathbb{E}[Z^2] = 3, i.e., Var(x2)=E[x4]−(E[x2])2=3σ4−σ4=2σ4\text{Var}(x^2) = \mathbb{E}[x^4] - (\mathbb{E}[x^2])^2 = 3\sigma^4 - \sigma^4 = 2\sigma^4).

Expand MSE.

MSE(c)=E[(cx2−σ2)2]=c2E[x4]−2cσ2E[x2]+σ4\text{MSE}(c) = \mathbb{E}[(cx^2 - \sigma^2)^2] = c^2\mathbb{E}[x^4] - 2c\sigma^2\mathbb{E}[x^2] + \sigma^4

Substituting:

=c2⋅3σ4−2cσ2⋅σ2+σ4=σ4(3c2−2c+1)= c^2 \cdot 3\sigma^4 - 2c\sigma^2 \cdot \sigma^2 + \sigma^4 = \sigma^4\left(3c^2 - 2c + 1\right)

Minimize over cc.

ddc(3c2−2c+1)=6c−2=0  ⟹  c∗=13\frac{d}{dc}\left(3c^2 - 2c + 1\right) = 6c - 2 = 0 \implies c^* = \frac{1}{3}

Check MLE. At c=1c = 1:

MSE(1)=σ4(3−2+1)=2σ4\text{MSE}(1) = \sigma^4(3 - 2 + 1) = 2\sigma^4

At c∗=1/3c^* = 1/3:

MSE(1/3)=σ4 ⁣(39−23+1)=σ4 ⁣(13−23+1)=2σ43\text{MSE}(1/3) = \sigma^4\!\left(\frac{3}{9} - \frac{2}{3} + 1\right) = \sigma^4\!\left(\frac{1}{3} - \frac{2}{3} + 1\right) = \frac{2\sigma^4}{3}

So the optimal estimator achieves one-third the MSE of the MLE!

Why the direction makes sense. The MLE x2x^2 is unbiased but has very high variance (2σ42\sigma^4). Since x2x^2 has a right-skewed distribution (chi-squared), the typical value of x2x^2 tends to overestimate σ2\sigma^2 on average in terms of squared loss. Shrinking by c∗=1/3c^* = 1/3 introduces downward bias but dramatically cuts variance. The bias-variance tradeoff strongly favors shrinkage here.

Decomposition at c∗=1/3c^* = 1/3: Bias2=(13σ2−σ2)2=4σ49,Variance=c∗2⋅Var(x2)=19⋅2σ4=2σ49\text{Bias}^2 = \left(\frac{1}{3}\sigma^2 - \sigma^2\right)^2 = \frac{4\sigma^4}{9}, \qquad \text{Variance} = c^{*2} \cdot \text{Var}(x^2) = \frac{1}{9} \cdot 2\sigma^4 = \frac{2\sigma^4}{9} MSE=4σ49+2σ49=6σ49=2σ43✓\text{MSE} = \frac{4\sigma^4}{9} + \frac{2\sigma^4}{9} = \frac{6\sigma^4}{9} = \frac{2\sigma^4}{3} \checkmark

The MLE is unbiased but not MSE-optimal — a recurring theme in statistics.

Source: The Elements of Statistical Learning, Hastie–Tibshirani–Friedman (2nd ed.); statistical folklore

Type: ML/StatsSource: The Elements of Statistical Learning, Hastie–Tibshirani–Friedman (2nd ed.); statistical folkloreEdit on GitHub ↗