Sunday, July 5, 2026
Today's topic: ML / statistics insight
The Blessing of Dimensions: When Does the Nearest Neighbor Lie?
You have training points drawn i.i.d. uniformly from the -dimensional unit hypercube , and you want to predict at the origin .
The 1-nearest neighbor classifier uses the single closest training point to make its prediction.
To "capture" a fraction of the data (i.e., so that the expected number of training points within a ball of radius around the origin is ), the required edge length of a sub-cube satisfies:
The puzzle: Suppose you want to use the nearest of the data to make a local estimate. Compute for and explain what this reveals about nearest-neighbor methods in high dimensions. What is the conceptual implication for the bias of 1-NN?
Note: Use throughout.
Ridge Regression as Augmented OLS: The Data-Augmentation Trick
Recall that the ridge regression estimator for minimizes
Problem: Show that this is exactly equivalent to performing ordinary least squares (no penalty at all) on an augmented dataset , where
That is, show that .
Bonus reflection: What does this say conceptually about what ridge regression is doing to the data?
Questions generated automatically twice a day · browse the archive →