Why we divide by √d_k.

The dot product of two random vectors is, on average, larger when the vectors are larger. That sentence is obvious and also important. If our queries and keys live in ℝ⁵¹², then on average q · k grows with the dimension — and softmax, reading those numbers, will collapse into a one-hot distribution before the network has had time to learn anything.

The fix is small and clarifying. We assume each component of q and k is roughly unit-variance. Then their inner product has variance d_k, and dividing by √d_k restores unit variance — softmax operates in a regime where it can still see gradients.

Think of it geometrically. As we add dimensions, two random unit-vectors become, in expectation, nearly orthogonal. Their dot product should be near zero — but its variance grows with d_k because we're summing d_k independent random variables. We're not making the mean bigger; we're making the spread bigger.

Dividing by √d_k is the classic trick for normalising the variance of a sum of independent unit-variance terms. Nothing more. It's the same reason you divide by √n when computing a sample mean.

Let q_i, k_i ~ N(0,1) independently. Then q·k = Σ q_ik_i. Each term has mean 0 and variance 1, so Var(q·k) = d_k. Scaling by 1/√d_k gives variance 1. Softmax now operates over values that don't explode with model width.

It changes gradient flow early in training. Without scaling, softmax saturates — gradients vanish, and the attention heads learn slowly. With scaling, the attention distribution stays diffuse enough to update.

It doesn't change the expressiveness of the model. The scaled and unscaled versions can represent the same functions — scaling just makes optimisation tractable.

It's not magic. It's variance.