Every Scaling Law Is a Statement About Bits

Probability theory and information theory are the two branches of mathematics AI runs on, and they fit together so tightly they are almost one subject. Probability supplies the semantics: a model is a distribution, learning is inference, a prediction is a set of probabilities. Information theory supplies the ledger: entropy measures the uncertainty in a distribution, and cross-entropy measures how much one distribution misses another. Every loss function in modern machine learning is a cross-entropy wearing a disguise, which is why these two theories keep appearing wherever a model is trained.

Cross-entropy, not squared error, is the loss classification actually wants because it measures what classification cares about: the distance between distributions. Written as $H(p, q) = H(p) + \mathrm{KL}(p \,\|\, q)$, its data term is constant, so optimizing it is pure KL minimization, and it is the only additive proper scoring rule for categorical outcomes. Squared error treats probabilities as Euclidean coordinates and ignores the geometry of the simplex: paired with softmax, its gradient collapses precisely when the model is confidently wrong, so the worst predictions learn slowest. Log loss inverts this. The penalty grows without bound as the assigned probability shrinks, and the gradient reduces to prediction minus label.

Entropy itself measures the expected surprise of a draw from a distribution, counted in bits, where surprise means the negative log of a probability: $H = -\sum_i p_i \log p_i$. Shannon’s formula is not one tally among many; it is the unique one consistent with three mild demands: continuity in the probabilities, invariance when a zero-probability outcome is added, and additivity when an event splits into sub-events. Under it, entropy is the average number of bits an optimal code needs per draw, the floor compression reaches toward, and the benchmark cross-entropy scores against.

Mutual information measures how much one variable reveals about another: $I(X;Y) = H(X) - H(X \mid Y)$, the uncertainty left about $X$ after $Y$ is observed. It is symmetric, non-negative, and zero only under independence. Contrastive learning maximizes this quantity in disguise. Pulling matching pairs together and pushing mismatches apart raises a lower bound on $I(X;Y)$ called InfoNCE. Mutual information is hard to estimate in high dimensions, so the field works through bounds: negative samples supply the denominator, and each extra negative tightens the bound. What looks like a similarity trick is representation learning by compression.

Perplexity is cross-entropy counted in bits per token: two raised to the power of the cross-entropy, $2^H$, the effective number of choices the model faces at each step. This is the ruler scaling laws are fitted with. Loss falls as a power law in compute, data, and parameters, and that loss is cross-entropy against the data, bits of surprise paid per token. The two theories meet in this one number: probability supplies the distribution, information theory prices it, and training is spending compute to drive the price down. Every scaling law is a statement about bits.