Bigger, always better?

A misleading intuition

For years, the AI industry ran on a simple belief: a model twice as big is better. GPT-2 (1.5 billion parameters) was eclipsed by GPT-3 (175 billion). PaLM, Megatron, Gopher — the parameter race seemed endless.

Then in 2022, a team at DeepMind published a paper that changed everything. Their thesis: the large models of that era were massively undertrained. Not too small — undernourished in data.

The model that proved the thesis was named Chinchilla.

Kaplan's law: the first formulation

In 2020, OpenAI publishes a paper by Jared Kaplan and colleagues — "Scaling Laws for Neural Language Models" — showing something remarkable. Across dozens of models trained at different sizes, validation loss follows a simple power law:

L ≈ L∞ + (C₀ / C)^α

Decoding the formula:

• C — total compute spent on training (in FLOPs).
• L — final validation loss.
• L∞ — the irreducible loss: the floor below which you can't go, even with infinite compute. It's the natural entropy of human language — there's always some unpredictability in the next word.
• C₀ — a normalization constant that depends on the architecture.
• α ≈ 0.05 — the power-law exponent.

In plain terms: doubling compute reduces loss by a predictable amount. The law is remarkably robust over 7 orders of magnitude.

Kaplan draws a conclusion that will guide the industry for two years: given a compute budget, allocate most of it to model size, little to data.

That's exactly what OpenAI did with GPT-3. 175 billion parameters, but "only" 300 billion training tokens.

Chinchilla flips the table

In 2022, Hoffmann et al. (DeepMind) redo the experiment with a different methodology. Instead of fixing model size and varying compute, they systematically explore the (N, D) plane at constant compute.

Their conclusion directly contradicts Kaplan: N and D should grow at the same rate. To minimize loss given a fixed compute budget, train a modestly-sized model on a lot of data.

More precisely, the optimal ratio is:

D ≈ 20 × N

For a 70-billion-parameter model, the optimum is about 1.4 trillion tokens (1,400 billion). GPT-3 (175 billion parameters, 300 billion tokens) had a ratio of 1.7 — around a dozen times below optimum.

DeepMind proved it by training Chinchilla: 70 billion parameters, 1.4 trillion tokens. Smaller than GPT-3, more tokens, and better on every benchmark.

The compute map

Two sliders, one point on the map. Put it first on a big, underfed model (high N, low D), then do the reverse. Watch the D / N ratio and the estimated loss at each move: the whole Kaplan-versus-Chinchilla debate is contained in what that number does when you move things around.

On the log-log plot, loss decreases as a power law of compute. The N (parameters) and D (tokens) sliders show the iso-compute curve: for a fixed budget, there is an optimal N/D ratio — about 20 tokens per parameter according to Chinchilla.

Move the point to explore the (N, D) plane. The Chinchilla diagonal is the line where each dollar of compute is spent optimally. Above it, you've trained too long on too small a model; below, the opposite.

You'll notice something interesting: LLaMA-3 sits well above the diagonal. With 70 billion parameters trained on 15 trillion tokens in 2024, its ratio is 214 — ten times above Chinchilla optimum. That was striking at the time; since then the overtraining trend has intensified further, and small models routinely exceed that ratio.

Why? Because Meta optimized for something other than training compute efficiency. They optimized for inference cost. A smaller model trained longer costs more to train (a bit) but much less to serve in production. Across billions of requests, the savings are massive.

Beyond parameters: data quality

Scaling laws aren't the end of the story. Several limits emerge.

The supply of high-quality data is finite. Common Crawl, Wikipedia, GitHub, ArXiv, scanned books — the inventory of genuinely clean text on the internet isn't infinite. An imminent "data wall" was announced for years: training a 1-trillion-parameter model at Chinchilla optimum would require 20 trillion tokens. That wall keeps receding. Quality filtering (FineWeb and its successors), multilingual data, controlled repetition over several epochs and synthetic data have made 15- or 30-trillion-token training runs routine. What's missing isn't text any more — it's good text.

Quality beats quantity, but only up to a point. Filtering a corpus to keep only high-quality data (textbooks, technical books, clean code) improves the model more than adding mediocre data. But over-aggressive filtering eventually impoverishes the distribution and hurts generalization.

Emergent abilities blur the curve. For certain tasks (multi-step reasoning, complex math, rare instructions), performance stays flat until a certain size threshold — then jumps sharply. These "emergent abilities" are controversial: some researchers (Schaeffer et al., 2023) show they disappear when you pick a more continuous metric. But the practical phenomenon remains: small models simply cannot do certain things, no matter how much fine-tuning.

The practical lesson

You'll probably never train a model yourself. But these laws explain the shape of the landscape in which you pick yours.

Why the models you use are small and overtrained. At a fixed compute budget, the training optimum is a D/N ratio near 20. Almost nobody sticks to it any more — and that's deliberate. A smaller model fed for longer costs a bit more to train, but far less to serve. Across billions of requests, it's that second line item that decides. Hence 8B and 70B models everywhere, trained well past Chinchilla, rather than 500-billion-parameter monsters.

Why a small model hits a wall on certain tasks, whatever you do. Emergent abilities only appear past a size threshold. If a model fails on a whole class of problems, no prompt and no fine-tune will rescue it: you have to change model, not method.

Why new generations improve without growing. For a few years now, the margin has come from data rather than parameters: filtering, diversity, quality. A model the same size as last year's can be markedly better without gaining a single parameter — what changed is what it read.

Scaling laws don't say you should keep growing indefinitely. They say there's a right ratio between parameters and data — and we spent years getting it wrong.

That said, these laws describe a curve, not a mechanism. They predict that a better-fed model will make fewer mistakes, without saying anything about what it learned to get there. What is actually inside those billions of parameters — is there anything readable in there, or just a mush of numbers that happens to work? The next chapter tries to open the box.

Updated August 2, 2026