How do we know a model is better?

How do you know if a model is good?

It seems like a simple question. The answer is complicated.

For a sorting algorithm, it's easy: does it sort correctly? How fast? For a language model, "good" can mean: accurate, honest, helpful, harmless, funny, concise, creative… and these qualities don't always point in the same direction.

Evaluating LLMs is a research field in its own right. Every benchmark captures something true and misses something important.

Automated benchmarks

MMLU — Breadth of knowledge

MMLU (Massive Multitask Language Understanding) tests 57 academic disciplines: medicine, law, chemistry, history, mathematics, philosophy… These are four-choice multiple-choice questions, automatically evaluated.

The human reference points are usually misquoted. Non-specialist humans score around 35% — 57 disciplines, nobody masters them all. The ~90% you see everywhere is the estimate for domain experts: a doctor on the medicine questions, a lawyer on the legal ones.

Frontier models have passed that bar. Which is precisely the problem: MMLU is now saturated, it no longer separates the best models, and the community has moved on to harder tests (MMLU-Pro, GPQA, Humanity's Last Exam). We come back to this cycle below.

What it measures: the breadth of knowledge stored in the parameters.
What it misses: the ability to reason about genuinely new situations, to acknowledge uncertainty, to detect a poorly framed question.

HumanEval — Code

164 Python programming problems. The model generates a function, automated unit tests verify it works. The standard metric is pass@k: generate k candidate solutions per problem (often k=1 or k=10), and count a success if at least one passes. pass@1 measures reliability, pass@10 measures raw capability.

What it measures: the ability to produce functional code on well-defined problems.
What it misses: the reality of development — understanding a bug in a 50,000-line codebase, refactoring, documenting.

MATH & GSM8K — Mathematics

MATH: 12,500 high school/competition-level math problems, in LaTeX. GSM8K: 8,500 arithmetic problems in natural language.

What it measures: multi-step mathematical reasoning.
What it misses: mathematical creativity, formal proof, discovery.

The human benchmark: LMArena

Arena is different. Anonymous humans pose any question to two models (displayed without names), read both responses, and choose their preference. The ELO score results from millions of these duels.

It's the only benchmark that measures what humans actually prefer — in all their subjectivity. Ideal length, tone, format, humor, perceived honesty.

What it measures: overall human preference.
What it misses: factual accuracy (humans don't always know which answer is correct), specialized tasks, reproducibility.

Explore the radar

Here are five models compared across six benchmarks — including BBH (Big-Bench Hard), a selection of the reasoning tasks models were still failing when the benchmark was built. Click on a model to see its detailed scores, or on a benchmark to understand what it evaluates.

This is a 2024 snapshot, not today's leaderboard. The models have changed, and several of these benchmarks are now saturated. It's kept as is because what it shows — different strength profiles, no winner across every axis — stays true in every generation.

Each axis is a benchmark. Models have different profiles — strong in code, weak in long-form reasoning, or the reverse. No single radar gives the verdict: you have to combine objective benchmarks with human preferences.

What the radar reveals

Look carefully at the patterns:

No dominant model. On this snapshot, Claude 3.5 Sonnet leads on HumanEval and BBH, GPT-4o dominates Arena and MATH, and Llama 3.1 70B holds its own without ever pulling ahead. The names have changed since; the shape of the graph repeats in every generation.

Arena and academic benchmarks don't correlate perfectly. A model can be excellent on MMLU and average on Arena — and vice versa. Humans appreciate something different from academic accuracy.

Benchmarks are saturating. MMLU was hard in 2020 (GPT-3: 43%). Four years later, every major model sat between 82 and 89%, and today the best ones are past the human-expert bar. Same story on HumanEval, where the frontier is above 95%. When everyone gets nearly everything right, the benchmark no longer ranks anything: differentiation moves to harder tests (MMLU-Pro, GPQA, SWE-bench, HLE).

The fundamental problems of evaluation

Data contamination

If training data contains benchmark answers, the model has "cheated" without knowing it. This is a serious problem with public datasets like MMLU.

The solution: private benchmarks, regularly updated, whose questions don't circulate online. Hard to maintain at scale.

Benchmark hacking

Some labs optimize their models on benchmarks rather than for the capabilities they're supposed to measure. A model can learn to recognize the format of an MMLU question without truly understanding the material.

This is Goodhart's problem: when a measure becomes a target, it ceases to be a good measure.

The human preference question

Arena suffers from a bias: humans tend to prefer long, formatted responses (bullet points, headers, examples) even when a short answer would be more useful. Models that optimize for Arena become verbose.

What no benchmark measures

• The ability to detect an ambiguous question and ask for clarification.
• Honesty: knowing how to say "I don't know" instead of making things up.
• Consistency over long conversations.
• Causal reasoning in genuinely new situations.
• Adaptation to the user's context.

These qualities are difficult to measure automatically — and yet they're often the ones that matter most in practice.

Toward new evaluation paradigms

Research is exploring several directions:

LLM-as-a-judge: use a powerful LLM to evaluate another's responses. Scalable, but circular — the judge's biases contaminate the evaluation.

Adversarial benchmarks: humans actively try to trick models. Measures robustness, not just capabilities under normal conditions.

Real-task evaluation:

• SWE-Bench — real GitHub bugs to fix in existing codebases. The model gets a repo, a bug description, and must produce a patch that passes the tests. Much harder than HumanEval.
• GAIA — multi-step questions requiring reasoning, web search, file manipulation. Measures agentic capability.
• GPQA (Graduate-Level Google-Proof QA) — physics, chemistry, biology questions at PhD level, designed so you can't answer them via Google search. Distinguishes models that reason from ones that retrieve.
• ARC-AGI — abstract visual puzzles, designed to measure general reasoning over novel concepts. No model passed a human-level threshold until late 2024.
• Humanity's Last Exam — questions at the level of the world's best researchers, in domains where classical benchmarks are saturated.

Continuous automated evaluation: systems that continuously generate new questions to track model evolution.

The golden rule

No single benchmark will tell you whether a model is suited to your use case.

The best evaluation is always the same: build a dataset of your own real use cases, evaluate models on it, and compare on what matters to you — not on what matters for leaderboards.

Benchmarks are proxies. The only real test is your problem.

Those leaderboards did surface something unexpected, though. On the hardest tests — GPQA, ARC-AGI, competition math — the very same model can go from mediocre to excellent without a single one of its weights being touched. The only variable that changes is how long you let it think before answering. How do you gain points by thinking longer? Next chapter.

Updated August 2, 2026