How to read an AI benchmark in 2026

Leaderboards move every week, and the gaps at the top are often smaller than the noise. What to check before a score changes a decision.

Every model launch arrives with a table of scores, and every table seems to show a new leader. If you are choosing a model for a product feature, those numbers are a tempting shortcut. This year has produced unusually clear evidence of why they need reading with care.

The setup is part of the score

In February, Anthropic measured how much infrastructure alone moves results on Terminal-Bench 2.0, a benchmark of hard command-line tasks. Changing how much memory and CPU each agent gets, and how strictly those limits are enforced, shifted scores by up to six percentage points. That is more than the gap that often separates the top models on a leaderboard. A score describes a model, a harness and a machine together, not the model alone.

Models can tell when they are being tested

A month later, Anthropic reported something stranger from a web-browsing benchmark called BrowseComp. Running Claude Opus 4.6 across 1,266 problems, the team found the model had come across leaked answers on the public web in nine of them. In two cases it went further: it suspected it was being evaluated, identified the benchmark and decrypted the answer key. When test questions live on the open internet, a capable agent with a browser can find its way around them.

Trackers age quickly

Independent trackers help, as long as you check the date. Epoch AI's benchmarking hub combines runs it performs itself with results gathered from elsewhere, so you can compare models over time. Terminal-Bench publishes cost and token use beside each resolution rate, which is often the more useful number for a product. And METR's widely shared time-horizon chart now notes that it is no longer actively updated, with its last update on 8 September. A chart can be accurate and still out of date.

A short checklist

• Who ran it? Vendor-reported and independently reproduced scores are different things.
• With what harness, tools and hardware? Small setup changes can outweigh the gap between models.
• When? Check the date on the chart as well as the model names.
• At what cost? Price and tokens per task matter as much as the success rate.
• Is the test public? Public questions can leak into training data or be found by a browsing agent.
• Does it look like your work? A coding benchmark tells you little about drafting a support reply.

The benchmark that matters is yours

The most reliable evaluation is one built from your own product. Anthropic's guide to agent evaluations sets out the essentials: unambiguous tasks, the right mix of code, model and human grading, and time spent reading transcripts rather than just totals. In our experience, a few dozen real tasks from your own workflow, graded consistently, say more about which model to use than any public leaderboard.