Every model launch now comes with a table of benchmark scores. Those numbers are useful, but only if you know what each test measures, who ran it and under which settings. Read that way, a benchmark table tells you which models deserve a closer look. Read carelessly, it can push you towards a model that is no better for your work, or simply more expensive.
Quick answer:
- Check who ran the test. Vendor-reported scores are claims; independent leaderboards are a cross-check.
- Compare only identical setups: same test version, same subset, same reasoning effort, same tool access.
- Treat a gap of one or two points as a tie unless the source shows confidence intervals that say otherwise.
- Use the benchmark closest to your real work as a shortlist filter, then test the finalists on your own tasks.
You can browse current scores on our benchmarks page; this guide explains how to read them.
What the common tests measure
Most headline benchmarks fall into a few families: coding and agent tasks, expert knowledge, computer use, professional work and human preference. The table summarises what each one tests and the trap to watch for.
| Benchmark | What it tests | How it is scored | Watch for |
|---|---|---|---|
| Terminal-Bench (2.1, 4.0) | An agent completing practical tasks in a command-line environment | Share of tasks completed | Versions differ a lot; a 2.1 score cannot be compared with a 4.0 score |
| SWE-bench (Verified, Pro) and DeepSWE | Resolving real software issues in code repositories | Share of issues resolved | The agent harness and scaffolding around the model affect the result |
| GPQA Diamond | Hard graduate-level science questions in multiple-choice form | Accuracy | Top models cluster near the ceiling, so small gaps say little |
| Humanity’s Last Exam (HLE) | Very hard expert-written questions across many fields | Accuracy | “With tools” and “no tools” results are different numbers |
| OSWorld (2.0, 2.1) | Operating a real desktop: apps, files, browsers | Partial or binary task success | Versions, subsets and partial vs binary scoring are all mixed in tables |
| GDPval-AA | Realistic professional work tasks across occupations | A rating, not a percentage | Compare only within the same version |
| Arena Elo ratings | Blind head-to-head votes between two outputs | A relative rating, often with a ± interval | Ratings move as new votes and models arrive |
Vendor-reported vs independent scores
Almost every number in a launch post is run or selected by the company selling the model. That does not make it false, but it is a claim. A clear example: xAI reports Grok 4.7 at 37.6% on Terminal-Bench 4.0 at its xHigh setting, while the independent Artificial Analysis figure for the same test is about 26%.
Vendor tables also include rivals’ scores, and those are not always run the same way. Google’s table for Gemini 4 Argon (self-reported 77.9% on DeepSWE v1.1) takes rival scores partly from leaderboards and system cards. Anthropic’s table shows Claude Opus 5.5 at 66.4% on Terminal-Bench 4.0 against 57.9% for GPT-6 Astra, while the same page’s text says Opus 5.5 “matches” Astra; the page does not explain the difference. Open-weight models are not exempt: Gemma 4’s GPQA Diamond score of 84.3% (31B) is a Google model-card figure that has not been independently reproduced.
The practical rule: when a vendor score matters to your decision, look for the same test on an independent leaderboard, and treat a large gap between the two as a warning sign.
Settings that change the score
The same model can post very different numbers depending on how it was run. Look for the small print next to each score:
- Reasoning effort. Anthropic’s Opus 5.5 scores are reported at max effort. OpenAI reports GPT-6.1 Sol at 75.2% on DeepSWE v1.1 at high effort. Higher effort usually costs more tokens and time, so the score you see may not match what you get at default settings.
- Tool use. On Humanity’s Last Exam, Opus 5.5 scores 67.7% and Claude Fable 5.1 scores 65.0%, both “with tools” (Anthropic-reported). A no-tools score for the same model would be a different number.
- Harness. The model card for Qwen3.8-27B says several of its benchmarks were run with the Claude Code harness. The software wrapped around the model is part of the result.
- Version and subset. Opus 5.5 reports 81.8% on OSWorld 2.1 (partial), Claude Haiku 5.5 reports 72.4% on the OSWorld 2.1 offline subset, and GPT-6.1 Sol reports 71.4% on the OSWorld 2.0 offline set at max effort. These are three different setups, not one leaderboard.
The same applies across versions: Gemini 3.8 Flash reports 89.4% on Terminal-Bench 2.1 and Gemini 4 Argon reports 57.4% on Terminal-bench 4.0. That says nothing about which is stronger, because the tests differ.
Why a 2-point gap rarely matters
Benchmarks are finite samples of questions. Scores move between runs, with prompt formatting and with small changes to the harness. On GDPval-AA v2.1, Anthropic reports 1846 for Opus 5.5 and 1844 for Claude Sonnet 5.5. A difference that small should not decide anything on its own.
Arena-style leaderboards make this visible. On Artificial Analysis’s text-to-speech arena, Eleven v4 Turbo sits at 1330±18 (rank 1) and Eleven v4 at 1328±17 (rank 2). The intervals overlap, so the ranking is effectively a tie. In image generation, GPT Image 2.5 Sunburst (1217) and Flare (1209) hold ranks 1 and 2, a gap that matters far less than price per image or the style you need.
When the top models are this close, decide on price, speed, context window, availability in your region and the tools you already use. Our comparisons, such as Claude Opus 5.5 vs GPT-6.1 Sol and GPT-6 Astra vs Gemini 4 Argon, put those factors side by side.
How to test models on your own tasks
Public benchmarks shortlist; your own tasks decide. A small, honest test takes an afternoon:
- Collect 10 to 30 real tasks from your work: actual emails, bug reports, documents or questions, including a few hard ones.
- Write down what a good answer looks like before you run anything, so you are not grading on impressions.
- Use the same prompt, files and settings for every model, and note the reasoning effort and tools you allowed.
- Hide model names when you compare outputs, or ask a colleague to grade them blind.
- Run each task at least twice; if a model passes once and fails once, count that as unreliable.
- Record cost and time per task. A slightly weaker model that is much cheaper can be the better choice.
- Repeat the test when a vendor updates a model or changes its defaults.
For coding, test inside the tool you will actually use; our best AI for coding page lists the main options.
Common mistakes
- Comparing scores from different versions of the same benchmark.
- Ignoring the effort level or tool access printed next to a score.
- Reading a rival’s number in a vendor table as if the vendor ran it under the same conditions.
- Treating rank 1 and rank 2 as different when their intervals overlap.
- Choosing a model on GPQA Diamond or HLE when your work is writing, support or coding.
- Skipping your own test because a leaderboard already “decided”.
Conclusion
Benchmarks narrow the field; they do not pick a winner for you. Note who ran the test, under which settings and on which version, ignore small gaps, and let a short test on your own tasks make the final call. Check the latest figures on our benchmarks page before you commit.
