Model launches now ship with a chart before they ship with a changelog. Artificial Analysis Intelligence Index here, Chatbot Arena score there, SWE-bench percentage in the keynote slide. Most readers treat the number as a verdict. Operators who have shipped agents know better: the same model can top a composite index, tie on coding benchmarks, and still lose on your frozen traces. This dossier teaches chart literacy for 2026: what the Artificial Analysis Index measures, how LMSYS Arena scores relate to Elo, why OpenAI stopped reporting SWE-bench Verified, and why Fireworks’ Kimi K3 + Fable oracle routing is a ceiling, not a product spec. If you evaluate RAG or agents in production, pair this with Your RAG Benchmark Is Lying to You and our Kimi K3 routing dossier.
The Artificial Analysis Index: composite, not crystal ball
One number hides weighted benchmarks, token burn, and provider pricing.
Artificial Analysis publishes an Intelligence Index: a composite across multiple evaluations (methodology updated over time; v4.1 includes suites such as GDPval-AA v2, Terminal-Bench v2.1, and others listed on their site). Models are compared on intelligence, speed, and price, with explicit definitions for blended token pricing and cost per task weighted by benchmark mix in the Index.
“Cost per Task: The weighted-average cost (USD) to complete one Artificial Analysis Intelligence Index task. This is calculated by multiplying input, cached, and output token prices by tokens consumed across the workload, weighted by the relative weights of each benchmark in the Intelligence Index, then dividing by task count.” – Artificial Analysis, Benchmarking Methodology
How to read the chart: Treat Index score as a y-axis for broad capability under AA’s harness. Look sideways at cost per task and time per task before you celebrate a +2 point move. A flat Index score with halved time per task is a win for agents, not stagnation.
Merits of the argument: Composite indices reduce arguments at launch time. They cannot replace workload-specific eval without becoming your workload.
Case study: Gemini 3.6 Flash and flat intelligence
Google cites Artificial Analysis; AA’s pre-release testing shows unchanged Index score with better efficiency.
Google’s July 2026 Gemini launch leans on Artificial Analysis for token efficiency claims. On the Index, Google states 3.6 Flash consumes 17% fewer output tokens than 3.5 Flash.
“For example, on the Artificial Analysis Index, we see 3.6 Flash consuming 17% fewer output tokens than 3.5 Flash. It also takes fewer reasoning steps and tool calls to accomplish multi-step workflows.” – Google, Introducing Gemini 3.6 Flash
Artificial Analysis’ own pre-release article reports Gemini 3.6 Flash (high) at 50 on the Intelligence Index, matching Gemini 3.5 Flash, with more than 50% reduction in average time per task (1.3 vs 2.7 minutes) and ~18% lower cost per task ($0.50 vs $0.59).
“Gemini 3.6 Flash scores 50 on the Artificial Analysis Intelligence Index, matching Gemini 3.5 Flash, and just below recently released models Muse Spark 1.1 (xhigh, 51) and GPT-5.6 Luna (max, 51).” – Artificial Analysis, Gemini 3.6 Flash and 3.5 Flash-Lite
Merits of the argument: Vendor blogs will highlight sub-benchmark gains (GDPval-AA, DeepSWE). The Index may stay flat whilst operations improve. Read the release notes table, not the headline badge alone.
Chatbot Arena: Bradley-Terry scores and style control
Crowdsourced pairwise battles; confounding by length and markdown.
LMSYS Chatbot Arena ranks models from anonymous pairwise human preferences. The leaderboard now publishes an Arena score from Bradley-Terry maximum likelihood estimation (formerly described as Elo-like). Scores use a 400-point scaling so win rates between models remain interpretable.
“To improve the quality of our rankings and their confidence estimates, we are adopting another widely used rating system called the Bradley–Terry(BT) model. This model actually is the maximum likelihood (MLE) estimate of the underlying Elo model assuming a fixed but unknown pairwise win-rate.” – LMSYS, Chatbot Arena Elo system update
Arena rankings correlate with capability but confound style: longer answers, headers, and lists win votes. LMSYS added style-controlled regression (length, markdown headers, lists) so model coefficients better reflect substance.
“By doing this, we expect that the Arena Score of each model will reflect its strength, controlled for the effect of length.” – LMSYS, Does style matter?
Merits of the argument: Arena is unmatched for open-ended chat preference at scale. It is a weak sole metric for tool-using agents, compliance-bound workflows, or exact citation tasks.
SWE-bench Verified: contamination and broken tests
OpenAI stopped reporting Verified; recommends SWE-bench Pro for frontier launches.
In February 2026 OpenAI documented two failures: flawed tests rejecting correct solutions, and training contamination on public repo problems. On a hard subset audit, at least 59.4% of audited problems had material test or specification issues.
“In our analysis we found that all frontier models we tested were able to reproduce the original, human-written bug fix used as the ground-truth reference, known as the gold patch, or verbatim problem statement specifics for certain tasks, indicating that all of them have seen at least some of the problems and solutions during training.” – OpenAI, Why SWE-bench Verified no longer measures frontier coding capabilities
“This is why we have stopped reporting SWE-bench Verified scores, and we recommend that other model developers do so too.” – OpenAI, Why SWE-bench Verified no longer measures frontier coding capabilities
Merits of the argument: A rising Verified percentage may measure memorisation and bad graders, not engineering skill. Prefer Pro or, better, your private frozen traces with independent tests, as argued in our RAG eval guide.
Oracle routing ceilings: Kimi K3 and Fable
93% is not your router; it is the best possible model pick per task with hindsight.
Fireworks benchmarked ~1,030 agentic tasks across SWE, terminal, algorithmic, multi-language, and legal families. Headline SWE scores for Kimi K3 and Fable 5 were nearly tied (92.4% vs 92.6%). The lesson is routing: oracle selection beat either model alone.
“Oracle routing is a method for measuring the best theoretical performance by running the task through each model and then picking the cheapest correct option (the cost/performance ceiling). In a practical router, you don’t get to run your task against multiple models.” – Fireworks AI, Kimi K3 is competitive with Fable
“We achieved 93% accuracy with routing between K3 and Fable.” – Fireworks AI, Kimi K3 is competitive with Fable
Oracle routing sent 72–96% of tasks to K3 in their study. Production routers must predict that split without running every task twice. See our Kimi dossier for cache-hit economics and harness constraints when you implement routing for real.
Merits of the argument: Leaderboards rank single models. State of the art for cost-aware agents is increasingly mixtures. Quoting 93% without the oracle caveat misleads procurement.
Failure cases
Case 1: Procurement by Index rank alone
You standardise on “whatever scores 50+” whilst your workload is 90% cheap retrieval QA. You overpay and add latency. Fix: weight benchmarks that mirror task mix; read cost per task on AA charts.
Case 2: Quoting SWE-bench Verified in 2026
Vendor slide still shows Verified at 80%+. OpenAI explicitly recommends stopping. You compare apples to contaminated oranges against competitors on Pro or private harnesses.
Case 3: Arena win rate as agent SLA
Chat preference rewards helpful tone on open questions, not correct SQL against your warehouse schema. Agents need tool-grounded eval, not voter applause.
Case 4: Treating oracle routing results as shipped product
Engineering plans assume 93% because a blog post said so. Production router without labelled traces lands far below oracle. Budget measurement for router training and fallback models.
Migration: build a leaderboard reading habit
- For each launch, write down three axes: Index or task score, cost per task, time per task (AA definitions).
- Check benchmark version strings (AA v4.1, SWE Pro vs Verified, Arena style-controlled table).
- Map vendor claims to primary URLs in a claim audit table before internal slides reuse numbers.
- Freeze 100 production traces per critical workflow; never migrate on public leaderboard alone.
- Separate chat preference eval from agent eval (tools, permissions, escalation).
- When comparing open vs closed, include routing economics (K3 default, premium model on tail) not single-model hero picks.
When headline leaderboard order is enough
- Early exploration when failure is cheap and no production traffic exists.
- Qualitative chat UX trials where Arena-like preference matches user delight.
- Vendor selection shortlists before you invest in private harness build-out.
Once agents touch customer data or money, headline order is insufficient.
What to check right now
- Do internal slides still cite SWE-bench Verified without a contamination footnote?
- When someone cites AA Index 50, do they also cite cost per task and time per task from the same card?
- Is Arena score labelled style-controlled or raw?
- Do routing proposals distinguish oracle ceilings from production routers?
- Does your eval programme include harness integrity checks (see RAG benchmark post)?
- Are Gemini or other vendor AA citations linked to the AA primary article, not recap blogs alone?
Sources and references
Primary links for every quotation and load-bearing number in this dossier. Verify before you reuse a figure elsewhere.
- Artificial Analysis – Benchmarking Methodology – Intelligence Index, cost per task, performance definitions
- Artificial Analysis – Gemini 3.6 Flash and 3.5 Flash-Lite – flat Index 50, time/cost per task deltas
- Google – Gemini 3.6 Flash launch – 17% token reduction claim on AA Index
- OpenAI – SWE-bench Verified analysis – contamination, 59.4% flawed hard subset, stop reporting recommendation
- LMSYS – Bradley-Terry Arena update – Arena score methodology
- LMSYS – Style control in Arena – length/markdown confounding
- Fireworks AI – Kimi K3 + Fable routing study – oracle routing definition, 93% ceiling, 72–96% K3 traffic
- SudoAll – Your RAG Benchmark Is Lying to You – harness and governance complement to public leaderboards
- SudoAll – Kimi K3 cost per task and routing – production routing economics and harness reality
nJoy 😉