← Blog · Part 1 of 2 — single-turn

The cheapest token isn't the cheapest answer

GPT-5.6 Sol, Claude Fable 5 and Kimi K3, run through one neutral gateway and graded deterministically — then priced not per token, but per correct answer. The new OpenAI flagship wins both axes at once. And the twist that names this post: the model with the cheapest tokens isn't the cheapest per answer.

Every week there's a new leaderboard telling you which model is smartest. Almost none tell you what being smart costs — and for anyone shipping, that's the whole question. So we ran a different comparison: three frontier models, the same pipeline, the same deterministic grading, scored on two axes at once — how often they're right, and how many rupees each correct answer costs.

One gateway, so it's actually fair

The quiet problem with most model comparisons is that each model is called through its own SDK, its own defaults, its own quirks. We sidestepped that by routing every call through BharatRouter — one OpenAI-compatible API over all three providers. Same endpoint, same auth, same parameters; the only thing that changes is the model name. BharatRouter is the referee, not a contestant. A useful side effect: the gateway logs tokens, latency and cost per request, so ₹-per-correct falls out of the same run that measures quality.

What we measured, and how we graded it

Five axes, each a standard public benchmark, each graded deterministically — no LLM-as-judge anywhere, so the whole thing reproduces: coding (HumanEval, executed against hidden tests), math (AIME 2025, exact integer), Indic (Global-MMLU Hindi, exact MCQ letter), long-context (a synthetic needle-in-haystack), and instruction-following (IFEval, programmatic checks). Accuracy carries a 95% Wilson confidence interval; gaps inside the interval are ties. It measures auto-gradable capability — not writing quality or taste. A clean limit, not a hidden one.

The headline

The new OpenAI flagship wins both axes at once — most accurate and cheapest per correct answer. That's the surprise: usually you trade one for the other.

Model	Accuracy (95% CI)	₹ / correct	Median TTFT	Throughput
GPT-5.6 Sol proprietary	95.3% [91–97]	₹1.43	40.5s	120 tok/s
Kimi K3 open-weight	92.1% [87–95]	₹2.01	30.9s	46 tok/s
Claude Fable 5 proprietary	86.3% [81–90]	₹4.68	35.9s	85 tok/s

The open-weight model beats one of the proprietary flagships outright. Kimi K3 is more accurate than Claude Fable 5 (92.1% vs 86.3%) and less than half the cost per correct answer (₹2.01 vs ₹4.68). Not a budget compromise — the better buy of the two on both quality and price.
Sol is cheapest despite premium per-token pricing because it's terse — it reaches the answer in far fewer output tokens than the others (more on this below).

By axis

Axis	GPT-5.6 Sol	Kimi K3	Claude Fable 5
Coding HumanEval	95.0%	92.5%	90.0%
Math AIME 2025	86.7%	86.7%	86.7% (tie)
Indic Hindi MMLU	95.0%	90.0%	76.7%*
Long-context needle	100%	100%	93.3%
Instruction-following IFEval	100%	95.0%	95.0%

Math is a dead heat — all three solve 13 of 15 AIME problems; the two they miss are timeouts, not wrong answers (even a 10-minute budget isn't enough for the hardest problems, for any of them). Hindi is where they separate. The asterisk on Fable matters: most of its Hindi "failures" were not wrong answers — they were Anthropic content-filter refusals on medical questions. A deliverability gap, not a knowledge gap (more below).

Why the cheapest token isn't the cheapest answer

Here's the twist that names this post. Kimi K3 has the cheapest tokens of the three — $3/$15 per million in/out, vs Sol's $5/$30 and Fable's $10/$50 (verified to the cent against Anthropic, Moonshot and OpenRouter list prices). Yet Kimi lands more expensive per correct answer than Sol. The reason is verbosity: you pay per token, but you buy correct answers, and a model that thinks out loud burns tokens getting there.

Model	Median output / answer	Reasoning share	Total output tokens
GPT-5.6 Sol	80	~48%	42,861
Claude Fable 5	183	—	57,328
Kimi K3	350	82%	183,716

Kimi emits 4.3× more output than Sol, most of it hidden reasoning tokens it still pays for. Even at half Sol's per-token rate, 4.3× the volume nets out ~2× the output cost — so the cheapest-per-token model isn't the cheapest per answer. The lesson for anyone budgeting a reasoning workload: token efficiency beats token price. (Caching gives 90% off input on Fable and Kimi, but the cost here is output-dominated, which caching never discounts.)

The content-filter gotcha (for anyone benchmarking Claude)

Our first run had Fable scoring 0% on long-context — obviously wrong. The cause: our needle prompt asked the model to retrieve "the secret access code for the data centre," and Anthropic's safety filter read that as credential exfiltration and returned empty (finish_reason: content_filter). Reframing the identical task as retrieving "the winning entry number at the mango festival" fixed it — Fable jumped to 93% and handled 40k-token contexts fine. The same filter clips ~11 of 60 Hindi medical MCQs. If you benchmark Claude on long-context or medical content, watch for silent filter refusals — they look like capability failures but aren't.

Cost is the story the leaderboards bury

Per correct answer, the spread is 3.3× between cheapest (Sol, ₹1.43) and priciest (Fable, ₹4.68). On some axes it's wider — a correct Hindi answer costs ₹0.18 from Sol vs ₹1.29 from Fable; a correct needle answer costs ₹5–8 from Sol/Kimi vs ₹33 from Fable (long context × premium token pricing). For a high-volume workload, that multiple is the difference that matters — and it's invisible on an accuracy-only leaderboard.

Run it yourself

Everything here is one command away; your provider keys stay in BharatRouter's BYOK vault, never in the repo.

git clone https://github.com/bharatrouter/br-model-eval && cd br-model-eval
cp .env.example .env    # your br- key — the ONLY secret; provider keys stay in BR's BYOK vault
make datasets && make preflight && make half && make metrics && make charts

Next: does terse still win inside a real coding agent?

This was single-turn — one prompt, one answer. But most real work happens in an agent: read the repo, plan, edit, run the tests, read the error, try again. Verbosity that costs you once here costs you every turn there. In Part 2 we drop the same three models into a real coding harness and measure ₹-per-task and solve-rate — to find out whether Sol's terseness still wins when the tokens compound, or whether Kimi's self-correction earns its keep. Already sold on Kimi and want it in your agent today? See Kimi K3 in your coding agent — four ways through BharatRouter.

Reproduce it: harness, dated prices and raw per-request data → github.com/bharatrouter/br-model-eval → · cookbook summary → benchmarks/quality-vs-cost ↗