Benchmarking Sangam — measured, not marketed.

The three engines

Every run puts the same prompts through three consensus engines. Two are ours; one is the router we're measuring against. The cost column is a rate-card estimate(assumptions below) — the harness replaces it with exact billed cost.

bharatrouter/sangam is the apples-to-apples Fusion competitor: a same-class frontier panel, but billed at our INR catalog rates — its GPT/Gemini/Claude members need your own keys for those providers. bharatrouter/open-sangam is the open-weight route — transparent, reproducible, and served on BharatRouter credit so it runs for a new org as-is, with India residency available via data_policy — the one to try first.

Where each engine wins

Raw benchmark quality is one axis, not the only one. On a hard reasoning leaderboard, Fusion's frontier panel is hard to top — so we say so plainly. But openness, reproducibility, cost, BYOK, and VPC-deployability are axes a closed frontier router can't follow. Pick the engine your constraints demand.

Read the last two rows together with the rest. If the question is "best possible answer, cost and residency no object," Fusion is a fair pick — and so is our own sangam, at INR rates. If it's "best answer on open weights I can audit, reproduce, and eventually run myself — on our credit or my own keys, with India residency when I need it," that'sopen-sangam.

Cost model

Rate-card estimateNot a measured result. Calculated from published per-token rates under the stated assumptions — the harness replaces every figure here with OpenRouter's exact billed cost.

When you run the harness, the cost columns stop being estimates: it records the real billed cost OpenRouter reports for each call, so you see what these engines actually charged onyour prompts.

Methodology — the harness

The benchmark isn't a slide; it's runnable code in the repo:bench/sangam-vs-fusion.mjs. It runs the same prompt set through all three engines, has a blind LLM judge pick the best answer, and records latency, tokens, and measured cost. It refuses to run without keys — because it spends real money on real provider calls, there's no fake-data mode.

Run it yourself — supply your own keys (it will spend real money):

BR_KEY=… OPENROUTER_API_KEY=… node bench/sangam-vs-fusion.mjs

One judge model is not the last word — a single judge has its own preferences. Treat the harness as a method for getting comparable numbers under controlled conditions, and read its output as a distribution across prompts and rounds, not a single score.

Measured results — first run

A first pass through the harness — 6 prompts (reasoning, India-tax, Hindi, code, summarization), 1 round, blind best-of-3 judge (gpt-5). A small, directional sample, not a leaderboard — but real numbers, nothing fabricated.

Consensus uplift — same base model

The result above compares whole engines. This one isolates a narrower question: does theconsensus + verify machinery itself add accuracy — separate from just using a bigger model? To find out we pit open-sangam (3-model panel + verifier + synthesizer) against its own lead model, llama-3.1-8b-instruct, runningalone. Same base model on both sides — so any gap is the ensemble machinery, not extra parameters. Grading is objective (every task has a known correct answer, checked by answer-match) — no LLM judge, so no judge bias.8 classic reasoning & arithmetic traps × 5 rounds =40 graded answers. We ran it twice, independently, and publish both — so what you see is a range across real runs, not one lucky shot.

Across two independent runs the ensemble scored 95.0–97.5% against the single model's stable 87.5% — a +7.5 to +10 point uplift on the same base model. That gap is consensus and verification doing their job, not a heavier model. The per-task breakdown (most recent run) shows where: the ensemble rescues traps a single small model slips on — e.g. clock-angle and speed-convert(5/5 ensemble vs 4/5 alone), and the harddigit-7 count (3/5 vs 2/5) where both still struggle — honest about the ceiling.

How this compares to what the field has published

Consensus isn't ours alone — Together, OpenRouter and Sakana have all published results showing a panel beats its own members. These are different benchmarks on different baselines, so this is not a ranking — read the direction, and note who is independently reproducible.

Read honestly: Fusion and Fugu lift frontier panels a few points higher (a higher ceiling than our open-weight panel, by design); Together MoA shows open models beating GPT-4o. Ours is the one that isolates the machinery — same base model both sides — and ships a runnable, objective-graded harness with two published runs. Fugu's numbers are vendor-reported and not yet independently reproduced; ours you can reproduce from the repo today. Sources:Together MoA ·OpenRouter Fusion ·Sakana Fugu.

Try it yourselfHow Sangam works