How do we make model thinking cheaper?

⚠ Provisional (2026-10-10): these verifier numbers were measured on synthetic gold, claims and labels written by Claude, not checked against human labels. They are not results until re-measured on a public human-labelled set. Further runs on this gold are stopped.

The question

Thinking makes our local verifiers much more reliable: qwen3:8b was fooled on reasoning claims at a false-accept bound of 0.39 with thinking off, against 0.15 with it on. But thinking takes 2–5× the time per claim. How do we keep the accuracy and pay for thinking only where it’s needed?

What the research says

1. Escalate instead of always thinking. Cheap-first cascades and routers cut cost sharply at equal quality: FrugalGPT up to 98%, AutoMix more than 50%, Hybrid LLM 40% fewer large-model calls. Keep the router simple, because elaborate learned routers degrade on new data. Trigger escalation on disagreement between cheap answers, not on the model’s own stated confidence, which is poorly calibrated.
2. Never let a budget cut off the answer. On Qwen3, thinking off ties or beats thinking on whenever the thinking budget is small enough to truncate. At the cap, close the thinking and force a verdict instead. Early exit saves 12–80% of tokens with no loss on published benchmarks. Kill loops early.
3. Distill verdicts, not long reasoning. A 770M checker trained on 35K synthetic examples (MiniCheck) comes within 0.6 points of GPT-4 at about 400× lower cost. Students of 3B or less learn poorly from long reasoning traces, so train them on verdicts plus short evidence.
4. Specialize models, not necessarily agents. Narrow specialists beat generalists on narrow tasks, and many adapters can share one base model. But multi-agent systems fail often: 41–87% of tasks across seven frameworks. They help only on work that decomposes into parallel parts, and the biggest single fix is checking the work at every hand-off.
5. Use each device for what it’s good at. The NPU suits fixed-shape encoder work: embeddings, rerankers, an NLI prefilter. Generative decoding stays on the GPU. Cloud GPUs are for bulk teacher runs and models too big for the card.

What we’ll do with it

• switchyard: a cascade that runs thinking off first and escalates on disagreement or “cannot tell”, with thresholds tuned on our own gold.
• offrig: force the answer at the cap, the loop kill, early exit, and a measured ladder of effort levels.
• The distilled verifier: verdicts plus quotes, self-consistent teacher items, and a tuned false-accept threshold.
• Role OS: its own model only for narrow, measurable roles; executable checks at hand-offs; a hierarchical role catalogue.

Caveats

• Most numbers come from abstracts, and most studies are on math or QA, not on checking claims about code. Our own measurements decide the thresholds.
• Composing stackable specialists (adapters per role) wasn’t covered this round. It’s the next question.

Receipts

mcp-tool-shop-org/rnd, entries/2026/2026-10-09-making-thinking-cheaper-study-swarm.md: every source with its arXiv link, and each claim with the lane that retrieved it.