Model Cascades: Send Easy Requests to the Cheap Model First

Most of the requests your product sends to a model are easy: classify this ticket, extract these fields, answer a short question. Most teams still send all of them to their most capable model, because one model is simpler to run. Moving everything to the cheap model saves money and loses the hard cases. A cascade does both: it sends each request to the cheapest model that can handle it and escalates the rest. This is for teams whose model bill has become a line item someone asks about.

The price gap is big enough to matter

The tiers within a single vendor now differ by more than an order of magnitude. Anthropic’s current list prices, per million tokens (Claude pricing (opens in a new tab)):

For short prompts, Haiku 5.5 is 40 times cheaper than Opus 5.5 on both input and output, and 20 times cheaper than Sonnet 5.5. That gap is what makes routing worth engineering effort. The idea isn’t new: the FrugalGPT paper made the case in 2023, when API prices already differed “by two orders of magnitude”, and reported matching the best single model with up to 98% lower cost (FrugalGPT (opens in a new tab)). That figure was measured on 2023 models and tasks, so treat it as proof the approach works, not as the saving you’ll get.

The last row matters more than it looks. Haiku 5.5 is priced by prompt length, and a request over 100,000 tokens pays the higher rate for the whole request, cached parts included. For long-context work, the gap to Opus shrinks from 40 times to 8.

How a cascade decides what escalates

There are three ways to decide which model handles a request, and they suit different kinds of work.

The first is to route before the call: a small classifier looks at the request and picks a tier. The RouteLLM paper trained such routers on human preference data and reported costs falling “by over 2 times in certain cases” without losing quality, with routers that kept working when the strong and weak models were swapped for others (RouteLLM (opens in a new tab)). Cheap classification models make this more practical than it was. Cloudflare’s Clef-flash, for example, now costs $0.038 per million input tokens and answers text-only decisions in about 130 ms at the median (Cloudflare (opens in a new tab)). Cloudflare positions it for classification such as spam and PII detection; using it as a router is an application, not something the post describes. A router runs on every request, so its cost and latency are paid even when the cheap model would have been fine.

The second is to try the cheap model first and check its answer. The cheap model responds, a check decides whether the answer is acceptable, and anything that fails goes to the bigger model. This works best when the output can be checked mechanically: structured extraction validated against a schema, generated code run against tests, an answer that must cite a document that actually exists. Escalated requests pay for both calls and take longer.

The third is to escalate on runtime signals. Inside an agent, failures announce themselves: malformed output, a tool error, a refusal, a loop. Those are natural points to hand the task to a stronger model.

Choose by whether you can check the output. Where a cheap, reliable check exists, try-then-escalate is simple and safe. Where it doesn’t, such as open-ended answers, a router is the only option, and it needs an evaluation set to tune.

What a cascade really costs

The token arithmetic usually favours the cascade by a wide margin. With try-then-escalate, a request costs the cheap call plus, for the escalated fraction, the expensive call. With a 40-times price gap, that beats sending everything to the big model unless almost every request escalates: the break-even escalation rate is about 97.5%. If the arithmetic is that easy, the real costs are somewhere else.

The first is the prompt cache. Caching only works when the exact same prefix goes to the same model, and Anthropic’s docs are explicit that “each model has its own cache. Switching models recomputes the entire request even when the content is identical” (Claude Code prompt caching (opens in a new tab)). Escalating a long conversation therefore hands the expensive model a cold cache. At Opus 5.5 prices, a 50,000-token prefix costs about $0.01 to read from a warm cache, $0.20 as uncached input, and $0.25 to write to a five-minute cache. A cascade that escalates turn by turn in long conversations can spend more than it saves. Route at the start of a task, not on every turn, and when a conversation must move to a bigger model, hand over a short summary rather than the full history.

The second is long-prompt pricing, covered above. Routing a 150,000-token request to Haiku 5.5 to save money buys an 8-times saving, not 40.

The third is latency. Every escalated request makes two model calls in sequence. For interactive features, that’s the user waiting while the system discovers the request was hard.

The fourth is the one that does real damage: wrong answers that pass the check. Escalation is visible and costs money. A cheap answer that’s wrong but passes validation costs trust, and nothing in the bill shows it. That failure is what your evaluation has to measure, using the same approach as pinning prompts to a regression suite before switching models: a fixed set of real requests, scored before and after the cascade goes live.

Measure it per request

A cascade you can’t observe is a guess. Record, for every request, which tier answered, whether it escalated and why, and what it cost. In an agent, those belong on the run’s root span alongside step count and outcome, as described in observability for AI agents. Aggregated by customer, the same numbers feed inference cost per customer, which is where the saving finally shows up.

Watch the escalation rate by task type, not just overall. A cascade averaging 15% escalation can hide one task that escalates 80% of the time, which belongs on the big model from the start.

Before building any of this for offline work, check the cheaper lever. Anthropic’s Batch API takes 50% off input and output for requests that don’t need an immediate answer, with no routing logic and no quality risk. Nightly enrichment, bulk classification and report generation often save more from batching than from a cascade.

When one model is the better choice

A cascade is engineering you maintain. It isn’t worth it at low volume, where the monthly saving is smaller than the time spent tuning it. It doesn’t pay when most traffic is genuinely hard, since a high escalation rate means paying for both calls on most requests. It works badly in long, multi-turn conversations, where per-model caches make switching expensive. And it’s risky where outputs can’t be checked and no evaluation set exists, because you can’t see the quality you’re losing.

Start with one task type whose output you can check automatically. Route it to the cheap tier with a validation gate, log every escalation, and run it against your evaluation set for two weeks. If the escalation rate stays low and the scores hold, move on to the next task type. If they don’t, you’ve learned that with one task’s worth of risk.

Sources

• Pricing (opens in a new tab), Claude Platform Docs
• How Claude Code uses prompt caching (opens in a new tab), Claude Code Docs
• FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance (opens in a new tab), Chen, Zaharia and Zou, 2023
• RouteLLM: Learning to Route LLMs with Preference Data (opens in a new tab), Ong et al., 2024
• Clef-omni, a faster Clef and a cheaper Clef-flash (opens in a new tab), Cloudflare Blog