Model routing to reduce agent cost

Inside a single agent task, the turns are not equally hard. Deciding which file to open, extracting a value from a tool result, formatting a diff, classifying an intent, these are easy, and a small model does them as well as a large one at a fraction of the cost and latency. The hard reasoning is a minority of the turns. Model routing exploits that gap: match each turn to the smallest model that clears the bar. There are three routing designs, in increasing order of sophistication.

Design 1: Static routing by step type

The simplest router assigns models by role. If your agent has named steps, plan, retrieve, extract, edit, synthesize, you already know which are hard. Pin the easy roles to the cheap model and the hard ones to the strong model. No classifier, no runtime cost, and it captures most of the available savings because the distribution of turns is usually dominated by the easy roles. Start here.

Design 2: Difficulty classification

When steps are not cleanly typed, add a lightweight classifier that reads the input and predicts whether the cheap model can handle it. Useful signals: input length, whether the input contains code, the size of the diff being requested, and whether a previous attempt on this step already failed. The classifier itself must be cheap, a small model or even a logistic model on those features, or it eats the savings it is meant to create.

Design 3: Cascade with escalation

The most robust design tries the cheap model first and escalates on evidence. Run the cheap model, then check its output: does it pass a verifier, compile, satisfy a schema, or agree with a second cheap sample? If yes, keep it. If no, redo the turn on the strong model. A cascade pays a small overhead on the turns that escalate but guarantees the ceiling is the strong model's quality: you never do worse than strong model everywhere, you just do it less often.

A good cascade is not cheap model instead of strong model. It is strong model only when the cheap one demonstrably fell short.

Guardrails that keep quality flat

• Never route the final synthesis down. The last turn the user sees is the one where quality is most visible and cheapest to protect. Keep it on the strong model.
• Escalate on failure, always. Any turn that fails a check should be retried on the strong model before it counts as a task failure.
• Constrain the optimizer with completion. Tune the router to minimize cost subject to task completion staying within a tolerance of the all-strong baseline, not to minimize cost outright.

The math, worked

Suppose an average task is 10 turns, the strong model costs 5 times the cheap one per turn, and you can safely route 7 of the 10 turns to cheap. All-strong is 10 × 5 = 50 cost units. Routed is (7 × 1) + (3 × 5) = 22 units, a 56% reduction. Add a cascade that escalates 15% of the cheap turns and you pay roughly one extra cheap turn plus a strong redo: about 27 units, still a 46% reduction, and now with a quality floor equal to all-strong. That is the shape of the win: roughly half the cost, no measurable quality loss, from a change that touches the orchestration layer rather than the prompts.