Model routing

What it is

A routing layer sits between the application and one or more model providers. It can be a simple rule set based on task type or input length, a trained classifier that scores difficulty, or a cascade that tries a cheap model first and escalates when confidence is low. Some vendors now build routing into their own products, choosing internally between fast and reasoning modes.

Why it matters

Model costs and response times vary widely, and most production traffic is not uniformly hard: a large share of queries can be handled well by smaller models. Routing lets teams spend capability where it changes the answer and save it where it does not. It also reduces lock-in, because traffic can be shifted between providers as prices, limits or quality change.

How it works

Practitioners start by classifying real traffic into task types, then benchmark candidate models on each type to find the cheapest model that meets the quality bar. Routing rules are deployed behind a feature flag, monitored per segment, and adjusted as models are updated. Fallback paths handle provider outages, rate limits and cases where the first model returns low confidence or a refusal.

When it applies

It applies when an application handles a mix of easy and hard requests at enough volume for cost or latency to matter, and when more than one viable model is available.

Examples

• An ecommerce assistant routes stock and delivery questions to a small fast model and product comparison questions to a larger reasoning model.
• A content pipeline uses a cheap model to draft meta descriptions and escalates only pages flagged as high value to a stronger model for review.
• A support tool tries a small model first and re-runs the request on a larger model when the confidence score falls below a set threshold.

How it is measured

• Blended cost per request and cost per resolved task across the routed mix
• Share of traffic handled by each model tier and escalation rate
• Quality scores or human review pass rate per route, compared with a single model baseline
• Median and p95 latency by route, including the cost of escalated retries