What is model routing?
Model routing chooses which model answers each request automatically, by difficulty, cost or capability, with fallbacks when one fails. Ladders, classifiers, cascades, and what goes wrong.
Model routing is the practice of deciding, per request, which model should answer it, instead of sending everything to one model chosen once. The decision can rest on what the request looks like, how hard it is, what it may cost, how fast an answer is needed, or which provider is currently working.
The reason to bother is the spread. Within one provider's range the strongest model costs a large multiple of the cheapest per token, and across the work a company sends to an assistant most requests are easy: a greeting, a lookup, a one-line question with one fact wanted. Sending those to the strongest model buys nothing. Sending a genuinely hard request to the cheapest buys a wrong answer. Routing is the attempt to spend where it matters.
Ways to route
Rules on the request. Short plain questions go one way, code or long conversations another, images somewhere that can read them. Cheap and predictable, and surprisingly good at the easy end.
Difficulty classification. A small, fast model, or a dedicated classifier, scores how hard the latest message is, and the score picks a model. Costs one short call, which is usually cached, and catches what rules miss.
Cascading. Try a cheap model first and escalate when its answer looks weak. Saves money on easy work but costs latency on hard work, and judging "looks weak" is its own problem.
Provider pinning. Stay within one provider for compliance or billing reasons but still route by difficulty inside it.
Fallbacks. Not routing in the strict sense, but inseparable from it: when the chosen model fails to start, the next one in a list answers. A good router tells a failing key apart from a failing model, so it exhausts the other keys for a provider before moving down.
Ladders
The common structure for difficulty routing is a ladder: an ordered list of models from cheapest to strongest, and a scale of difficulty levels that maps onto it. Each rung can carry its own fallbacks. A separate ladder for software tasks is common, because the model that is best at reading a schema is not always the model that is best at writing an email.
Where a router serves a whole organisation, ladders need layers: a company default, a team's own choices over it, and a person's over that, with anything left empty inherited from above.
What goes wrong
- Misclassification. Rules and classifiers get requests wrong at the margins. The cost of a wrong "easy" is a poor answer; the cost of a wrong "hard" is money. Floors and ceilings on the level are the usual guard.
- Agent work is never easy. A multi-step investigation driven by a weak model fails in expensive ways, so agent-style requests want a floor well above the cheapest rung whatever the question looked like.
- Opacity. If nobody can see which model answered and why, routing becomes a thing people blame. The decision should ride with the response and sit in the log.
- Routing only what is routed. A request that names its own model never climbed the ladder, so budgets attached to the ladder do not see it. Either accept that or lock model choice to the router.
Where Memtro fits
Ask Memtro for the model auto and a five-level ladder answers: heuristics score the easy cases, a short cached classification call scores the rest, agent mode never routes below level three, and each rung has fallbacks behind it. Ladders are set per organisation, team and person, with a separate coding ladder, and the decision comes back in the response. Cost control puts a token allowance on each choice, and model choice is locked to the router unless an administrator opens it. The routing documentation has the mechanics.