The problem with a big tool list
An agent with a hundred useful capabilities cannot put all hundred in the system prompt. Tool definitions are verbose, they compete for attention, and every one of them is paid for on every single turn — which, as the cache post argues, means they are paid for once and then discounted, but they still occupy the context window and they still dilute the model's focus.
So we defer them. The system prompt advertises a small index, and when the model needs something that is not loaded, it calls a search action to pull the right definition in. That works — provided something knows which capability a request actually needs.
What the audit found
We instrumented the router against 346 real local sessions, which is the only corpus that matters, because synthetic prompts are written by people who already know the answer.
The results were bad in a specific and instructive way:
- The remote router missed 37% of deferred-tool turns. On more than a third of the turns where a capability was needed, nothing was loaded.
- 52 turns fumbled three or more search round trips before finding what they needed.
- The misses were workhorse tools, not exotica. The router was not failing on obscure edge cases; it was failing on the plumbing.
- 40% of misses arose mid-turn, after work had already started — which is exactly the case a single up-front routing decision cannot cover.
The remote harness was structurally handicapped
This is the part that made me stop trying to prompt my way out. The side-query had three properties that guaranteed a bad outcome:
- A 2.5-second timeout that silently degraded to a lexical fallback. Silent, in this context, means the router appears to work while quietly switching to a much weaker algorithm.
- A candidate prefilter that narrowed 319 options to 80 before the model ever saw them. If the right answer is not in the 80, no amount of model quality recovers it.
- A "pure chat means no tools" rule that misfired — a plausible heuristic that produces confidently empty results.
Every one of these is an architectural choice, not a tuning problem. A network round trip on the critical path of a turn, with a timeout short enough to be plausible, will fail mid-turn. That is not a bug you fix; it is a shape you avoid.
The shape we chose
A local encoder with two heads. Not a chat model — it generates no text at all.
| Property | Value |
|---|---|
| Backbone | ModernBERT-base, 149M params |
| Head one | Softmax over ~455 skills + <none> |
| Head two | Sigmoid multi-label over deferred tools |
| Export | ONNX int8, ~150 MB |
| Runtime | In-process, no network |
| Latency | Under 30 ms |
| Marginal cost | Zero per call |
Head one answers which single capability owns this request, if
any. The <none> class is not a detail; without
it the model is forced to choose something for every message,
including the ones that need nothing, and you get a router that
confidently preloads an irrelevant tool. Head two is multi-label
because "which tools should be preloaded" is not a single-answer
question — related capabilities travel together.
The input is a compact serialised context
The encoder reads a short, structural view of the conversation rather than the whole thing:
[MSG] … [ATT] … [RECENT] … [PREV_SKILL] … [STEPS] …Each segment exists because of something the audit showed:
-
[MSG]— the current request, the obvious signal. -
[ATT]— what the last turn produced. Intent often lives in the trajectory, not the sentence. -
[RECENT]— a compressed window of prior turns, because "do that again" is meaningless alone. -
[PREV_SKILL]— what was loaded last, which is the strongest predictor of sticky multi-step work. -
[STEPS]— how far into the task we are, because the 40% mid-turn misses need to be catchable after work has begun.
Why local changes everything
At 2.5 seconds and a network hop, you can afford to route once, at the start of a turn. At under 30 milliseconds and no network, you can afford to route often — before the turn, mid-turn, between tool calls, on every message. The architectural limitation that produced the 40% mid-turn miss rate simply stops existing.
Two consequences follow that are easy to miss:
- It works offline and on a slow connection. A router that depends on a remote call is a router that degrades exactly when the user's network does.
- It is private. The routing decision never leaves the machine, so the user's message is not shipped to a second provider just to decide which tool to load.
Put the cheap, high-frequency decisions on the smallest model that can make them, as close to the caller as possible. Save the large model for the work that actually needs it. A 149M encoder deciding which tool to load is not a compromise on capability; it is the right-sized component doing one job well.
What I would tell someone building this
- Audit against real sessions before you design. The 37% number and the 40% mid-turn figure shaped every decision after them. Without that measurement we would have built a better up-front router and missed mid-turn misses entirely.
- Always include a "none" class. A router forced to choose will choose wrongly on a large fraction of ordinary messages.
- Distrust silent fallbacks. A timeout that quietly switches algorithms is worse than a timeout that fails, because it hides its own error rate.
- Measure the prefilter, not just the model. A candidate list that excludes the right answer caps your ceiling at zero regardless of how good the classifier is.