How I trained a small coding task classifier to route models in ~50ms

The first idea everyone has for model routing is to ask a model. Send the prompt to a cheap LLM, ask it “which model should handle this?”, route accordingly. It works in a demo and falls apart the moment you do the arithmetic: you’ve added a network round-trip and a few cents of tokens to every single turn, including the one-line typo fixes that were supposed to be the cheap ones. The router becomes a tax on the savings it’s trying to find.

So spawn doesn’t ask a model. It runs a small classifier locally, before the first token leaves your machine, and it answers in about the time it takes to render a frame.

Predict properties, not a model name

The most important design decision was what the classifier outputs. The naive version predicts a label like opus-4.8 directly. That bakes today’s model lineup into the model weights. The day a new model ships or a price changes, your router is wrong and there’s nothing you can edit.

Instead it predicts three durable properties of the task, things that stay true regardless of which models exist this quarter:

task_type: bugfix, feature, refactor, test, design, docs, migration, exploration
complexity: easy, medium, hard
risk: low, medium, high (how costly is a wrong answer here?)

A separate config maps those properties to an actual model. The classifier’s job is just to read the task; the routing decision stays in plain YAML you control. The ontology outlives the model zoo.

# spawn classify "rename getUser → fetchUser across the repo"
{
  task_type:  "refactor"   · 0.92
  complexity: "easy"       · 0.89
  risk:       "low"        · 0.94
}
# 8.1ms tokenize · 41.7ms infer · 0 API tokens

Three heads on one small encoder

The model is a distilled DeBERTa-v3-small encoder with three classification heads sharing one backbone. One forward pass, three softmaxes. Multi-task training was worth it beyond saving parameters: the heads regularize each other. “Hard” tasks and “high risk” tasks share vocabulary (migrate, auth, schema, concurrency), and learning them jointly sharpened both.

I deliberately did not reach for a bigger model. The whole proposition is that this runs on the critical path of every turn, on whatever laptop you happen to have. A 7B parameter “small” LLM would have been a non-starter at 50ms.

The data was the actual work

There’s no public dataset of coding prompts labeled with task type, complexity, and risk, so I built one in three layers:

Seed taxonomy. A few hundred hand-written prompts per label that I was confident about. These are the anchors for each class.
Synthetic expansion. I used a frontier model to generate variations across phrasings, languages, and verbosity: terse Slack-style asks (“fix the off by one”) all the way to paragraph-long specs. Crucially I generated hard negatives: prompts that look easy but aren’t (“just change the timezone handling”) and prompts that look scary but are trivial.
Mined real sessions. Anonymized opening turns from my own coding sessions, labeled with a frontier model and then spot-checked by hand. This is what pulled the distribution toward how people actually type to an agent, which is nothing like how people write training prompts.

The single biggest accuracy gain didn’t come from the model. It came from the hard negatives, teaching it that “just” and “quick” are not evidence of low complexity.

Calibration matters more than accuracy

For a router, a confidently wrong prediction is worse than an uncertain one. If the classifier is only 60% sure something is easy, I’d rather it route up a tier than gamble. So I temperature-scaled the logits on a held-out set until the confidences were honest. The routing policy keys off those calibrated probabilities, so low confidence falls back to the configured default rather than to the cheapest option. The risk head is also asymmetric on purpose: false-low on risk is the expensive mistake, so it’s tuned to over-predict risk rather than under-predict it.

Getting to ~50ms

The trained PyTorch model was nowhere near fast enough on CPU. The path to the budget:

Export to ONNX, then quantize to int8, which gives roughly a 3× speedup with no measurable accuracy loss on my eval set.
Run on ONNX Runtime with the graph frozen and threads pinned, so there’s no per-call warmup.
Cap and truncate the input. Task intent lives in the first couple hundred tokens; feeding the whole pasted stack trace just burns time.

The result lands around 50ms p50 on a normal laptop CPU, no GPU, no network. Tokenization is single-digit milliseconds; inference is the rest. Against a turn that’s about to spend seconds and cents on a frontier model, it’s free.

Where it goes next

The ontology is the contract, and it’s the part I want to keep stable while everything underneath improves. Next up: per-language calibration, a confidence signal surfaced in the UI so you can see why a turn routed where it did, and an opt-in local feedback loop so the classifier adapts to the kinds of tasks you actually send. That last one is the whole reason it lives on your device, which is the subject of the other post.