A model router for Claude Code, and why it should live on my GPU

For people running Claude Code who wonder whether a small classifier can send their prompts to cheaper models. Short answer: a little, with caveats. The code lives in my private infra repo; the numbers below are from its log.

My timeline has been full of “what Jev can do for you” posts. Jev sizes your request, Jev picks the right model, Jev saves you money. Fine. But every one of those setups has the same shape: before any work happens, your prompt goes to someone else’s model, just so that model can answer with one word.

That bothered me enough to go looking for alternatives. Open weights, hosted locally, on hardware I already own. This post is what came out of the first day: a model router for Claude Code, first built the way everybody builds it, with Jev in the cloud, and then moved onto a local classifier called Kev.

Before anything else, the honest part:

• The savings are small. Claude Code can’t switch the main session’s model per message. The main model still reads every prompt and then hands the job to a helper agent. If the main session runs on opus and the router picks opus, nothing is saved.
• The cloud timeout overshoots. I set 3 s. The log shows calls of 4.07, 4.31 and 4.41 s. More on why below.
• “Jev” isn’t what answers. The router asks OpenRouter for typesafe/jev-router. The log says the answers came from three different general-purpose LLMs. What that alias actually is, I don’t know.
• With the cloud backend, prompts leave the machine. Every prompt goes to OpenRouter and on to whichever vendor serves it. Infra details included. Only the Kev backend keeps them local.
• It doesn’t save money yet. A fine-tuned local sizer beats the cloud one on accuracy and latency, and looks better-calibrated too — though that last comparison isn’t apples-to-apples yet (details below). Its confidence gate also overspends: it now sends everyday work to expensive models 26% of the time. Whether that costs more than it saves is workload-specific and unresolved.

It starts as a skill

I didn’t write the router by hand. I wrote a Claude Code skill, model-router, and let Claude build it. (The skill lives here.) The skill is mostly a prompt, written the way I would ask a colleague:

build me a model router that uses Jev. the idea: Jev sizes up every message i send, and small jobs go to a smaller, cheaper AI model instead of the biggest one.

It then lists what I want: four sizes (tiny, everyday, large, hardest), one helper agent per size, each pinned to a model I actually have. When Jev is less than 60% sure, or my message is a short reply like “yes do that”, the main session handles it itself. It needs an on/off switch, a status check with counts and cost, and it has to be OFF by default. And it must never slow me down or block a message.

Two preconditions in the skill turned out to matter more than the recipe:

• “Find Jev. Do not guess what Jev is.” If the config can’t be found, stop and ask. That one line is why the router logs who actually answered, and that log is where the most interesting finding came from.
• “Be honest about limits.” Claude Code can’t swap the main model per message, so the skill makes the builder say so up front instead of promising a switch it can’t deliver.

The skill ends with a test: 8 sample messages from tiny to hardest, plus 2 short replies, and a table of what was picked and how sure the sizer was. It also has to end with a reminder that while the router is on, my messages pass through OpenRouter. That reminder is what eventually pushed me towards Kev.

What got built

Claude Code has hooks. UserPromptSubmit fires before a prompt reaches the model, gets the event as JSON on stdin, and can return extra context for the model. That is enough to build a router:

1. I type a prompt. The hook pipes the event into router.py, a stdlib-only Python script, with a 5 s hook timeout.
2. Control words are handled locally: router on, router off, router status and router backend auto|kev|openrouter. The script blocks those prompts, so they never reach any model.
3. If state/enabled doesn’t exist, the script exits silently. That file is the on/off switch.
4. Otherwise it asks a sizer to pick one of five labels: tiny, everyday, large, hardest, or contextual for replies that only make sense inside the conversation. Which sizer depends on the backend (next section).
5. If the answer is a tier and confidence is at least 0.6, the script returns additionalContext: “Jev sized this message as ’large’ (confidence 0.78). Delegate the whole task to the router-large subagent (runs on opus)…”. Anything else routes to “self” and prints nothing.
6. The main session delegates to router-tiny, -everyday, -large or -hardest, pinned to haiku, sonnet, opus and fable. Each must end its answer with a Model: line, so I can see who did the work.

It fails open. Any exception means no output and exit 0, and the prompt goes through as if the router wasn’t there. A router that can block my work is worse than no router.

And yes: it routed the request to write up its own first session. “large”, 0.78, opus.

Three backends

The router has a backend setting, kept in state/backend per project:

• openrouter: cloud only. A chat-completions request to typesafe/jev-router, asking for {"size", "confidence"} as JSON.
• kev: local only. A typed request to Kev on 127.0.0.1:8008. If Kev is stopped, there is no routing, and nothing leaves the machine.
• auto (the default): Kev if it’s listening, OpenRouter otherwise.

auto is convenient and a bit of a trap. Kev runs on demand, so whenever it’s stopped, auto quietly means “cloud”. For anything private, I use kev or switch the router off. In this blog’s repo it’s set to kev.

Jev in the cloud: what the log says

The first 16 prompts all went through OpenRouter:

Total cost for sizing: about $0.0032. Latency across the 9 successful calls: 2.99 s on average, from 1.26 to 4.41 s. That’s three seconds before Claude even starts thinking about my prompt. The 7 errors are all ValueErrors, most likely replies without a {…} in them. The first version didn’t log the reply; the current one does, so the next failure will say what came back.

Two things surprised me.

The label lies. I thought I was calling TypeSafe’s Jev, a closed “System One” decision model that answers typed choice and score questions in a single forward pass. The answered_by field in the log says otherwise: openai/gpt-6-luna five times, deepseek/deepseek-v4.1-flash three times, moonshotai/kimi-k3 once. So the router works, but it isn’t calling what I thought it was. A general-purpose LLM, asked nicely to reply with JSON, will sometimes reply with prose instead. That probably explains the errors.

Timeouts aren’t wall-clock. urllib’s timeout=3 applies to each socket operation, not to the whole request. Connecting, sending and waiting for the first byte each get their own 3 s. So a 4.4 s call is still “within” a 3 s timeout. What Claude Code does when it kills a hook at 5 s I haven’t tested. Unknown.

The benchmark found a bug in the thing it was benchmarking. Our first Jev accuracy run came back at 40.2% with a 15% request-failure rate, and I nearly published that as a Jev availability problem. It was our own bug: max_tokens was 400, and jev-router resolves to reasoning models that spend 300–400 tokens thinking before emitting the JSON — measured: 329 reasoning tokens for a 163-token prompt. The budget ran out mid-thought and the reply came back empty. The same ceiling was live in the router itself, where an empty reply hits the fail-open path — so in cloud mode, roughly one prompt in seven had never been routed, silently, indistinguishable in the log from a deliberate decline. Both are fixed now. Worth remembering: the first version of a benchmark can be measuring its own bug, not the thing it claims to measure.

Kev on my GPU

Why send every prompt to a vendor to get a one-word answer, when there’s an RTX 4090 right here?

The candidate is Kev-4B (jaredpalmer/kev-4b, Apache-2.0): a LoRA plus a classification head on Qwen3.5-4B-Base, serving a Jev-shaped POST /v1/systemone. It reports 0.817 against Jev’s 0.857 on Kev’s own Transfer-v4 suite. That’s self-reported, on their benchmark, not mine. I looked at two alternatives. Kev-9B buys +0.005 for about 17 GB of VRAM, so no. Von (395M) is the fallback if Kev-4B turns out to be too heavy.

Kev runs as kev-serve, a local server installed by an Ansible role:

• it pins the Kev commit, since upstream has no tags or releases
• it uses Python 3.12 via uv, because my system python3 is 3.14 and Kev wants <3.14
• it’s a systemd user unit with no [Install] section, so it can’t be enabled by accident: systemctl --user start kev-serve when I want it, stop before I play
• it binds 127.0.0.1:8008, with no auth unless KEV_API_KEY is set

Measured warm latency: 60–130 ms, against 1.3–4.4 s for OpenRouter. GPU memory was 16.7 GB with the plain kernels and 20.4 GB with the fused ones, at the same latency for one request at a time. So fused stays off. The games need the rest of the card, and that’s also why Kev is on demand rather than always on.

Accuracy: how confidence matters more than the headline number

All three sizers, measured on the same 150 held-out prompts, zero request errors in any of the three runs:

Per-tier recall:

Random baseline is 25% throughout. Training: LoRA rank 16 on Qwen3.5-4B-Base, 2 epochs, 1,068 steps, 76 minutes, ~11.8 GB VRAM on an RTX 4090.

A 4-billion-parameter model on a gaming GPU, fine-tuned for 76 minutes on our own session history, sizes our prompts more accurately than the hosted router it was standing in for: 48.0% against 38.7%, on the same 150 held-out prompts. It answers in 85 milliseconds instead of 4.7 seconds, costs nothing per call, and never sends the prompt off the machine.

The accuracy number is not the interesting part. The confidence is.

Calibration by confidence bin — how often each sizer was actually right, conditioned on how sure it said it was:

Jev is confident and flat. It reports 82.8% average confidence while being right 38.7% of the time — overconfident by 44 points — and its confidence carries almost no information: when it says 0.9 or higher it is right 36% of the time, which is worse than its 0.8-0.9 band at 43%. The router’s 0.6 confidence gate therefore admits 145 of 150 prompts at no better than the base rate. The gate is a no-op.

The fine-tuned Kev’s confidence rises with its accuracy: 38%, 41%, 58% as confidence climbs. So its gate does real work — it admits only 26% of prompts, and on those it is right 56.4% of the time against 48.0% overall. A calibrated 48% model is more useful for routing than a confidently-wrong 39% one, because only the first tells you when to trust it.

Before the fine-tune, the gate had never opened at all. Zero prompts out of 150 cleared 0.6. The local sizer had been installed, running, and routing precisely nothing — a bug we only found by measuring it.

Two things stop this being a clean win. The tier bias inverted rather than disappeared: the base model spent 51% of the ideal budget, sending large jobs to cheap models, while the fine-tune spends 114%, sending everyday jobs to expensive ones. So we cannot yet claim it saves money — that depends on how you price a wrong-cheap answer against a wrong-expensive one, and our metrics do not answer it. Note that the conventional “mean cost error” metric improved, 5.83 to 4.21, purely because it is an absolute value and hides the direction entirely.

And no sizer solves the top tier. Across all three models, “hardest” was identified correctly once in 36 attempts. Whatever separates “expensive if wrong” from merely “big” is not in the signal any of them reads.

We also tried giving the model’s pointer head a ten-times-higher learning rate. It changed nothing measurable, made the model spend more, and made its confidence less reliable. Negative results are cheap to run and worth writing down.

A note on methodology: The labels come from behaviour, not length. Each prompt is labelled by the work it actually caused — assistant turns and output tokens until the next user message — read from the AgentsView session database. 2,787 labelled prompts at zero human labelling cost, versus the 50–80 hand-labelled samples originally planned. Prompt length, which an earlier version used, correlates with real workload at r = -0.017 over 2,935 prompts; “we trust typesafe” is 17 characters and caused 31 turns and 50,636 output tokens. Holdout is split by session so near-duplicates cannot leak.

Why Jev’s confidence is flat, and why that objection is testable, not just plausible: There is a fair objection to the calibration comparison above: Kev and Jev were not asked the same kind of question. Kev answers a typed API and returns real token probabilities. Jev, through OpenRouter, was handed a prompt and asked to reply with JSON — including a confidence number it makes up about itself. Comparing a model’s token probabilities against another model’s prose self-estimate is not obviously fair, and it is a plausible explanation for why Jev’s curve is flat.

That objection is testable, because TypeSafe publishes the same typed interface Kev imitates. Its System One endpoint (POST https://api.typesafe.ai/v1/systemone) takes the same request shape — {state, questions} — and returns the same answer fields, probabilities included, down to the same rescaled confidence convention that reads 0 at a uniform distribution. We confirmed this from the TypeSafe SDK that ships inside Kev’s own dependency tree. So Jev can be asked the identical question and made to show its real probabilities instead of a self-report. The test harness is written. It needs a TYPESAFE_API_KEY we do not have — we simply don’t have a key for it, no claim beyond that.

Two outcomes, both informative. If Jev’s calibration curve rises once it is answering a typed question, the flat curve was an artefact of the transport, and part of what we measured says more about our harness than about Jev. If it stays flat while returning genuine probabilities, Jev really is overconfident on this workload, and the local model’s calibration advantage is a real result rather than an unfair comparison.

Until that runs, treat the calibration comparison as suggestive rather than settled. The accuracy and latency gaps do not depend on it.

One thing this incidentally confirms: Kev’s claim to be “TypeSafe-compatible” holds at the schema level — same path, same request keys, same answer fields, same rescaled confidence convention. A local Kev is a genuine drop-in for the hosted API rather than merely a lookalike, which is the reason the router can switch backends at all.

Plugging Kev in wasn’t a URL swap

I hoped to change a base URL and be done. No. Kev doesn’t speak chat completions. It speaks the typed /v1/systemone API: you send a state (my prompt) and a set of questions. The router sends exactly one, a choice question called size, with the five labels and their descriptions as criteria. Kev answers with answers.size.choice and a probability for every option.

Two details matter:

• Kev’s confidence isn’t Jev’s confidence. Kev’s own confidence field is rescaled so that 0 means “uniform over the options”. It doesn’t fit a 60% gate at all. The router uses the probability of the chosen label instead. Same threshold, different number, and without reading the API that would have been a silent bug.
• A stopped port doesn’t refuse, it hangs. On my workstation (WSL2), connecting to the port while kev-serve is stopped doesn’t fail fast. It waits for the full timeout. With Kev on demand, that would add 1.5 s to every prompt whenever Kev is off. So before calling Kev, the router reads /proc/net/tcp and checks whether anything is listening on 8008. Reading a file is instant; a hanging connect() isn’t.

Where the savings actually are

The math is unkind. The main model reads every prompt, and delegation adds a round trip. Routing pays off only when the main session is expensive and the task is small: opus in the main seat, a rename handed to haiku. For everything else I pay for sizing plus delegation and get roughly what I would have gotten anyway.

Kev changes one part of that. Sizing becomes free and fast, so the overhead is only the delegation. The 39 of 150 gated prompts in the held-out test set suggest routing does happen enough to measure. Whether it saves money depends on whether 48% accuracy on routed prompts outweighs the overhead and the cost bias — that is workload-specific, and real data will tell.

What’s not done

• Fix tier collapse. The baseline collapsed toward “everyday”, the fine-tune toward “large”. Both are failures. The training code already ships --focal_gamma to down-weight over-predicted majority classes, which is the direct fix. That knob is what stands between this run and an actual cost win.
• Fix the “hardest” tier. It is the rarest category in training (240 of 2,325 records) and remains nearly broken (8% recall). Balanced oversampling or weight adjustment is the next step.
• Measure Kev on your workload. The benchmark uses 2,787 prompts from one person’s archive. Your 60–70 prompts a week over months will tell whether the model generalizes to your patterns.
• Wire up the feedback loop. The router has a router feedback command (not yet wired in) to log which model actually solved each task. This closes the training loop.
• Rewrite the skill around Kev. The skill still describes a Jev-based router. It should be rewritten around Kev, with Jev as the optional cloud fallback.
• Decide on network exposure. Whether Kev should be reachable from other machines is undecided. Without an API key, no.
• Re-run the calibration comparison fairly. TypeSafe’s System One endpoint takes the same typed request Kev does and returns real probabilities instead of a self-reported number. The harness exists (eval-sizer.py --backend jev-direct); it’s blocked on a TYPESAFE_API_KEY I don’t have.

Lessons

• Read the log, not the label. “jev-router” was answered by three different LLMs. Log who actually answered.
• Write “don’t guess” into the skill. The line that forbade guessing what Jev is did more for correctness than any other part of the recipe.
• A timeout is per socket operation, not per request. Measure wall-clock yourself if it matters.
• A confidence number is only useful if it’s calibrated — and check that you’re comparing like with like. Jev reports 82.8% average confidence at 38.7% accuracy — flat and overconfident, so its gate admits 97% of prompts and filters nothing. Fine-tuned Kev’s confidence tracks its accuracy bin by bin, so its gate actually filters: 26% admitted, 56.4% right on that subset. But Jev was prompted to self-report a number in JSON, while Kev returned real token probabilities from a typed API — different questions, so part of this gap could be the harness, not the model. The fix is testable, not just arguable: TypeSafe exposes the same typed endpoint Kev imitates, so the comparison can be redone fairly once there’s a key for it.
• A benchmark can measure its own bug. Our first Jev run showed 15% request failures from a max_tokens ceiling too low for a reasoning model’s thinking tokens — the same ceiling silently broke the router’s fail-open path in production. Measuring the benchmark found a bug the benchmark itself had.
• Fail-open is a feature, not a bug. Before fine-tuning, Kev never reached the 0.6 confidence gate — every prompt went through unrouted. That’s honest behavior for an uncalibrated model: it knows what it doesn’t know, even if what it knows is “nothing yet”.
• Routing only pays when the main model is expensive and the task is small. Opus in the main seat, haiku for renamed variables.
• Local beats remote on privacy and latency (60–180 ms vs 700–1400 ms), at the cost of 14–17 GB of VRAM on a gaming GPU. Hence on demand, and hence backend kev for anything private.