LLM Orchestrator

NVIDIA Research's position paper Small Language Models are the Future of Agentic AI (Belcak et al., arXiv:2506.02153) argues that most agent calls don't need a frontier model, and gives a concrete six-step procedure for converting an LLM-backed agent to smaller specialist models: log real usage, curate and strip it, cluster the tasks, pick candidate SLMs, fine-tune, then retrain on a schedule.

Rather than take the claim on faith, I ran the procedure against my own agent traffic. Two of the paper's arguments shaped the build directly:

• Agentic interactions are the dataset. The paper treats ordinary usage as the data-collection pathway, so the router logs every decision to a JSONL ledger — that ledger is step one, not an afterthought.
• Don't judge SLMs on generalist benchmarks. The paper is explicit that MMLU-style scores mispredict agentic utility, so the evals here score routing decisions on real logged tasks instead.

The first check was the paper's own estimate that 40–70% of LLM calls in published agent systems are SLM-replaceable. Replaying my telemetry in hindsight — asking, for each decision, what the cheapest model that still succeeded would have been — put the ceiling at 66% of local-tier spend. Probing a real 35B against those same tasks brought the defensible figure to 57%, landing inside the band the paper predicted, on a corpus it never saw.

That measures whether conversion is worth doing. It doesn't do it. So I ran the actual pipeline — twice.

Full method, every caveat, and the twenty-five findings: the reproduction write-up in evince55/aria-llmops.

Round one converted the router's own task-difficulty classifier. A QLoRA-tuned 3.2 GB Gemma-4-E2B matched a 5.8 GB keyword+9B hybrid on a promotion gate fixed before the first run — promote iff accuracy ≥ incumbent and no tier recall regresses by more than 0.05 — and ships as the default. It is a tie, not a win; the case for shipping it is operational, at 55% of the memory with no network dependency.

Round one's own conclusion was that I had picked the wrong thing to convert. A perfect classifier changes only 16.5% of routing decisions, so the pipeline was sound and the target was low-leverage. Round two acted on that: it chose its target by measurement first — probing two candidates and discarding the one whose base model already scored 1.00 — and converted tool-call emission, an agentic subtask rather than a classification. Ground truth is authored with each example, so verification is exact structural comparison: no LLM judge, no teacher, and the round costs nothing to run.

Running every request against Claude is fast and smart, but the bill scales with volume and every keystroke leaves the building. Running everything locally on one consumer GPU is cheap and private — but a single quantized model isn't reliable on the hard ten percent, and naive "just use the big model" routing throws the cost advantage away.

I wanted one harness-agnostic interface that defaults to local inference, escalates to Claude only when a task genuinely needs it, and can prove — per task — what each routing decision cost. The constraints were real:

• One AMD RX 7600 XT (16 GB) — ROCm/Vulkan, no CUDA — so model size, quantization, and KV-cache tuning are hard limits, not nice-to-haves.
• Must work across harnesses (Claude Code + opencode) without bespoke logging bolted into each one.
• Every routing decision has to be measurable after the fact, not guessed.