For 35 hours, one of our local coding-agent systems looked busy.
The model was generating at a healthy 121 tokens per second. The GPU was working. Several sessions were active, and the logs showed thousands of successful calls. Yet only 9 of 27 work packages had merged.
The explanation was hiding in a metric we had not known to watch. Up to seven coding sessions were taking turns on one inference slot. Every switch evicted the previous session’s prompt cache, forcing the model to read an entire, and continually growing, conversation again before producing its next token.
Across those concurrent sessions, the builder accumulated 150 hours of prompt processing and only 4.4 hours of code generation during 35 hours of wall-clock time. Cache hits fell from 99% with one session to zero with five or more. Time to the first generated token rose from roughly one second to as much as 216 seconds.
Nothing was wrong with the GPU, and the model was performing normally. We had orchestrated it incorrectly.
That failure captures our main lesson from building the OrangeLoops AI Lab: once a model is capable enough for a bounded role, useful performance depends on the system around it. Evaluation, tools, routing, context, queues, observability and safety controls become part of the product.
The model matters. It just was not the whole problem.
The real product was the harness
The industry increasingly calls this surrounding discipline harness engineering: designing the tools, context, execution loop, validation, routing, permissions and feedback that turn a language model into an agent.
We did not use that phrase when we started. In March, our work was local-inference engineering: bringing up hardware, comparing runtimes and benchmarking models. The agent systems, and most of the operational evidence in this article, came from an intensive build-and-run period in October. This is an eight-month engineering journey, not eight months of a mature platform in production.
The lab is made from hardware a small team can own:
- two AMD Strix Halo machines, one with 96 GB and one with 128 GB of unified memory, linked over Thunderbolt
- a consumer RTX 3090 PC that joins the lab when it is switched on
- a Mac mini for Apple builds
- Raspberry Pis running the LLM gateway, work queue and control plane
Since March, we have tried roughly 35 open-weight models across more than 60 recorded runs, spanning several generations of our benchmark. On top of them we built local and hybrid workflows: software builders and reviewers, information-feed judges, an interview engine, and operations agents.
Five lessons changed how we design those systems.
1. Evaluate the deployed system, not the model in the abstract
For about five months, the lab’s reasoning role ran on Qwen3-235B. It was the largest model we could serve, and choosing it felt conservative: more parameters should mean more capability.
Our workload evaluation said otherwise.
We compared it with Qwen3.8-Flash-Next on 40 tasks drawn from our actual work: operations diagnosis, tool use, coding and multi-step reasoning. Flash-Next scored 92.1%; the Qwen3-235B deployment scored 57.1%. The confidence interval for the difference was 23 to 47 percentage points. Flash-Next was also about 2.7 times faster and used roughly 30 GB less memory.
This does not prove that Flash-Next is generally more intelligent. It was a comparison of deployed systems. In our tool-driven interface, Qwen3-235B tended to submit an answer immediately, while Flash-Next produced several thousand tokens of working before answering. With different prompting, the larger model could solve some of the same tasks. What mattered was fitness under the interface we actually needed.
That distinction, general capability versus fitness for a role, is easy to miss on a leaderboard. We had already learned to distrust our own. An earlier benchmark ran unattended for months and produced a tidy ranking. Later we found that:
- three tasks scored zero in every run because the checker read a variable that did not exist
- nine tasks were too easy to separate the models
- reasoning models exhausted their output allowance, and their scratchpad was stored as the answer
- one model family was being judged by a model from that same family
After repairing the checkers, one incumbent’s score rose from roughly 77% to 97% without any change to the model. In another test, raising the output allowance moved a 122B model from last place to first.
The lesson was not simply “build a private benchmark.” A private benchmark can mislead you just as confidently as a public one. Evaluation code is production code. It needs known-good and known-bad answers, deterministic scoring where possible, repeated samples, exact deployment identities and tests for the scorer itself. Current practice calls these behavioral evaluations or agent evals: tests of the behavior the complete model-harness system must exhibit.
A surprising result is a bug report first, and a model insight only after the harness survives inspection.
2. Make models replaceable before choosing the “best” one
Once Flash-Next won, replacing the incumbent took 52 seconds and required no application changes.
Applications in the lab do not request a model by name. They request a purpose: decide, act, reason or fast GPU. An LLM gateway maps that purpose to a model, runtime and machine, and applies availability checks, priorities, context limits and explicit fallback rules.
This routing layer mattered more than getting the first model choice right.
Models change rapidly, but so do the runtimes beneath them. In our early Strix Halo experiments, Vulkan outperformed ROCm and was dramatically more reliable. Months later, a specialized mixture-of-experts runtime made the same model family roughly twice as fast as our initial deployment. A model that first required both Strix Halo machines over Thunderbolt later fit on one machine. Other optimizations ranged from no gain to 29%, depending on the build and workload.
The architectural question changed from “Which model should we standardize on?” to “How cheaply and safely can we replace a model when new evidence arrives?”
Routing by purpose keeps application contracts stable while model selection remains an operational policy. An on-demand GPU can announce whether it is awake. One workload may fall back; a code reviewer may not, because a weaker review is worse than no review. One project may permit cloud inference while another requires every role to remain local.
3. Schedule conversations, not requests
The coding-pipeline failure exposed the difference between serving chat requests and operating agents.
An agent carries a growing history: instructions, code, tool results, test output and previous decisions. Deciding what enters that history, what remains and how it is recovered is now commonly called context engineering.
With self-hosted models, context has a physical cost. Re-reading it consumes memory bandwidth and delays the first token. If several agents alternate on one slot, conventional request-level concurrency destroys the prompt cache that makes each conversation affordable.
In our failed run, one session maintained a 99% cache-hit share. With five or more sessions competing for the slot, it fell to zero. Adding more agents had produced more apparent parallelism and less useful work.
We changed the harness in two ways:
- Conversation parking. The inference server keeps several paused conversations in system memory. Restoring one takes roughly 20 milliseconds instead of reading tens or hundreds of thousands of tokens again.
- Bounded session pools. The orchestrator limits active sessions per model role. Three builders is a useful default for our current setup: generation uses the GPU, while tests and tools create room for another session to run. Beyond that, added concurrency mostly creates queueing and cache churn.
A later controlled run on a small project maintained a 99% cache-hit share. That validated the mechanism; the productivity comparison on a full release is still to come.
This changed what we monitor. Tokens per second is not an agent-throughput metric. A fast model can support a slow system. We also need time to first token, cache reuse by conversation, queue time, tool and test time, and, ultimately, completed work. Cache share and queue wait now have their own monitoring instead of hiding inside aggregate GPU utilization.
For local agents, the unit of scheduling is the conversation, not the HTTP request.
4. Route by role, data and work shape
Our coding pipeline began with a clean division of labor: Claude planned each release, a local model on the RTX 3090 built work packages, another local model on a Strix Halo reviewed them, and Claude performed a second review before merge.
The early evidence was encouraging. In one release, local models produced 85% of the output tokens. In a calibration experiment, the local reviewer caught a planted bug on the correct line and raised no issue on the clean version. Narrow server changes with a test contract could move from brief to merge in under an hour.
But output tokens are not delivered value, and the pipeline failed when we treated them as equivalent. Work packages estimated for a frontier model at 45 to 90 minutes sometimes took the local builder 2 to 13 hours. During a larger release, cloud agents completed roughly 35 items in about six hours while the local pod moved one through review.
Our own numbers rule out the claim that local models replace frontier models at equal productivity. We kept the local pipeline and changed the work around it. Local builders now receive narrower packages with a machine-checkable finish line. The cloud planner handles decomposition and can take over after repeated review failures. A subsequent hybrid release shipped 17 items with 1,383 tests green.
“Local or cloud?” is therefore not one decision. We route according to three project properties, with a fourth, how much human involvement a project wants, still being defined:
- Role. Planning, building, reviewing and judging have different capability and assurance needs.
- Data policy. A project may require local-only execution or permit selected cloud roles. Mixed routing is our practical default.
- Work shape. Local models perform best for us on narrow, verifiable tasks. Frontier models retain a large advantage on open-ended work and critical-path recovery.
Cloud systems have their own failure modes. In one paid experiment, a reasoning model consumed most of its output allowance thinking, returned no answer, and was still billed. Local inference replaces per-token billing with hardware, capacity and operational costs; it does not make the work free. The useful comparison is measured outcome against measured outcome, under the project’s data policy.
5. Safety has to live outside the prompt
Agents in the lab interact with real repositories and machines, so governance is part of the harness.
A prompt saying “do not modify files” is not a security boundary. We confirmed that when a supposedly read-only reviewer, restricted by a command allowlist, could still write through an allowed programming-language command. Read-only operations now rely on OS-level sandboxes, separate users, network restrictions and branch protection.
Agents may propose actions, but they do not approve their own actions. Routine, reversible operations can receive standing approval only when rollback and verification are defined first. Everything else requires a person or a deterministic gate.
We also assume that agent-written infrastructure can be plausible and wrong. One firewall rule looked correct and interrupted two services for about 11 minutes. On another occasion, the wrong model served for three days without producing an error. The durable fixes were a safer rule pattern, live verification and an hourly comparison between declared and actual state, not a stronger sentence in the system prompt.
These controls make the system safer by construction; they do not prove that it is safe. The boundary has to live somewhere the model cannot negotiate with it.
What this evidence does and does not support
The lab gives us operational evidence about our own workload. It is not an independent benchmark, a universal model recommendation or a total-cost-of-ownership study. We have not normalized hardware, power, maintenance and engineering time against equivalent cloud outcomes, so we do not claim that local inference is always cheaper. We have not presented a threat model or an adversarial security evaluation. Several policies, including per-project data and involvement settings, are still being adopted.
Those limitations define the decisions the evidence can support: which model-harness combination performs a lab role, where its bottlenecks are, and which controls are needed before granting it more responsibility.
What we would do differently
If we were starting again, we would spend less time searching for a permanent model winner and establish the surrounding contracts sooner:
- build evals from real work and test the scorers
- record the exact model, quantization, runtime and hardware for every result
- route by purpose rather than model name
- schedule conversations rather than requests, and measure cache share and completed work, not only generation speed
- size tasks for the model that will perform them
- declare local and cloud use per project and role
- design permissions, rollback and verification before granting autonomy
After eight months, our conclusion is not that one local model has finally become “good enough.” It is that useful agents emerge from the system around the model: the evidence that selects it, the gateway that can replace it, the scheduler that preserves its context, the work design that plays to its strengths, and the controls that limit its authority.
The model still matters. For some work, it remains the limiting factor. But a stronger model cannot repair a broken scorer, preserve an evicted cache, enforce a permission boundary or prove that the right work was completed.
That is the work of harness engineering, and it is the part we now know how to build.
