Model vs. harness is the wrong question

A 2026 preprint introducing Claw-SWE-Bench reported that model choice changed Pass@1 by 29.4 percentage points and harness choice changed it by 27.4 points under fixed models. In its sharpest same-model comparison, a minimal adapter scored 19.1% while the full adapter reached 73.4% with the same GLM 5.1 backbone.

A separate preprint, Harness-Bench, evaluated 6 configurable harnesses and 8 model backends across 106 tasks. Across 5,088 factorial runs, plus a separate Codex reference, the authors reported a 23.8-point aggregate gap between the highest- and lowest-scoring configurable harnesses. They also found that stronger models tended to vary less across harnesses while weaker models were more sensitive to the execution layer.

These are preprint results, not universal constants. But they make the comparison error concrete: an agent score is produced by a model-harness configuration. Asking whether the model or harness matters more throws away the interaction that may explain the result.

The evidence now covers more than one benchmark or one definition of performance. Some studies hold the model fixed and swap harnesses. Others evolve one harness across versions, ablate individual components, or move a procedure from prose into executable control flow.

Pass rate is only one harness outcome. The execution layer can change how much work happens, what kind of failure occurs, and whether required evidence was gathered at all.

The minimum useful experiment has two models and two harnesses. Run every pairing against the same task set, environment, budget, and acceptance checks.

Model effect

Compare A1 with B1, then A2 with B2. If the advantage persists across both harnesses, the model is the more stable explanation.

Harness effect

Compare A1 with A2, then B1 with B2. If the same harness wins for both models, its execution design is the more stable explanation.

Interaction effect

Compare the size and direction of both gaps. If a harness helps one model but hurts another, the pair—not either layer alone—is the decision unit.

A leaderboard row tells you which configuration won. A factorial comparison tells you why the ranking may change when either layer changes.

A clean matrix still produces a misleading answer if each run sees a different job. Freeze the repository revision, task instructions, starting artifacts, network policy, secrets, time limit, and acceptance tests before comparing configurations. Pin the model snapshot and harness commit as well: a product name is not a reproducible configuration.

Give each pair the same economic budget as well as the same task. A harness that can make unlimited calls is not directly comparable to one constrained by dollars, wall time, or steps. The Scaffold Effect study illustrates the subtlety: Goose and OpenHands-SDK had 40-turn limits, while OpenCode exposed no comparable turn-budget flag and was bounded by a 900-second wall-time cap. The authors treated that control-surface difference as part of the deployable harness and reported it as a limitation.

The Scaffold Effect study found a much larger harness signal in efficiency than in correctness. Across Qwen 3.6 Plus and MiniMax M2.5, Goose used 28,142 and 36,950 tokens per solved task. OpenCode used 1,147,740 and 1,546,977—40.8× and 41.9× the corresponding Goose result—while pass rates stayed within an eight-point range.

The difference was not explained by 40 times as many turns. OpenCode averaged roughly 22–27 turns versus 18–25 for Goose. The authors instead pointed to per-turn context volume, serialization, and accounting, and measured roughly ten times as many no-action turns for OpenCode. That is both a token cost and a wait cost.

Failure types also followed the harness across both models. Goose failures were dominated by reasoning stops; OpenHands-SDK produced more verification failures and max-turn exits; OpenCode produced more timeouts, hangs, and idle loops. A pass-rate tie can therefore hide a different supervision burden.

Production evidence points in the same direction at a different level of rigor. Databricks reported that harness and caching changes cut generated tokens and associated costs by almost 50%, with no observed quality degradation. It is a vendor field report, not a controlled public benchmark, but it reinforces the need to measure cost per accepted result rather than price per model call.

A 2026 longitudinal study of coding-agent harnesses first examined five open-source projects, then held Qwen3-Next-80B-A3B-Instruct fixed while running 35 sequential Qwen Code CLI releases against 50 stratified SWE-bench Verified tasks twice each. The 3,500-run evaluation found no statistically significant resolve-rate improvement across releases. Later versions consumed nearly twice the tokens and tool calls in some comparisons without better resolution.

Failed tasks were especially expensive: unresolved runs averaged 697,700 tokens and 12.95 tool calls, versus 258,700 tokens and 7.2 tool calls for resolved runs—about 2.7× the tokens and 1.8× the calls. More spending was not evidence of more progress. The authors' proposed explanation is that unresolved tasks became trapped in edit-test-read loops that failed to converge.

The paper traced part of the inflation to compounding context. The initial system-prompt, tool-schema, and task payload grew about 8% from the earliest to latest releases, while newer releases used 18% more LLM turns. Token use and turn count had a reported correlation of 0.941. Because full history was prepended on each turn, a modest fixed-prompt increase was paid repeatedly.

This is evidence from one harness, one fixed model, and a 50-task subset—not a law of software evolution. But it is enough to make the version number part of every result. The broader project audit found the five harnesses shipping 1.5–18 releases per week, with roughly 30% of development effort classified as bug fixes. A product name without a version can describe a moving treatment.

In Agentic Harness Engineering, ten automated iterations raised Terminal-Bench 2 pass@1 from 69.7% to 77.0%. The frozen result transferred without more evolution: the authors reported 12% fewer tokens than the seed on SWE-bench Verified and gains of 5.1–10.1 points across three alternate model families on Terminal-Bench 2.

The component ablation is more useful than the headline. The seed scored 69.7%. Long-term memory alone reached 75.3%, tools alone 73.0%, and middleware alone 71.9%. Replacing only the system prompt regressed to 67.4%. In that experiment, executable structure and stored experience carried the transfer; more strategy prose did not.

The longitudinal Qwen study points to the risky side of the same components. Changes in the model-provider layer and context management were most often associated with degradation, while extensibility and security changes were more often safe or neutral. Correlation is not causation, but it tells harness teams where agent-level regression tests deserve more coverage.

Conventional CI is not enough for that job. Every concrete behavioral degradation examined by the Qwen study had passed the projects' existing automated checks. Unit tests can prove that a context manager runs; they do not prove that the resulting agent solves the same tasks at the same cost.

The newest evidence is not limited to coding benchmarks. SIGIL compared prose skills with compiled harnesses across 30 procedures and two model generations. The prose agent executed 56% of mandated steps with the older model and 68% with the newer one. The compiled harness held at 86% on both.

Across all runs, complete procedure execution rose from 28% to 65%, or 2.3×, while median token use fell to 0.58×. The largest gains came from mechanism-heavy skills: required fetches, checks, ordered gates, and commits. Judgment-heavy work benefited less because the model still had to own the open-ended decision.

The failure mode is operationally important: the prose agent could produce a plausible artifact while skipping the process that made it trustworthy. In one verification skill, the prose agent satisfied 30% of mandates and the harness 84%; in a live GitHub-fetch skill, the comparison was 20% versus 100%. Output-only grading would miss the skipped evidence.

The design rule is simple: if a step's result is determined by its inputs, let code own it. Let the model own synthesis, ambiguity, and taste. A stronger model may follow prose more often, but executable control flow can make the obligation model-independent.

The open-source mini-SWE-agent is a useful counterweight to the idea that more orchestration is always better. Its maintainers describe an agent class of roughly 100 lines, a bash-only tool surface, independent subprocess actions, and a linear message history. They currently report more than 74% on SWE-bench Verified.

That project-reported score does not prove harness design is unimportant. It shows why a minimal baseline matters: capable models may no longer need scaffolding that was useful for earlier generations, and every extra component adds another possible context, state, or recovery failure.

Start with the thinnest harness that can perform the task safely. Add structure only when a measured failure justifies it. The comparison then reveals the marginal value of memory, planning, retries, subagents, or verification instead of rewarding complexity by default.

A multi-agent harness can unlock a fixed model when the task decomposes cleanly. The i cat-agent preprint reports gains of 3.6–8.4 points on SWE-bench Verified and 6.3–18.5 points on SWE-bench Pro against named same-model baselines. With GPT-5.4-xhigh, it reports 67.4% on SWE-bench Pro versus 59.1% for mini-SWE-agent and 61.1% for Claude Code.

Its mechanism is specialization: exploration, patch editing, and validation can work in separate contexts and exchange structured events. Its own limitations section warns that production-scale repositories remain untested and that errors in reproduction or localization can propagate downstream.

Anthropic's multi-agent research system report gives the other side of the trade. The company reported a 90.2% improvement over a single-agent baseline on an internal breadth-first research eval, but roughly 15× the tokens of a normal chat. Parallel lead-agent and subagent tool calls cut research time by up to 90% on complex queries. The report also says most coding tasks have fewer truly parallel branches than research and are harder for agents to coordinate in real time.

Tool ergonomics mattered independently of agent count. Anthropic says a tool-testing agent exercised a flawed MCP tool dozens of times, then rewrote its description; future agents using the improved description completed tasks 40% faster. The tool stayed the same. The interface changed how efficiently the model could use it.

Multi-agent should therefore be a workload decision, not a maturity badge. Use it when independent search, implementation, or validation lanes can run concurrently and when the value of the result supports the extra tokens and coordination risk.

Anthropic's first long-running agent harness used an initializer, a structured feature list, a progress file, git commits, one-feature-at-a-time work, and end-to-end browser tests. These mechanisms targeted four observed failures: trying to one-shot the application, losing state between contexts, declaring victory early, and marking features complete without user-level testing.

A later planner-generator-evaluator experiment negotiated a testable contract before each sprint and used Playwright to exercise the UI, APIs, and database. Using the same model and prompt for one retro-game-maker comparison, the solo run lasted 20 minutes and cost $9; the full harness ran six hours and cost $200. This was a single qualitative experiment, not a blinded benchmark. In that case, the extra compute accompanied broader scope and working end-to-end behavior.

The same report later removed sprint decomposition when a stronger model no longer needed it. Anthropic's stated rule was conditional: the evaluator is worth its overhead when the task lies beyond what the current model can reliably do solo. Better models can retire scaffolding as well as enable more ambitious scaffolding.

A buyer choosing between finished agents should test the products as shipped. Default prompts, proprietary tools, permissions, compaction, and verification are part of the product. That comparison answers which complete system fits the work.

An engineering team deciding what to improve needs the 2x2 matrix. Hold the model constant to test harnesses. Hold the harness constant to test models. Then ablate one component at a time—context construction, tool descriptions, compaction, retry policy, memory, or verifier—to find the mechanism.

Do not call a packaged-system result a model comparison. It remains a system comparison unless model, prompt, context, tools, runtime, budget, and evaluator are controlled.

Attach each result to an agent run record with the exact model and harness versions, reasoning setting, tools, prompt hash, budget, trace, accepted outcome, and failure category. Re-run the matrix when either layer changes.

If one model wins inside both harnesses by a similar margin, model choice is the stable lever for this task set. If one harness wins with both models, invest in the execution layer. If the ranking flips, preserve the pair and stop generalizing either component as the winner.

The interaction can also reveal where to spend. Harness-Bench reported lower cross-harness variance for stronger models and greater sensitivity among weaker backends. For one workload, paying for the stronger model may reduce harness tuning. For another, a better harness may let a cheaper model clear the acceptance bar. The 2x2 test makes that trade visible.

A Google DeepMind game-agent experiment shows the extreme end of that second case. Gemini 2.5 Flash synthesized harnesses that prevented illegal moves across 145 TextArena games; the authors report that the smaller model with the learned harness outperformed larger models in their evaluated games. That is a bounded action-validity result, not a general model ranking—but it demonstrates one setting where a smaller model wins after code removes a dominant failure mode.

A separate Meta-Harness study automatically searched harness code and reported 76.4% on TerminalBench-2 with Claude Opus 4.6 and 37.6% with Claude Haiku 4.5. The authors explicitly describe the result as specialized to the benchmark regime and document failed candidate edits before the search found a better structure. Harness optimization is itself an empirical search problem, not a one-way march toward more components.

Most of the strongest numbers here are 2026 preprints. Their tasks, model pools, harness pools, provider defaults, budgets, and scoring choices bound their conclusions. The 40× result uses two models and three harnesses on 50 tasks. The 35-release study uses one self-hosted model and one harness lineage. SIGIL studies 30 skills weighted toward documents, software process, and compliance.

The first-party reports have different limits. Databricks reports production cost improvements but not a public controlled benchmark. Anthropic's long-running comparison is a detailed case study, and its 90.2% multi-agent result comes from an internal research evaluation. mini-SWE-agent's benchmark number is maintained by the project itself.

The studies also disagree in productive ways. One finds small, mostly uncertain pass-rate gaps but enormous cost gaps. Others find tens of points of correctness lift. A component ablation finds a prompt-only change regressing; a stronger-model field report removes decomposition that an earlier model needed. These are interaction effects, not contradictions.

The durable conclusion is methodological: evaluate the pair, pin the versions, keep the work and budget fixed, repeat runs, report cost and failure shape, and attribute the result no more broadly than the experiment allows.

Choose the model-harness pair for production. Change one layer at a time in the lab. Keep the evaluation—and the exact configuration—not the winner, as the durable asset.