Science Explained/Brief

Vendor-Native Coding Harnesses Show No Clear Average Advantage in Paired Test

A new arXiv paper compares agentic coding harnesses paired with the same models on a private, contamination-controlled suite. It reports no resolved average advantage for either harness, with opposite results across task types and unresolved cost ordering.

BriefPublished 14 September 20261 min read1 linked source · 8 checked facts

telescope domes of a large observatory at night — Illustration: Night time view of the ESO Very Large Telescope (eucys-0539) · ESO · CC BY 4.0 licence

The study ran 80 tasks under claude-agent-sdk and deepagents on claude-opus-4-8, and under openai-codex SDK and deepagents on gpt-5.5. 792 of 800 planned runs were graded by an isolated oracle. For Opus 4.8, the average difference was -1.25 percentage points (48.8% vs 50.0%, 95% CI [-10.0, +7.5]); for GPT-5.5, +1.25 pp (55.6% vs 54.4%, CI [-4.4, +6.9]). The Opus average hid opposite strata: native trailed by 9.0 pp on 61 repository tasks and led by 23.7 pp on 19 contest tasks, a partition chosen after seeing the data. Cost per solved task favored the neutral harness in observed usage, but missing usage records leave the billed ordering unresolved.

What the reporting says

Each line below is stated by the page named above it, and only by that page.

Stated by arXiv

The same 80 tasks ran under claude-agent-sdk and under deepagents on claude-opus-4-8, and under the openai-codex SDK and deepagents on gpt-5.5.
792 of 800 planned runs were graded by an isolated oracle.
Neither contrast resolves an average advantage for either harness: -1.25 pp for Opus 4.8 (48.8% vs 50.0%, task-bootstrap 95% CI [-10.0, +7.5]) and +1.25 pp for GPT-5.5 (55.6% vs 54.4%, CI [-4.4, +6.9]).
The Opus average combines opposite strata: the native harness trails by 9.0 pp on the 61 repository tasks and leads by 23.7 pp on the 19 contest tasks (label-permutation p = 0.003).
The partition was chosen after seeing the data and needs a designed replication.
Re-priced from raw per-turn usage at frozen list prices, the neutral harness cost 1.3 to 1.6 times as much per solved task on Opus 4.8 and 1.2 times on GPT-5.5.
On the Anthropic account 58 runs left no usage record, and allocating that spend to either cell would move the Opus ratio between 0.7 and 2.3, so the billed ordering is unresolved.
This revision corrects an August 2026 manuscript whose cost figures rested on a usage-semantics defect in our own telemetry (Section 5.1).

Our view

The result undercuts a simple vendor-native advantage story, but the task-type split and cost uncertainty mean it is not a settled verdict.

What the reporting says: Neither same-model harness contrast resolves an average advantage, with CIs crossing zero, and that the Opus average combines opposite strata whose partition was chosen after seeing the data and needs a designed replication.

Sources

arXivText stored 14 September 2026 · checked 14 September 2026

Found through monitored feeds: arXiv. A feed address is how the desk finds a story, not the evidence for it — the sources listed above are the pages the facts were taken from.

How this story was checked

This story was written from the 1 page listed above, read by the collector on 14 September 2026 and stored.
Each line under What the reporting says was checked against that stored text on 14 September 2026. The check reads the stored text; no claim rests on a fresh look that did not happen.
Of 8 reported statements recorded for this story, 8 were confirmed against the page that carries them and are shown above; 0 could not be confirmed and were removed rather than published.
Some of the source text is the publisher's own feed summary rather than the full page, and the sending desk says so where it matters.
Figures in the text were required to appear in the stored source text before publication: yes. Identifiers: yes.
Where the reporting was silent, the text says so instead of filling the gap.

Vendor-Native Coding Harnesses Show No Clear Average Advantage in Paired Test

What the reporting says

Our view

Sources

Related reading

More from Science Explained

Preprint proposes intent-driven privacy filtering for LLM prompts

Preprint proposes audit protocol for AI update admission, tested only in simulation

Clinical calculator AI: code solver lifts 32B model, not 7B, preprint finds

How this story was checked

What the reporting says

Our view

Sources

Related reading

Harness choice shows no resolved average task advantage in paired agentic coding tests

More from Science Explained

Preprint proposes intent-driven privacy filtering for LLM prompts

Preprint proposes audit protocol for AI update admission, tested only in simulation

Clinical calculator AI: code solver lifts 32B model, not 7B, preprint finds

How this story was checked