Science Explained/Brief

Study probes when LLM draft-verify-revise pipelines disagree on 'previous'

A preprint abstract reports that in draft-verify-revise LLM pipelines, a context-dependent expression such as 'previous' can be resolved differently by different stages, producing a deictic shift. The study used a synthetic dataset of 10 base examples in three conditions and tested six models across 21 reasoning effort configurations.

BriefPublished 14 September 20261 min read1 linked source · 5 checked facts

telescope domes of a large observatory at night — Illustration: Night time view of the ESO Very Large Telescope (eucys-0539) · ESO · CC BY 4.0 licence

Balanced accuracy ranged from 0.156, below chance, to near-perfect. GPT-5.2 rose from 0.156 without reasoning to 0.942 at its highest reasoning effort level, while Gemini 3 Pro stayed above 0.94 at every level. Gemini 3 Pro at low reasoning effort outscored GPT-5.2 at xhigh reasoning effort for roughly 5% of the cost per trial. When the meta-evaluator erred, it tended to rely on surface cues rather than operational reasoning. The abstract advises context engineers to make the intended referent explicit at each stage.

What the reporting says

Each line below is stated by the page named above it, and only by that page.

Stated by arXiv

Draft-verify-revise is a common LLM orchestration pattern for scaling inference-time compute.
The phenomenon was studied with a synthetic dataset of 10 base examples, each rendered in three conditions.
Six models from three providers were tested across 21 reasoning effort configurations using e-values for sequential testing.
Balanced accuracy ranged from 0.156, below chance, to near-perfect.
GPT-5.2 rose from 0.156 without reasoning to 0.942 at its highest reasoning effort level.

Our view

This is a controlled synthetic demonstration, so it shows a possible failure mode rather than a settled rate for real-world pipelines.

What the reporting says: Balanced accuracy ranged from 0.156, below chance, to near-perfect, and that GPT-5.2 rose from 0.156 without reasoning to 0.942 at its highest reasoning effort level.

Sources

arXivText stored 14 September 2026 · checked 14 September 2026

Found through monitored feeds: arXiv. A feed address is how the desk finds a story, not the evidence for it — the sources listed above are the pages the facts were taken from.

How this story was checked

This story was written from the 1 page listed above, read by the collector on 14 September 2026 and stored.
Each line under What the reporting says was checked against that stored text on 14 September 2026. The check reads the stored text; no claim rests on a fresh look that did not happen.
Of 5 reported statements recorded for this story, 5 were confirmed against the page that carries them and are shown above; 0 could not be confirmed and were removed rather than published.
Some of the source text is the publisher's own feed summary rather than the full page, and the sending desk says so where it matters.
Figures in the text were required to appear in the stored source text before publication: yes. Identifiers: yes.
Where the reporting was silent, the text says so instead of filling the gap.

Study probes when LLM draft-verify-revise pipelines disagree on 'previous'

What the reporting says

Our view

Sources

More from Science Explained

Vendor-Native Coding Harnesses Show No Clear Average Advantage in Paired Test

Preprint proposes intent-driven privacy filtering for LLM prompts

Preprint proposes audit protocol for AI update admission, tested only in simulation

How this story was checked