Practical AI/Brief

New pipeline generates synthetic enterprise QA datasets with up to 25 employees

A new arXiv paper introduces WinSyn, an automated pipeline for creating synthetic enterprise question-answering datasets that simulate long-running projects. It aims to address the limited complexity of existing benchmarks.

BriefPublished 14 September 20261 min read1 linked source · 4 checked facts

supercomputer cluster of compute racks in a NASA machine room — Illustration: Discover Supercomputer (GSFC) · NASA/Goddard Space Flight Center · PD licence

Existing benchmarks for enterprise question-answering often have limited real-world complexity, short-form responses, and unnatural queries, according to the paper. The new pipeline generates synthetic emails reflecting realistic workplace scenarios, along with long- and short-form questions and gold answers grounded in the data. It simulates projects spanning several months and involving up to 25 interacting employees across multiple roles, emphasizing ambiguity and distributed information. When evaluated with standard agentic baselines using frontier models, aggregate scores remained below 80% for each dataset.

What the reporting says

Each line below is stated by the page named above it, and only by that page.

Stated by arXiv

The pipeline generates synthetic datasets of emails reflecting realistic workplace scenarios, along with long- and short-form questions and gold answers grounded in the data.
The method simulates long-running enterprise projects spanning several months and involving up to 25 interacting employees across multiple roles.
Aggregate scores averaged over all queries remain below 80% for each dataset when evaluating standard agentic baselines with frontier models.
Existing benchmarks typically have limited real-world complexity, short-form responses, and unnatural queries.

Our view

This pipeline could provide a more realistic stress test for enterprise AI agents, revealing gaps that simpler benchmarks might miss.

What the reporting says: Aggregate scores averaged over all queries remain below 80% for each dataset, indicating significant room for improvement.

Sources

arXivText stored 14 September 2026 · checked 14 September 2026

Found through monitored feeds: arXiv. A feed address is how the desk finds a story, not the evidence for it — the sources listed above are the pages the facts were taken from.

How this story was checked

This story was written from the 1 page listed above, read by the collector on 14 September 2026 and stored.
Each line under What the reporting says was checked against that stored text on 14 September 2026. The check reads the stored text; no claim rests on a fresh look that did not happen.
Of 4 reported statements recorded for this story, 4 were confirmed against the page that carries them and are shown above; 0 could not be confirmed and were removed rather than published.
Some of the source text is the publisher's own feed summary rather than the full page, and the sending desk says so where it matters.
Figures in the text were required to appear in the stored source text before publication: yes. Identifiers: yes.
Where the reporting was silent, the text says so instead of filling the gap.

New pipeline generates synthetic enterprise QA datasets with up to 25 employees

What the reporting says

Our view

Sources

More from Practical AI

New method explains self-driving car AI decisions in real time

Autonomous AI Agents: What They Can Do and How to Manage the Risks

ChatGPT ads now open a conversation instead of a website

How this story was checked