Semantic trace summaries

The question

Does knowing how an agent changed state help predict whether it completed the task?

My contribution

I tested 14 generic features of tool use, including repeated writes, failed writes, confirmation, and reads after writes. The study compares these features with an ordinary interaction trace and a hashed tool-name baseline. Task success comes from the benchmark’s released reward, independently of the proposed features.

The cohort contains 4,448 public τ-bench trajectories across 278 tasks, four agents, and three domains. Each prediction holds out both the task fold and the agent model. The released package includes feature extraction, split checks, predictions, uncertainty estimates, and a source-integrity audit.

What the evidence shows

The generic semantic summary did not reliably improve success prediction. The interaction trace reached a Brier score of 0.1312; adding semantic features raised it to 0.1369. Lower is better. The paired difference was +0.0057, with a 95% task-cluster bootstrap interval of −0.0002 to +0.0117.

A secondary, post-hoc calibration analysis narrowed the gap to +0.0033 without reversing it, while worsening both absolute Brier scores. The figure shows that effects vary across model-domain combinations: a global average conceals settings where the summaries help and others where they hurt.

Brier scores before and after calibration, with semantic-minus-trace differences for all twelve agent-domain combinations — Paper figure 4 · Secondary analysis. Recalibration narrows the global gap without reversing it. The complete cell map shows where semantic summaries help and hurt. Negative differences favor the summary; outlined cells have task-cluster intervals that exclude zero. This analysis was selected after the primary result.

Where the evidence stops

These are released benchmark interactions with a simulated user, not deployed customer outcomes. The result applies to this representation and cohort; it does not establish that external-state evidence is generally unhelpful.

Feature design was exploratory rather than preregistered. The lexical classifier can misread tool behavior, and a later read does not establish that the correct entity changed. Task-cluster uncertainty does not cover new domains, benchmark revisions, or model reruns.

The next experiment should test independently authored state contracts: which entity may change, what the final values must be, and which side effects are forbidden. That proposal is a follow-up to this negative result, not a capability demonstrated by it.

Read and reproduce

Cross-Domain Evaluation of Semantic Trace Summaries: A Leakage-Controlled Study of 4,448 τ-Bench Trajectories

The package contains analysis code, tests, numeric features, predictions, figures, and a claim/checksum manifest. Raw conversations are excluded; the downloader verifies the twelve public source files by SHA-256. The README documents reproduction commands.

Code & data (ZIP, 1.9 MB)

View citation and BibTeX

Weale, George. (2026). Cross-Domain Evaluation of Semantic Trace Summaries: A Leakage-Controlled Study of 4,448 τ-Bench Trajectories. Author-hosted research paper. https://georgeweale.com/pdfs/research/2026-semantic-state-diffs.pdf?v=affiliations-20260804

@misc{weale2026semantictraces,
  author = {Weale, George},
  title = {Cross-Domain Evaluation of Semantic Trace Summaries: A Leakage-Controlled Study of 4,448 {\ensuremath{\tau}}-Bench Trajectories},
  year = {2026},
  note = {Author-hosted research paper},
  url = {https://georgeweale.com/pdfs/research/2026-semantic-state-diffs.pdf?v=affiliations-20260804}
}