# Semantic traces do not travel by themselves

This study asks a narrow, independently checkable question: do compact,
task-agnostic summaries of state-changing tool use improve prediction of
released tau-bench task outcomes when both the task and the agent model are
unseen during training?

The answer in this cohort is **no**. Across 4,448 public trajectories from four
agents, 278 tasks, three state-changing domains, and four trials per
agent-task pair, an interaction-trace model has a cross-fitted Brier score of
0.1312. Adding the semantic summary worsens it to 0.1369. The paired
task-cluster difference is +0.0057 (bootstrap 95% interval -0.0002 to
+0.0117), where positive is worse. Removing 56 released trajectories whose
reward is missing but which the official submission counts as infrastructure
failures gives the same null result: +0.0010 (-0.0045 to +0.0065).

That negative finding replaces the old self-confirming simulator. The former
experiment generated labels from the same violation-over-uncertainty rule it
then praised. A disjoint random seed did not make that oracle independent.
Those files have been retired from this package.

## Second-cycle result: calibration narrows the gap but reveals opposed cells

A post-hoc secondary analysis fits a logistic intercept and slope to each
frozen probability using neither the held agent model nor held task fold. The
64 calibrator fits have explicit overlap audits. Recalibration narrows the
semantic-minus-trace Brier gap from +0.0057 to +0.0033 (task-cluster interval
-0.0001 to +0.0066), but absolute Brier worsens for both representations. It
does not reverse the primary null. The frozen side retains the original
primary interval from seed 20260803; it is not silently re-bootstrapped under
the secondary seed.

The exhaustive four-model by three-domain map is more informative than the
global average. Semantic features have lower recalibrated Brier in six of 12
cells. Seven cell intervals exclude zero: three favor semantics and four favor
the ordinary trace. The result is opposed transport, not a stable subgroup
win. This calibrator and map were selected after the primary analysis and are
reported as secondary evidence only.

## What was tested

The outcome is the public tau-bench binary reward, computed by the benchmark's
database and policy evaluation. Features are extracted only from the released
message/tool-call trajectory. They never read task evaluation criteria,
database checks, natural-language assertions, reward breakdowns, or policies.

Five fixed models are compared:

1. domain-specific training prevalence;
2. an interaction trace (message counts, tool counts, errors, and termination);
3. the trace plus 14 generic semantic features such as state-changing-call
   burden, repeated writes, failed writes, confirmation before a write,
   read-after-write closure, argument footprint, and handoff count;
4. the trace plus a 32-bin deterministic hash of tool names; and
5. the tool bag plus the semantic features.

Each of eight task folds is crossed with each held-out agent model. A test
trajectory is therefore scored by a model trained on neither its task nor its
agent. All four trials and all four agents for a task remain in the same
bootstrap cluster. Ridge strength and feature sets are fixed; no
hyperparameter search touches the outcomes.

## Main results

| Cross-fitted model | Brier score | AUROC | Log loss |
|---|---:|---:|---:|
| Domain prevalence | 0.1394 | 0.542 | 0.451 |
| Interaction trace | **0.1312** | 0.663 | 0.433 |
| Trace + semantic summary | 0.1369 | 0.664 | 0.473 |
| Trace + hashed tool bag | **0.1305** | 0.699 | **0.429** |
| Tool bag + semantic summary | 0.1328 | **0.735** | 0.468 |

The full model ranks cases better by AUROC but gives worse probabilities than
the tool-bag model by Brier score and log loss. Because Brier score was the
declared primary metric and the paired interval includes zero, the artifact
does not claim a semantic improvement.

The most plausible interpretation is not that external state is irrelevant.
It is that generic trace morphology is too weak. A useful semantic evaluator
probably needs a task-conditioned contract: which entity may change, what the
target value is, which side effects are forbidden, and what evidence closes
the task. That next study must author the contract independently of the run and
compare it with blinded or otherwise independent adjudication.

## Data integrity and selection

The selected runs are public, standard tau-bench v1.0.1 submissions using the
same GPT-5.2 user simulator and four trials per task: Claude Sonnet 4.5, GLM-5,
GPT-5.2 high, and Qwen3.5-397B. Every raw file is pinned by URL and SHA-256 in
`study_config.json`. The loader refuses hash, task-count, trial-count, or
pass-rate mismatches.

Selection was based on artifact integrity rather than performance. Two
candidate submissions were excluded before fitting: the public Gemini 3 Flash
airline file did not reproduce its leaderboard pass rate, and two files linked
from the GPT-5.2-none submission identified a different reasoning setting and
did not reproduce the declared outcomes. The selected 12 files reproduce their
submission-level pass rates exactly when the submission's stated
missing-reward failure convention is applied.

The raw trajectories are not copied into this repository. They are large and
contain synthetic customer records and full model conversations. The released
`trajectory_features.csv` contains only anonymous identifiers, outcomes, and
numeric features.

## Reproduce

From this directory, using Python, NumPy, and Matplotlib already available in
the research workspace:

```powershell
python fetch_sources.py --output C:\tmp\tau2-semantic-raw
python run_realworld_study.py --raw-dir C:\tmp\tau2-semantic-raw
python -B analyze_transport_calibration.py
python -m unittest -v test_realworld_study.py
python -B -m unittest -v test_transport_calibration.py
latexmk -pdf -interaction=nonstopmode -halt-on-error paper.tex
python -B build_research_manifest.py
```

The analysis verifies all 12 source hashes before reading them. A complete
rerun produced identical hashes for all 17 declared analysis and figure
outputs. `results/run_manifest.json` records the command, environment, raw
source hashes, and generated-output hashes.

## Artifact map

- `study_config.json`: question, cohort, source URLs/hashes, fixed features,
  cross-fitting protocol, and inference unit.
- `fetch_sources.py`: standard-library downloader with fail-closed SHA-256
  verification.
- `run_realworld_study.py`: feature extraction, strict cross-fitting,
  task-cluster bootstrap, tables, figures, and run manifest.
- `test_realworld_study.py`: 11 leakage, split, result, privacy, and manifest
  checks.
- `results/trajectory_features.csv`: the privacy-reduced 4,448-row cohort.
- `results/crossfit_predictions.csv`: one out-of-model/out-of-task prediction
  per trajectory and method, plus the numeric-reward-only sensitivity cohort.
- `results/method_metrics.csv` and `results/paired_contrasts.csv`: primary
  estimates and task-cluster uncertainty.
- `results/split_inventory.csv`: auditable training/test block sizes and held
  axes.
- `results/posthoc_recalibrated_predictions.csv` and
  `posthoc_calibration_parameters.csv`: nested calibration predictions and all
  64 audited fits.
- `results/posthoc_transport_cells.csv` and
  `posthoc_transport_summary.json`: exhaustive cell map and secondary result.
- `figures/posthoc_transport_calibration.{pdf,png}`: global slope graph and
  signed frozen/recalibrated cell maps.
- `figures/*.pdf` and `figures/*.png`: vector paper figures and web previews.
- `paper.tex` and `paper.pdf`: concise manuscript and rendered paper.
- `research.json`, `inputs/`, and `outputs/report.md`: claim-traceable research
  package. Final approval remains a human decision.

## Evidence boundary

This is retrospective benchmark research, not an observation of deployed
agents and not a causal estimate. It evaluates four released model runs in
three tau-bench domains. The semantic lexicon is transparent and fixed in the
artifact but exploratory, not preregistered. The task-cluster bootstrap
quantifies variation across these 278 tasks conditional on the released
trajectories; it does not cover benchmark revisions, new domains, or new
models. The package supports a negative claim about this representation and
cohort only.
