# Generic semantic trace summaries do not improve transport

The released cohort contains 4,448 trajectories from four agents, 278 tasks,
three state-changing domains, and four trials per agent-task pair.
<!-- claim:cohort-composition -->

Tau-bench evaluates customer-service agents against task-specific database and
policy outcomes. This study uses those released outcomes as labels, but derives
features only from message and tool-call traces.
<!-- claim:independent-outcome -->

Every prediction holds out both its complete task fold and its agent model.
All repeated trials for a task remain in the same fold and uncertainty cluster.
<!-- claim:leakage-control -->

The interaction trace reaches Brier 0.1312. Adding the compact semantic summary
reaches 0.1369, a paired difference of +0.0057 with a task-cluster bootstrap
95% interval from -0.0002 to +0.0117; positive is worse.
<!-- claim:primary-negative -->

After removing 56 released trajectories with missing numeric rewards, the
semantic-minus-trace difference is +0.0010 (-0.0045 to +0.0065).
<!-- claim:missing-reward-sensitivity -->

Every secondary calibrator excludes both the held agent model and held task
fold. The audit contains 64 fits and preserves one prediction per released
trajectory, model, method, and cross-fit block.
<!-- claim:nested-calibration-guard -->

The frozen comparison retains its primary interval, -0.0002 to +0.0117, from
seed 20260803. Nested recalibration narrows semantic-minus-trace Brier from
+0.0057 to +0.0033, with a secondary task-cluster interval from -0.0001 to
+0.0066. It worsens absolute Brier for both representations and does not
reverse the primary null.
<!-- claim:calibration-does-not-reverse -->

Across the exhaustive 12 model-domain cells, semantic features have lower
recalibrated Brier in six. Seven cell intervals exclude zero: three favor the
semantic representation and four favor the ordinary trace. The global null
therefore masks opposed transport rather than a general semantic gain.
<!-- claim:opposed-transport -->

The supported conclusion is narrow: this generic, task-agnostic semantic
summary does not show transportable predictive value in the studied cohort.
A task-conditioned contract may still be useful, but it requires an evaluator
authored independently of the run and independent adjudication.
<!-- claim:bounded-conclusion -->

Limitations include retrospective model selection, three benchmark domains,
an exploratory lexical state-change classifier, benchmark rather than
production outcomes, and uncertainty conditional on the released trajectories.
