Agent evaluation · 2026 · Retrospective benchmark study
Predicting whether an AI agent succeeded
Using 4,448 benchmark runs, I tested whether summaries of an agent’s actions helped predict task success. They added no reliable benefit over ordinary interaction records when both the tasks and agent models were new to the predictor.
The question
Does knowing how an agent changed state help predict whether it completed the task?
What I worked on
I tested 14 generic features of tool use, including repeated writes, failed writes, confirmation, and reads after writes. The study compares these features with an ordinary interaction trace and a hashed tool-name baseline. Task success comes from the benchmark’s released reward, independently of the proposed features.
The cohort contains 4,448 public τ-bench trajectories across 278 tasks, four agents, and three domains. Each primary prediction holds out both the task fold and the agent model. Domains remain represented in training; this is not a test on unseen domains.
What I found
The generic semantic summary did not reliably improve success prediction. The interaction trace reached a Brier score of 0.1312; adding semantic features raised it to 0.1369. Lower is better. The paired difference was +0.0057, with a 95% task-cluster bootstrap interval of −0.0002 to +0.0117.
The 6 September revision withdraws the secondary recalibration analysis and its figure because calibration inputs could indirectly depend on outer-test labels. The uncalibrated comparison above is retained.
What this does and does not tell us
These are released benchmark interactions with a simulated user, not deployed customer outcomes. The result applies to this representation and cohort; it does not establish that external-state evidence is generally unhelpful.
Feature design was exploratory rather than preregistered. The lexical classifier can misread tool behavior, and a later read does not establish that the correct entity changed. Task-cluster uncertainty does not cover new domains, benchmark revisions, or model reruns.
The next experiment should test independently authored state contracts: which entity may change, what the final values must be, and which side effects are forbidden. That proposal is a follow-up to this negative result, not a capability demonstrated by it.