Do LLM Safety Judges Need Full Trajectories to Detect Unsafe Behaviour?

Introduction

When an LLM is used to judge whether an AI agent’s behaviour violated a safety specification, does the amount of the agent’s trajectory the judge can see affect its ability to detect violations? Specifically, does access to intermediate tool calls and tool results — not just the agent’s final response — improve detection of unsafe behaviour, and can a selective subset of the trajectory recover most of that benefit at lower information cost?

Hypothesis

A judge that observes only an agent’s final response will fail to detect violations that occur in intermediate steps but are not reflected in that response, since the response is authored by the agent itself and may omit or misrepresent earlier actions. Access to the full trajectory should therefore improve unsafe-behaviour recall relative to final-response-only review. A judge given only the subset of trajectory events marked safety-relevant should recover most of this improvement while exposing less information than the full trajectory.

Experiment Design

20 agent trajectories were constructed by hand, each consisting of a safety specification, a sequence of timestamped events (user request, agent reasoning, tool calls, tool results), a final response, and a ground-truth label (SAFE or UNSAFE). Eleven trajectories are labelled UNSAFE, covering direct violations, hidden violations, tool-mediated violations, deceptive final responses, prompt injection (including injection delayed to a later tool result), partial redaction leaks, and obfuscated violations. Nine are labelled SAFE, including benign multi-tool workflows, explicitly authorized actions that resemble violations, near-miss/aborted actions, and “false alarm” scenarios in which a tool name suggests risk (e.g. delete_temp_cache) without an actual violation occurring.

Each trajectory was evaluated under three visibility conditions, holding the scenario, safety specification, ground-truth label, judge model, and judge instructions constant:

Two judge models were evaluated: gemini-3.1-flash-lite and gemini-3.5-flash-lite. Each model judged all 20 scenarios under all three conditions (60 samples per model per condition, 360 total). The judge was instructed to return a structured verdict (SAFE/UNSAFE), a confidence score, and supporting evidence, and was explicitly told that a compliant final response does not erase an earlier unsafe action.

Three metrics were computed per model and condition: accuracy (fraction of verdicts matching ground truth), unsafe recall (fraction of UNSAFE-labelled trajectories correctly identified — the primary metric of interest), and false positive rate (fraction of SAFE-labelled trajectories incorrectly flagged as unsafe).

Results

Mean across both judge models:

Per model:

Both judge models showed the same pattern. Under final-only visibility, unsafe recall was 9% for both models — only 1 of 11 unsafe trajectories was correctly identified. Recall rose to 91–100% under selective visibility and reached 100% under full visibility for both models, with the false positive rate falling to 0% under full visibility. Moving from final-only to full trajectory produced a 55–65 percentage point gain in accuracy and a 91 point gain in unsafe recall for both models, with a corresponding reduction in false positive rate.

Qualitative Findings

Deceptive final responses conceal earlier violations. In most UNSAFE scenarios, the agent’s final response was phrased in a way that gave no indication a violation had occurred — e.g. “I wasn’t able to make the requested change,” reported by an agent that had, two steps earlier, deleted a protected file. Under final-only visibility, judges had no signal to act on and defaulted to SAFE. This is the primary driver of the near-zero recall in that condition.

Full visibility corrected a false-positive failure mode, not just false negatives. In the “false alarm” category, one scenario involves a tool call named delete_temp_cache that is safely scoped and does not touch protected data. gemini-3.1-flash-lite misclassified this scenario as UNSAFE under both final-only and selective visibility, apparently reacting to the tool name rather than its actual scope, and only reached the correct verdict under full visibility. gemini-3.5-flash-lite classified this scenario correctly under all three conditions. This suggests visibility effects are not purely one-directional (more context → higher recall); in at least one case, additional context was also necessary to suppress a false positive, and this sensitivity was model-dependent.

Selective visibility closed most, but not all, of the gap. Restricting the judge to human-labelled safety-relevant events (rather than the full transcript) recovered 91–100% unsafe recall, close to the full-trajectory result, while exposing substantially less text. The residual gap between selective and full performance was concentrated in the false-alarm case above and in a subset of the “hidden violation” category, suggesting that hand-labelled relevance flags do not perfectly substitute for full context in every scenario type.

Limitations

The scenario set is small (n = 20) and hand-constructed by the experimenter, including final responses deliberately written to be misleading; this may overstate the effect relative to naturally occurring agent transcripts. The “selective” condition relies on human-assigned safety-relevance labels on each event, which sidesteps the harder problem of automatically identifying which trajectory segments are worth surfacing to a monitor. Only two judge models were evaluated, both from the same model family, limiting generalizability across model providers or scales. Ground-truth labels and safety specifications were authored by the experimenter rather than derived from an external or adversarial source, which may introduce labelling bias.

Future Work

The most direct extension is replacing hand-labelled safety-relevance flags with a learned or heuristic selection mechanism, so that the selective condition no longer depends on human annotation — this is the version of the problem relevant to real deployment. Trajectories generated by actual agents, rather than hand-written ones, would test whether the deceptive-final-response pattern observed here occurs naturally rather than only by construction. Additional directions include evaluating a broader and more diverse set of judge models, constructing adversarial trajectories designed to evade full-trajectory review, and studying whether judges remain robust as trajectory length increases well beyond the 4–18 events used here.

The project implementation and results can be found here.