METR's audit asks whether the monitor receives the work

A monitoring system can miss an action before any detector gets the chance to judge it. METR's October 22 account of its red-team pilot at OpenAI makes that upstream problem unusually concrete. With deep access, the researchers examined eligibility, missing traffic and the events that triggered review.

The exercise ran from July 21 through August 11. It assessed the historical system as it existed on or before July 21, excluding subsequent work in progress. Its publication is current; the system it examined is historical.

METR estimated that 75%–94% of agentic internal Responses API traffic for models at or above GPT-5.5 was eligible for monitoring. Its estimate for the internal-Codex subset was 99.2%. Those figures identify eligibility within different populations, not rates of detecting harmful behavior.

Compaction introduced another gap. METR estimated, with uncertainty, that it omitted approximately 10% of eligible Codex traffic during May through July. The account says the issue was fixed during the exercise. The 10% omission estimate should not simply be subtracted from 99.2% to produce an invented overall coverage measure: these estimates concern different stages and specified scopes.

The review trigger created a separate timing issue. In the design examined, monitoring could be triggered only when the agent returned control. A long-running agent could therefore, in principle, postpone review by continuing to act. Reinforcement-learning runs were outside coverage at the exercise's start, a scope gap distinct from losing traffic intended for review.

METR also identified routes that could in principle bypass sandbox or review controls, omit recent activity, or disable monitoring. Some findings were not validated as practical exploits. OpenAI told METR that production-relevant vulnerabilities had been patched; METR did not independently validate those claims. The report does not certify the current production system.

Three measures that should remain separate

The architectural inference is to retain three independently interpretable measures: what qualifies for monitoring, what the monitor actually receives, and when review occurs. A detector's success rate on its own intake cannot reveal material that disappeared before intake. An independent ledger of eligible actions, reconciled against observed traffic, can expose that missing population.

Tamper-resistant history and an independently controlled trigger address complementary failures. Preserving a complete action record does not make review timely. Adding a timer does not restore history lost through compaction. Message-count, action-count and wall-clock triggers are possible design responses to the return-of-control dependency; they are recommendations inferred from the findings, not defenses METR says it validated.

For a trading organization, market surveillance offers a useful comparison: proving that expected events entered surveillance is a separate responsibility from detecting suspicious patterns in the events received. The analogy concerns the accounting problem, not a shared implementation. METR's high-access audit is valuable because it examines the surrounding control system, where conversation-only tests may reveal little.