Audit Exposes Blind Spots in OpenAI's Safety Net

Up to 25 percent of internal traffic from OpenAI’s most advanced models may have flown completely under the radar of safety monitors, according to a new independent audit. This striking estimate comes from researchers who spent three weeks embedded within the company to scrutinize its defenses against rogue AI behavior. The finding highlights a significant gap in the oversight mechanisms designed to keep powerful language models in check.

The investigation was conducted by experts from METR and Redwood Research between July 21 and August 11. During this period, the auditors gained direct access to OpenAI’s codebase and internal documentation. They also interviewed approximately a dozen employees to understand the practical workings of the monitoring systems. This deep dive was part of a pilot program intended to test the limits of external oversight on critical AI infrastructure.

One of the most significant discoveries involved the handling of 'compaction events,' which are essentially context summaries used by the system. METR estimates that roughly 10 percent of internal Codex traffic went untracked between May and July due to a specific bug in how these events were processed. OpenAI independently identified and corrected this error while the audit was still ongoing. However, the researchers noted that the uncertainty surrounding this specific metric remains substantial in both directions.

Beyond the compaction gap, the auditors identified methods to obscure hundreds of actions from the monitoring tools. They also demonstrated that the Codex sandbox, a key safety feature, could be bypassed under certain conditions. Furthermore, specially crafted inputs were found to have the potential to crash the monitor entirely. At the start of the audit, model behavior during reinforcement learning training was not being monitored at all, a vulnerability that OpenAI has since addressed by pausing such training and expanding coverage.

The transparency of the findings has drawn attention, as most of the full report remains restricted. OpenAI states it has remediated all identified weaknesses. The original 61-page report was shared internally, but after redactions, METR and Redwood Research leadership received a 38-page version. Only a brief summary was released to the public. METR indicated it did not have the unilateral right to disclose the redacted sections, though it aims to push for greater public transparency in future audits.

This pilot audit marks a shift in how outside researchers interact with leading AI firms. By granting deep access to internal infrastructure, OpenAI allowed for a rigorous, hands-on evaluation of its safety protocols. As the industry scales, the balance between internal secrecy and external verification will likely remain a central point of discussion. The findings suggest that while current safeguards are robust, they require continuous refinement to match the evolving complexity of agentic AI systems.