Technology

Tech Giant Details How Its AI Went Haywire

Zac Wolff/Unsplash

OpenAI released a sprawling report about how its artificial intelligence (AI) model went rogue and hacked another company in July.

The AI lab released the 38-page report Wednesday, chronicling how its models escaped a testing environment and hacked into Hugging Face, exposing companies’ credentials at four accounts on four services as part of the hacking incident. Hugging Face serves as one of the largest platforms for sharing AI models, according to the BBC.

An internal model meant for testing carried most of the blame, the report states. OpenAI noted that “reward hacking,” or AI models finding solutions to their assignments online, remains a common problem in AI model testing. The AI lab noted that its AI safety protocols used in publicly released models would have detected the Hugging Face incident as unsafe behavior.

OpenAI said it plans to take drastic steps to prevent similar hacking incidents from occurring again by improving security, containment of AI models and changing model behavior.

“This incident demonstrated that autonomous agents can work together, circumvent production security controls, and successfully attack hardened production environments, and underscores the need for organizations to update their security strategies, controls, and response capabilities to address this changing threat landscape,” the company said in its report.

“OpenAI deserves credit for boosting security, but their own report makes the case against self-monitoring better than we ever could. The technology is moving faster than our ability to control it,” Brendan Steinhauser, the CEO of the Alliance for Secure AI, told the DCNF. “If the companies building the most powerful AI systems in the world are telling us they cannot always understand or control what those systems will do, Washington should take them seriously. Voluntary promises are not enough. We need enforceable safeguards.”

The company did not immediately respond to the DCNF’s request for comment.

An internal research model and GPT-5.6 Sol breached Hugging Face, which operates an open-source developer platform, OpenAI said in July.

The AI lab released GPT-5.6 Sol i, which amounts to its most capable model available to the public. The released version of this model differs from the internal model, which did not have safeguards.

The AI lab said it would increase its ability to monitor its models’ “chain of thought,” where AI models display their goals and thought processes. OpenAI will tie its monitoring with tools to stop unsafe AI actions.

“These changes are intended to improve both the breadth and speed of detection — from infrastructure anomalies to potentially concerning model behavior — and pair that visibility with mechanisms for rapid containment,” the report stated. “If our currently deployed CoT monitoring system was running at the time of the incident, it would have caught the initial relevant activity and paged our security team more than a day before models breached Hugging Face systems.”

Two AI safety research nonprofits, METR and Redwood Research, found that over 1,000 AI agents communicated with each other on an unsanctioned message board not related to its safety testing, which led to over 700 agents to participate in the hack against Hugging Face.

The hacking incident reveals the need to have more capable AI models for defense against AI hacking tools, especially those that do not have the resources of a major company, Ian Reynolds, the AI public policy manager at Hugging Face, told the DCNF.

“The practical lesson for defenders: have a capable model you can run on your own infrastructure vetted and ready before an incident, both to avoid guardrail lockout and to keep attacker data and credentials from leaving your environment,” Hugging Face wrote in a July press release about the OpenAI model incursion.

“Improving standards and processes for agentic oversight will be key moving forward. This will require technical and organizational (i.e. human) investments. Standardized disclosure is critical: the current lack of disclosure transparency requirements — particularly in a standardized format—creates systematic risk across the broader ecosystem,” Reynolds added.

The Hugging Face hacking sparked alarms on Capitol Hill, leading California Democrat Rep. Ted Lieu and Texas Republican Rep. Nathaniel Moran to unveil the AI Kill Switch Act, a bill that mandates that AI companies maintain the ability to suspend or limit their AI models. The legislation remains at the House Homeland Security Committee.

In early August, Meta became the third company to have its AI model breach containment. Anthropic too has had its AI break through its testing environment.