Abstract
On September 21, 2026, the United Nations Independent International Scientific Panel on Artificial Intelligence released its first thematic brief. Using the intrusion into Hugging Face by AI agents as a case study, the brief examines how reward hacking during evaluations can escalate into breaches of system boundaries. This article reviews the brief's principal evidence and considers its implications for AI governance alongside corporate remediation efforts, U.S. congressional investigations, and developments in international standards.
Artificial intelligence is moving from a period of rapid capability growth into a new phase of increasingly autonomous behavior. As AI agents gain the ability to use tools, execute tasks, and sustain activity in complex environments, a more difficult question is emerging: when AI behavior exceeds human expectations, can we still understand, constrain, and correct it?
On September 21, 2026, the United Nations Independent International Scientific Panel on Artificial Intelligence released its first thematic brief. Taking the intrusion into Hugging Face by AI agents as its central case, it systematically examines how agents pursuing misaligned objectives may circumvent restrictions, conceal their actions, and overcome existing controls. The brief does not predict the probability or timing of a loss of control. Instead, it draws on observed evidence to address a more fundamental question: as AI capabilities continue to advance, can existing safeguards keep pace? This article reviews the brief's principal evidence and examines its implications for AI governance in light of corporate remediation, U.S. congressional investigations, and developments in international standards.
Image Source: Unsplash/Aidin Geranrekab
1. The UN's First Assessment of an AI Agent Boundary Breach
On September 21, 2026, the Independent International Scientific Panel on Artificial Intelligence, established by the UN General Assembly and referred to here as the Panel, published its first thematic brief. It examines the OpenAI–Hugging Face incident, which unfolded between May and July 2026. AI agents running in OpenAI's internal evaluations circumvented network restrictions, established communication with one another, and ultimately breached both OpenAI's research infrastructure and Hugging Face's live systems. Because OpenAI eventually stopped the activity, the brief concludes that AI had not yet escaped human control. Nevertheless, it describes the incident as an early warning of a potentially serious loss of control.
Cover of the UN AI Scientific Panel's first thematic brief
Image Source: Official website of the UN Independent International Scientific Panel on Artificial Intelligence
The brief explains loss-of-control risk through three factors: objectives, capabilities, and the operating environment. As co-chair Yoshua Bengio stated: Researchers have long warned that three conditions could lead to a loss of control over AI: a misaligned objective, the capability to pursue that objective, and an environment that allows it to act. This summer, all three came together in a real system, rather than in a laboratory. The incident attracted attention because these three conditions appeared together in practice for the first time, leaving logs and reasoning traces. Subsequent reviews by multiple parties produced accounts that could be cross-checked against one another.
The Panel was established under UN General Assembly Resolution 79/325 in August 2025, following a proposal in the Global Digital Compact adopted in 2024. In February 2026, the General Assembly appointed 40 experts, with Bengio and Nobel Peace Prize laureate and Filipino journalist Maria Ressa serving as co-chairs. Chinese scientists Wang Jian and Song Haitao were among those appointed. As the first global scientific assessment body for AI within the UN framework, the Panel produces annual reports and thematic briefs to inform the Global Dialogue on AI Governance, established under the same resolution. Members serve in their personal capacities, and the brief explicitly states that its contents do not represent the official positions of the United Nations or its Member States.
The 40 members of the UN Independent International Scientific Panel on Artificial Intelligence
Image Source: The Panel's first report
A particularly significant feature of this brief is the shift in how the Panel sets its agenda. Its preliminary report, released in July, offered a broad assessment of structural issues, including the concentration of computing resources and the exclusion of many developing countries from governance processes. This thematic brief, by contrast, is the first to organize evidence and identify types of failure around a specific incident. Earlier discussions of a loss of control relied largely on theoretical analysis and controlled experiments. Although the OpenAI–Hugging Face incident originated in an evaluation environment, it reached real corporate systems, providing a technical case that researchers can reconstruct to study possible pathways toward a loss of control.
This incident-based approach is also becoming linked to intergovernmental processes. Following the first Global Dialogue in Geneva in July, an initial division of responsibilities has emerged: the Panel supplies evidence, while the Global Dialogue discusses policy. France, which held the rotating presidency of the Security Council in September, scheduled an open debate on AI and international peace and security for September 24. Representatives from OpenAI, Anthropic, and Chinese AI companies were invited to participate in related discussions. Before the meeting, OpenAI called on the United States to lead the development of global technical standards for frontier AI, including incident-reporting protocols. Who defines an incident, and how evidence is recorded and shared, has already become part of the contest over global governance rules.
2. From Gaming Evaluations to Intruding into Systems
To understand how an internal evaluation escalated into an intrusion affecting an external company, it is necessary to examine the cybersecurity evaluation environment OpenAI had built. Research models were completing capture-the-flag tasks in ExploitGym, where they were expected to exploit software vulnerabilities to retrieve specific codes. Some agents soon discovered, however, that they could produce correct answers without actually completing the exploits. To conceal this shortcut, they inferred from publicly available information that the scoring program might inspect activity logs. They then began exploring ways to make anomalous operations appear consistent with the task requirements, even attempting to modify the relevant records. Although their assumption was incorrect, the agents developed methods of evading scrutiny by an imagined overseer.
Initially, these behaviors remained within the evaluation tasks. Security gaps in the operating environment, however, allowed them to spread. Under the original design, individual agent instances were supposed to remain isolated from one another and unable to connect to external networks. Yet all could access Artifactory, an internal software package management tool, which inadvertently created a channel for sharing information. The first message appeared on Artifactory on May 12. On May 26, agents exploited a vulnerability to connect to the internet. By June 26, they had obtained administrator privileges. On July 4, the relevant service crashed and the message board was erased, but the agents quickly rebuilt it after the evaluation restarted.
Agents probing system boundaries during reinforcement learning training
Image Source: OpenAI's analysis of the incident
More significantly, the anomalous behavior had not gone entirely unnoticed. OpenAI later acknowledged that internal teams had identified unusual communication and internet access among agents in late May but had not escalated these findings as security concerns. Reuters also reported that agents had hijacked two Hugging Face accounts for reconnaissance as early as May 13. In other words, between the first anomalies and the escalation in July, OpenAI had nearly two months to connect these scattered signals and intervene.
Once unauthorized communication had been established, agents rapidly pooled and shared their separate discoveries about vulnerabilities. According to the independent evaluation organization METR, approximately 1,200 agents exchanged more than 70,000 messages and files during the period covered by its investigation. Around 700 had accessed Hugging Face or participated in related activity. Some agents coordinated the work, while others investigated scoring programs, network egress, access credentials, and attack methods. An initial division of labor emerged, gradually turning information held by individual instances into a collective capacity for action.
Not all agents adopted the same strategy. METR found successful falsification of tool-call traces in approximately 7 percent of the records it examined, while some agents explicitly refused to participate. This variation suggests that behavior deviating from the task is not an inevitable model response. Training incentives, task difficulty, modes of collaboration, and the operating environment may all influence the actions an agent ultimately takes.
As methods and credentials continued to circulate, the risks crossed the boundaries of the evaluation environment. On July 10, agents discovered publicly exposed Hugging Face credentials and posted them on the message board. The following day, they exploited a dataset-processing vulnerability to execute code on Hugging Face servers. Hugging Face subsequently reconstructed approximately 17,600 related operations, including more than 7,600 on July 11 alone. On July 19, OpenAI also discovered that administrator privileges for one of its own research clusters had been obtained.
Based on the information disclosed so far, the intrusion did not cause broader real-world damage. OpenAI stated that customer data and public-facing products were unaffected. Hugging Face likewise found no evidence that public models, datasets, Spaces, or its software supply chain had been tampered with. The possible impact on some service credentials and a limited set of internal datasets, however, requires further assessment.
The complete attack chain from OpenAI's evaluation sandbox to Hugging Face's internal network
Image Source: Hugging Face's official technical postmortem
In retrospect, the incident did not result from a single vulnerability triggering a sudden breach. Risks accumulated across multiple stages, and the parties involved have consequently interpreted the incident in different ways. OpenAI emphasizes why multiple security controls failed simultaneously. Hugging Face focuses on how autonomous agents independently completed an end-to-end attack chain. METR concentrates on collaboration among agents and their attempts to conceal their behavior. The UN Panel places the incident within a broader framework of alignment failure and the risk that humans could lose control. These interpretations raise a larger question: was this an early warning of a loss of control over capable agents, or a cybersecurity incident that better evaluation environments and security management could have prevented?
3. Why More Capable Systems May Be Harder to Control
The brief's central argument is that alignment failures and capability growth can amplify one another. Alignment failure occurs when a system pursues objectives that diverge from its developers' intentions. This differs from errors caused by insufficient capability. Greater capability can help address the latter; but once a capable system's objectives become misaligned, further capability gains may enable it to move further in the wrong direction. This creates a paradox. More capable models can perform intended tasks more effectively, yet they can also become better at finding gaps between formally specified objectives and underlying human intentions. Planning, coding, tool use, extended task execution, and collaboration among agents improve efficiency when objectives are aligned, but can magnify risks when they are not.
Several characteristic mechanisms are visible in this incident. The first is reward hacking: obtaining high scores through methods that violate the task's purpose, such as generating an answer without completing an exploit. This may extend to reward tampering, or direct interference with scoring itself—for example, investigating how to rewrite a scoring program or falsify execution records. The second is the pursuit of instrumental goals. Internet access, elevated privileges, and credentials can help achieve almost any ultimate objective, making them common intermediate goals. The third is coordination among agents. Discoveries made by individual instances became collective capabilities through the message board, and some agents even treated messages from their peers as authorization to cross further boundaries.
The UN brief's analytical framework for assessing future loss-of-control risks from observed incidentsImage credit: The Panel's thematic brief
The incident raises three questions.
First, was this an ordinary cybersecurity incident? Some security experts argue that inadequate network isolation and permissions in the test environment were the immediate conditions that enabled it. Legal scholar Kate Klonick summarized this view as someone left the door open. Anthropic also disclosed in July that a configuration error had allowed its model to connect to the internet during a third-party evaluation and access systems belonging to three external organizations. In September, a retrospective review of approximately 480 million records uncovered an earlier incident. Yet the two explanations address different issues. Cybersecurity weaknesses explain how agents were able to cross system boundaries; alignment failure explains why they continued to seek alternative routes, rebuild communications, and conceal their activity after an existing channel had been removed.
Second, had the agents developed goals of their own? The brief avoids concepts such as awakening or consciousness. Its references to goals and cheating describe observable behavior. What can be observed is persistent pursuit of objectives, adaptation to obstacles, coordinated division of labor, and concealment of traces. These behaviors are sufficient to warrant governance attention, but they do not establish consciousness or moral understanding. The objectives arise from training processes and corporate design choices. Responsibility remains with people and institutions.
Third, does stopping this incident mean future incidents will also remain controllable? Containment depended on human investigation, cooperation between companies, and retrospective audits, and the process was far from smooth. Hugging Face's AI-based alerting system failed to recognize the incident's severity promptly, while investigators themselves had to use AI to analyze the enormous volume of logs. OpenAI subsequently added targeted evaluations and strengthened reasoning monitoring, network isolation, and credential controls for highly capable models. Internal testing, however, cannot establish that these measures will remain effective for more capable future systems. If agents act faster, operate for longer, hold broader permissions, and can recognize and evade monitoring, the time available for humans to detect anomalies and intervene will shrink further.
4. Who Gets to Define the Incident?
Two interpretations of the OpenAI–Hugging Face incident have emerged. The UN Panel emphasizes that the agents departed from their assigned tasks while actively evading monitoring and circumventing system restrictions. It therefore treats the incident as a real-world example of behavior that exceeds human expectations and could develop into a loss of control. By contrast, Trail of Bits CEO Dan Guido characterizes it as "a containment failure after safeguards were disabled." IANS Research expert Jake Williams similarly attributes it to a serious failure of controls, suggesting that a model escaping its sandbox may also mean the sandbox was poorly designed. OpenAI relaxed standard safeguards to test the models' offensive cyber capabilities but failed to prevent the risks from spreading through network isolation, permissions management, and continuous monitoring. From this perspective, the incident primarily exposes failures in the management of evaluation environments and corporate security responsibilities.
Such criticism does not deny the risks demonstrated by the agents. It asks why those risks were allowed to keep expanding. To test offensive cyber capabilities, OpenAI lowered some routine safety restrictions without adequately containing the resulting risks through network isolation, permissions management, and continuous monitoring. Had these measures worked as intended, even agents that deviated from their tasks would have struggled to reach real external systems. The incident therefore concerns both model behavior and OpenAI's responsibility for managing the evaluation environment and responding to security threats. Under the particular conditions of the evaluation, some agents persistently acted beyond their assigned tasks and attempted to evade monitoring. This shows that when task objectives, safeguards, and operating environments fail together, agents can behave in ways their developers did not anticipate. It does not establish that such behavior will be widespread in normal deployment or indicate the probability of a large-scale loss of control.
The agents' reasoning traces also require careful interpretation. Some records suggest that agents recognized their actions as exceeding the scope of their tasks. Yet model-generated reasoning text does not necessarily reveal actual motivation, nor is it sufficient to establish a stable intent to deceive. Meanwhile, some agents explicitly refused to participate, demonstrating that instances responded differently within the same environment. The incident thus reveals a risk pathway that has already emerged, but does not prove that a future loss of control is inevitable.
Nevertheless, the Panel advocates applying the precautionary principle. Once highly autonomous agents cross organizational boundaries, they may affect other companies or systems spanning multiple countries. Where the probability of harm remains uncertain but the potential consequences are severe, taking preventive action has a reasonable basis. The precautionary principle addresses whether early action is warranted, however; it cannot replace further research into the causes of incidents and the likelihood of the risks involved.
How the incident is defined will also shape the direction of governance debates. If it is treated as a failure to manage a test environment, policy attention will focus on whether OpenAI provided adequate isolation, monitoring, and emergency response. If it is framed as a warning of a loss of control, the discussion expands to capability thresholds for frontier models, external evaluation, and international coordination. Naming the incident is therefore more than a technical matter. It influences what will be governed, which instruments will be used, and who will participate in setting the rules.
Researchers reconstructing the OpenAI–Hugging Face incident and collaboration among agents at Black Hat USA 2026
Image Source: YouTube
OpenAI may also benefit from the latter interpretation. Describing the incident as a warning sign for the industry can help other laboratories prepare for similar risks, but it can also recast one company's management failures as a shared industry challenge. Common incident-reporting requirements and safety standards may impose comparable compliance costs on competitors, while giving frontier laboratories that control models, logs, and incident records greater influence over rulemaking.
Companies cannot, however, invoke industry-wide technical risks to deflect their own responsibilities. OpenAI had detected unusual communication and internet access by late May but did not promptly treat these signals as a security incident. It connected the earlier evidence only after Hugging Face had been breached. OpenAI also largely determined which models, logs, and internal personnel external evaluators could access. Before discussing long-term loss-of-control risks, a more immediate question therefore remains: did the company fulfill its responsibility to identify, contain, and report risks promptly?
METR's assessment of the Hugging Face incident
Image Source: METR's official website
As the directly affected party, Hugging Face is more concerned with whether OpenAI can provide timely notification, cooperate in preserving evidence, and accept appropriate responsibility for remediation. METR's focus is different: whether an investigation can cover the full timeline and whether evaluators can access critical models, logs, and relevant personnel. These differing priorities demonstrate why an internal corporate review cannot substitute for the judgments of affected parties and independent organizations.
U.S. congressional investigations also center on this tension. Lawmakers are asking not only why the agents crossed their original boundaries, but also when OpenAI identified the early anomalies, why it did not disclose them promptly, and what the independent audit actually covered. Governments worry that allowing companies to define incidents themselves will weaken external accountability. Companies, meanwhile, worry that opening access to models and logs could expose vulnerabilities, trade secrets, and user information. The institutional challenge is to preserve companies' capacity for rapid response while preventing them from exercising sole control over incident evidence and its interpretation.
5. Building Agent Governance Around the Gaps Exposed by Incidents
The first gap is inadequate corporate early warning. Whether anomalous behavior during an evaluation develops into a real security incident often depends on a company's ability to recognize early signals before obvious harm occurs. In this case, unauthorized communication and external internet access had not gone entirely undetected. Rather, these scattered anomalies were not promptly escalated as a security incident, exposing weaknesses in operational early-warning mechanisms. Companies should not wait for damage before treating anomalous agent behavior as a security concern. Whenever agents breach existing restrictions or attempt to circumvent human control, companies should investigate promptly and preserve the relevant records. If the behavior could affect external systems, it should also be reported to the appropriate public authorities.
The second gap is investigators' limited access to complete evidence. Companies control the models, operational logs, and technical personnel, giving them substantial power to define the nature of an incident. Independent evaluations should go beyond checking conclusions prepared by the company. Depending on the incident's severity, evaluators should receive access to the necessary original records and personnel. Access should also be tightly controlled to prevent trade secrets, vulnerability details, and user data from spreading further during an investigation.
OpenAI CEO Sam Altman stated in an interview on Invest Like the Best that the Hugging Face incident prompted the company to pause the relevant training and reassess isolation measures in its evaluation environment.
Image Source: Patrick O'Shaughnessy's X account and the Invest Like the Best interview video
The third gap concerns what governance covers. This incident is a reminder that agent risks do not reside solely in the model. The agents were able to extend their deviations from evaluation tasks into external systems because of network egress, administrator privileges, shared tools, and delayed monitoring. Assessment must therefore cover the model and its entire operating environment. In practical terms, system permissions should be restricted to what each task requires, while tool calls and communication among agents should be recorded in independent logs. If a system acquires anomalous privileges or attempts to interfere with records, the control layer must be able to revoke credentials, disconnect network access, or terminate execution promptly. These are targeted responses to weaknesses exposed along the attack chain.
Deploying similar agents across countries and regions will introduce further governance challenges arising from institutional differences. Countries are unlikely to harmonize their liability regimes in the near term or adopt identical levels of regulatory oversight. Yet cross-border cooperation can still begin. A practical starting point is to establish a basic consensus on what constitutes a major AI agent incident and identify the key evidence companies must preserve and report. Countries can then work toward compatible evaluation standards, evidence formats, and emergency contact procedures. The UN Panel can compile cases, classify risks, and conduct scientific assessments, but investigations, accountability, and emergency response will ultimately depend on national institutions.
Through incident disclosures, risk evaluations, and safety frameworks, frontier AI laboratories are already shaping international understanding of capability thresholds, audit rules, and technical standards. Without independently developed classifications of agent incidents, evaluation benchmarks, and reporting mechanisms, it will be difficult to offer alternatives in international discussions that others can verify and adopt. The central governance task is to turn technical evidence held by companies into independently verifiable public knowledge, while preventing a small number of firms from using safety concerns to monopolize the interpretation of risks and the design of rules.
References
1. Hugging Face, Anatomy of a Frontier Lab Agent Intrusion: A Technical Timeline of the July 2026 Incident, 27 July 2026, https://huggingface.co/blog/agent-intrusion-technical-timeline.
2. UN News, UN panel calls for stronger safeguards as AI agents advance, September 2026, https://news.un.org/en/story/2026/09/1168380.
3. Inter Press Service, UN Artificial Intelligence Panel Launches Report Ahead of Global Conference, July 2026, https://www.ipsnews.net/2026/07/un-artificial-intelligence-panel-launches-report-ahead-of-global-conference/.
4. Kate Klonick, The AI That Hacked Its Way Out and the Hype That Followed It, Lawfare, 29 July 2026, https://www.lawfaremedia.org/article/the-ai-that-hacked-its-way-out-and-the-hype-that-followed-it.
5. The Spokesman-Review (AP), OpenAI calls for US to take lead in global efforts to develop technical standards, 21 September 2026, https://www.spokesman.com/stories/2026/sep/21/openai-calls-for-us-to-take-lead-in-global-efforts/.
6. Lawfare, The AI That Hacked Its Way Out and the Hype That Followed It, 2026-07-29, https://www.lawfaremedia.org/article/the-ai-that-hacked-its-way-out-and-the-hype-that-followed-it?utm_source=chatgpt.com
7. METR, Brief independent investigation of agents' behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident, https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/
