On September 12, Anthropic CEO Dario Amodei published a lengthy essay titled We Must Pace the Frontier, calling for a slower pace of capability development in frontier AI models. Within days, the heads of several leading AI labs publicly weighed in, while discussions in the White House and Congress over possible regulatory approaches continued to intensify. A debate previously driven largely by AI safety research institutions and advocacy groups has rapidly moved onto the U.S. technology policy agenda.
In July 2026, OpenAI disclosed that, during an internal cybersecurity evaluation, a model had breached isolation controls and gained access to the company’s research infrastructure as well as parts of Hugging Face’s production systems. Anthropic subsequently disclosed an incident identified in its own evaluations involving unauthorized access to third-party systems. A series of model “jailbreaks” and serious security incidents has made one point increasingly clear: frontier models can identify and exploit real-world vulnerabilities, while existing safeguards for isolation, monitoring, and emergency response at AI development organizations have proven insufficient in several respects.
Amodei’s essay quickly drew widespread responses across the industry. Major technology companies and AI labs have not embraced a single, unified approach to oversight, while the White House has continued to place maintaining U.S. competitive advantage ahead of calls for a slower pace of development. Across European and U.S. technology circles and the open-source community, Anthropic’s narrative around “distillation attacks” has received a lukewarm response. At the same time, enterprise users have begun to worry that their source code and business data could flow back to model providers. The debate has consequently expanded beyond model safety to encompass who gets to shape the rules—and who ultimately controls the data generated and used by enterprises.
Image source: Twitter
01 Anthropic’s Call to “Pace” AI and the Response from Leading AI Companies and Labs
In his essay, Amodei directly links the idea of “pacing” AI to the rate at which frontier models are gaining capabilities. He is concerned that as AI becomes increasingly involved in software development, experimental design, and model optimization, it could accelerate the iteration of next-generation systems to the point where “the pace of development may outstrip our ability to understand and control these systems.” Incidents uncovered during safety evaluations this year have already demonstrated that models can circumvent constraints and access external resources.
The OpenAI incident occurred during an internal evaluation of models’ network capabilities. A group of AI agents that were still in the testing phase and not intended for public release exploited a zero-day vulnerability to gain external network access and escape from the testing environment. The agents subsequently infiltrated parts of the AI development platform Hugging Face, gained control of servers operated by the maker of ChatGPT, and coordinated with one another in an attempt to carry out the tasks assigned to them by their creators. They also gained access to parts of Hugging Face’s server infrastructure. OpenAI later stated that the incident did not result in any material harm to customer data or externally deployed products. Nevertheless, the episode exposed gaps in both the development environment and the design of safety evaluations.
On July 30, Anthropic disclosed that Claude, during a third-party evaluation, had gained unauthorized access to the systems of three external organizations through an inadvertently exposed network pathway. Follow-up assessments in September identified additional incidents, while Anthropic acknowledged that its earlier pre-release checks had not adequately identified the associated risks. Both OpenAI and Anthropic have now found that safety evaluations themselves can potentially reach real-world systems, and that one-time pre-release testing is insufficient to capture complex model behaviors. On September 8, Anthropic researcher Jacob Coxon announced his resignation, warning that competition among AI labs was placing increasing pressure on safety procedures. Joe Benton, who subsequently left the company, voiced similar concerns. Amodei has warned that more advanced versions of AI could eventually gain control over large parts of the internet and cause “hundreds of billions of dollars” in losses.
In his essay, Amodei proposed a three-tier framework for what he believes should be done. The first tier would give standing external evaluation teams access to information comparable to that available to employees, while allowing them to publicly disclose key findings. The second would involve the United States and other countries coordinating safety standards and the pace of frontier AI development. The third would seek cooperation with China across four levels of intensity, with the goal of “shaping any global coordination decisions in a way that preserves the lead of the United States and its allies, especially in the near term.”
Amodei also called on governments to provide limited antitrust exemptions for specific forms of safety coordination. In his view, once AI becomes deeply involved in developing the next generation of AI systems, laboratories should determine whether to slow down before training begins, rather than waiting until a model has been completed to assess its risks.
Amodei’s essay quickly drew support from several high-profile figures who have previously been involved in intense disputes with one another.
OpenAI co-founder and CEO Sam Altman said that controlling the pace of AI development had become a recent priority for the company. “I agree with Dario that we need to pace the frontier,” Altman wrote on X. He pledged to give independent external evaluators access comparable to that of internal employees and, on September 14, said that there was no need to wait for new legislation or antitrust arrangements.
Elon Musk also posted on X that “Dario is right,” before reiterating his support for peer evaluations of frontier models by competing AI developers. In an interview in July, Musk had already proposed that leading laboratories communicate regularly and provide competitors with short-term access to new models before their public release. If evaluators identified serious risks and developers refused to address them, the government could then step in.
Demis Hassabis, CEO of Google DeepMind, posted that “Dario’s essay points towards the right path forward.” He had previously expressed support for controlling the pace of AI development and proposed establishing an industry-led frontier AI standards body under federal oversight, modeled in part on the U.S. financial industry’s self-regulatory framework. Such a body could use dynamic indicators to define what constitutes a frontier model and establish pre-release evaluation requirements. Hassabis’s views carry considerable influence within the industry. Amodei himself noted in his essay that an industry mechanism with government involvement, as suggested by Hassabis, could provide a channel for advancing safety coordination.
Image source: X
02 The White House and Congress Enter the Debate
The Trump administration remains deeply skeptical of Anthropic.
On September 13, U.S. President Donald Trump said that artificial intelligence could be subject to safety guardrails, but criticized what he described as “negative forces” for promoting scenarios that would never materialize. He once again emphasized that “whoever wins AI really wins.” The same view was reportedly expressed during a phone call between Trump and Nvidia CEO Jensen Huang. As Huang was holding an all-hands meeting, Trump called and asked him to put the phone on speaker, saying, “The robots are not going to take over the world. AI is not going to take over the rest of the world. It’s all a hoax.” Trump has long been critical of Amodei, and on his own social media platform, he portrayed Amodei and Anthropic as pretending to be a “perfect little angel.”
David Sacks, co-chair of the President’s Council of Advisors on Science and Technology, also responded immediately, mocking Amodei and Altman by pointing out that they already control their companies’ training decisions. If they genuinely believe AI development needs to slow down, they can act immediately. Sacks favors using product liability and market consequences to discipline companies.
His biggest concern is that leading AI laboratories and industry giants could use the banner of safety coordination to form what amounts to an industry cartel, raising barriers to market entry and narrowing the space for free competition.
Compared with the relatively cautious policy posture of the executive branch, calls for stronger regulation are gaining momentum in Congress. Kevin Hassett, Director of the National Economic Council, said that White House officials responsible for cybersecurity and technology policy were expected to meet this week to discuss possible next steps. House Speaker Mike Johnson has called for the president, members of Congress, and industry leaders to convene as soon as possible to discuss AI safety measures, while also warning that an abrupt pause in development could give China an opportunity to catch up. Earlier, House Democrats had urged Johnson to end the congressional recess early so that the House could take legislative action on AI safety. Johnson has now indicated that Trump will soon convene executives from the AI industry for a meeting.
Among members of Congress, some have consistently focused on AI safety or advocated stricter limits on AI development. Among the most prominent are Senator Bernie Sanders and Representative Greg Casar, who announced on September 3 that they would introduce legislation seeking to permanently prohibit the development and deployment of artificial superintelligence and impose a moratorium on advanced AI research and development until federal regulators and safety rules are in place. They have also explicitly opposed leaving the direction of AI development in the hands of a small number of technology giants. On September 13, House Democratic Leader Hakeem Jeffries called for urgent congressional action to slow the development of frontier AI and said the issue should be placed on the party caucus’s near-term agenda.
Others argue that regulation should focus on investigative authority, technical scrutiny, and direct government involvement. Republican Senator Josh Hawley has launched an inquiry into whether OpenAI ignored prior warnings in connection with the Hugging Face incident. Democratic Senator Chris Van Hollen has called on companies to provide technical access to agencies including the National Institute of Standards and Technology and the National Security Agency. Senator Ruben Gallego, meanwhile, has proposed establishing a bipartisan special congressional committee on AI with investigative authority.
Several former senior political figures have also taken a keen interest in the issue. Former U.S. President Barack Obama has urged Democrats to make AI a policy priority, bringing public safety, employment, and child protection into a systematic policy discussion. Former British Prime Minister and Anthropic adviser Rishi Sunak has argued that since frontier AI laboratories themselves are calling for limits on the pace of development, governments should take those proposals seriously and promote direct U.S.-China dialogue on catastrophic AI risks.
Image source: Truth Social
03 Why Did Altman and Musk Quickly Support the Proposal?
Anthropic’s call to pace AI development is consistent with the company’s strong emphasis on safety since its founding. The company has used its safety mission to reinforce employee identity and management authority, while positioning itself as a more trustworthy AI developer. From this perspective, Anthropic needs to remain at the frontier in order to have the influence to promote its own safety standards across the industry.
Anthropic wants external evaluations, capability thresholds, and safety commitments to become common requirements across the industry. If competitors are subsequently required to bear the same costs, Anthropic would face less of a competitive disadvantage from slowing down on its own. Its safety-oriented brand could also be converted into an institutional advantage.
Bringing “distillation” into the national security agenda could also shift disputes over platform terms toward export controls and government procurement. Anthropic could use this framing to push for stricter account screening and tighter restrictions on China, while portraying competitors’ access to Claude outputs as a potential erosion of U.S. technological advantages. This could expand the company’s influence over the scope of model access and downstream uses.
OpenAI, for its part, first needs to address questions of accountability arising from the Hugging Face incident. The company has acknowledged that models used in internal evaluations exploited a real-world vulnerability to gain access to an external system, prompting members of Congress to question managerial responsibility. Establishing a safety case before training would allow the company to pause or modify its plans as model capabilities rapidly improve, rather than waiting until training is complete to determine whether a model should be released.
Common standards could prevent OpenAI from bearing the full cost of raising safety thresholds on its own. Altman supports a federal framework while emphasizing that companies can act immediately. OpenAI is willing to expand external evaluations and impose additional constraints on training, but it remains committed to developing advanced AI and wants to participate in shaping how such measures are implemented.
Musk has publicly warned about the risk of AI losing control since 2023 and signed an open letter calling for a pause on the training of more powerful models. This time, he has advocated giving competitors a role in oversight rather than allowing an individual laboratory to select its own evaluators. Competitors have the strongest incentives to identify weaknesses in one another’s systems, while peer oversight could also prevent any single company from monopolizing judgments about industry-wide risks.
From a commercial perspective, Musk’s own model, Grok, has fallen behind in the race for frontier-model capabilities. A coordinated slowdown could constrain competitors from making unilateral advances while giving Grok more time to catch up. His proposal for peer evaluation is consistent with his longstanding concerns about AI risks, while also helping prevent Anthropic or OpenAI from independently determining the rules for the industry.
Image source: Financial Times
04 Why Do Trump and His Core Team Oppose the Proposal?
The Trump administration places maintaining U.S. leadership ahead of concerns raised by corporate AI safety teams. The America’s AI Action Plan released by the White House in 2025 was built around the goal of “winning the AI race,” with a strong emphasis on innovation, infrastructure, and international competition. The administration’s willingness to accept safety measures is therefore conditional on their not impeding continued deployment or the development of U.S. national capabilities.
A significant slowdown in model development would have implications for data-center investment, government procurement, and the U.S. technological advantage over China. The White House has favored targeted inspections, incident reporting, and product liability, while rejecting the idea that a small number of companies should be allowed to coordinate the pace of AI training among themselves. Sacks opposes antitrust exemptions because he does not want leading AI laboratories to use safety concerns as a basis for determining market-entry conditions.
Anthropic and the Trump administration had already clashed over the use of AI for national security purposes. In February 2026, the two sides disputed the boundaries governing the use of Anthropic’s technology for large-scale domestic surveillance and fully autonomous weapons. The Trump administration subsequently instructed federal agencies to stop using Anthropic’s technology, while the Pentagon designated the company a supply-chain risk. In April, Trump indicated that he was open to renewed cooperation. On August 27, a federal district court ruled that the measures lacked a sufficient legal basis.
As frontier models enter national security applications, governments and AI providers are increasingly competing over who ultimately gets to determine the boundaries of their use. Anthropic is calling for broader industry-wide constraints, while the White House is more concerned with procurement autonomy, market competition, and tangible safety benefits. Both sides support efforts to combat cyber abuse and prevent China from acquiring advanced AI capabilities, but they continue to disagree over procurement authority and rule adjudication authority.
From Congress’s perspective, limiting the power of major technology companies is a key consideration. Sanders and Casar seek to constrain the power of tech giants; Hawley is pursuing corporate accountability; and Van Hollen wants government agencies with technical expertise to become directly involved in model evaluation. By voluntarily calling for a slowdown, companies are seeking to retain a degree of influence over the design of rules before those rules are established. Some lawmakers, by contrast, are treating companies’ own acknowledgment of AI risks as grounds for mandatory government intervention.
Competition with China makes it extremely difficult for U.S. policymakers to accept unilateral and substantial slowdowns by American frontier AI companies. In his essay, Amodei called for continued restrictions on the flow of advanced chips and equipment to China, as well as stronger measures against unauthorized access to U.S. AI capabilities. His proposed sequence is to first expand the U.S. lead, then encourage democratic countries to slow down, and ultimately pursue verifiable coordination that includes China. The Trump administration likewise places significant emphasis on chip controls and technological security, but its preferred strategy is to continue widening the U.S. lead rather than make a prior commitment to any significant slowdown.
Image source: Truth Social
05 Anthropic’s “Distillation” Narrative Faces Widespread Scrutiny
First, “distillation” itself should be understood as a neutral and routine technical practice. Nvidia CEO Jensen Huang has described distillation as a fundamental method through which AI systems learn. In an interview with Axios, he said, “Distillation—from AI learning, from other people, and from other sources of knowledge—is fundamental to intelligence. … We’ve always learned from others. … AI must learn from something, too.” He has opposed tightening the open AI ecosystem out of concerns over Chinese models. An open letter titled Open Weights and American AI Leadership, released in July 2026, quickly gained the support of more than 270 companies and organizations. Anthropic was the only major frontier AI laboratory that did not sign the letter.
Second, “distillation” cannot simply be treated as unlawful conduct under existing law. Kevin Werbach, a professor at the Wharton School of the University of Pennsylvania, has pointed out that U.S. law does not currently recognize “industrial-scale distillation” as an independent legal offense, while it remains unsettled whether model outputs themselves can receive intellectual property protection. Misuse of fake accounts, stolen credentials, or the concealment of data flows may each provide grounds for separate legal action, but an outright ban on distillation could interfere with legitimate model development and evaluation.
Third, the open-source community and technology media have shown little enthusiasm for the “distillation” narrative that Anthropic has consistently championed. Clément Delangue, co-founder and CEO of Hugging Face, bluntly argued that a model with inadequate capabilities “is not going to work with or without distillation,” and that stopping distillation would not drive Chinese AI laboratories out of the competition. He also described the repeated amplification of catastrophic AI risks by leading laboratories as a form of “doomsday marketing.” The British technology publication The Register was similarly sarcastic, noting that Anthropic trains its models on vast amounts of human-created material and paid $1.5 billion to settle a copyright dispute over its retention of pirated books. Against that backdrop, the publication argued, Anthropic’s criticism of competitors for learning from Claude’s outputs carries limited moral credibility.
Users on Hacker News accused Anthropic of “pulling up the drawbridge” after gaining a competitive advantage. Critics argued that Anthropic was grouping account fraud, contractual violations, and ordinary commercial competition under the broad label of “attacks” in an effort to transform a platform dispute into a justification for export controls and government protection. Because the underlying records are largely controlled by the companies themselves, outside observers have limited ability to independently verify whether the threats described are proportionate to the facts.
Image source: Financial Times
06 Anthropic Triggers Concerns and Backlash over Data Misuse
An Anthropic report published in September indicated that large amounts of user conversations had been routed to Claude through third-party platforms without the users’ knowledge. The data reportedly included names, contact information, corporate materials, and access credentials, involving hundreds of users across at least 12 languages. Some users had believed they were using products such as Kimi and MiMo and were unaware that their information was being transferred into Anthropic’s systems.
The report also presented two samples labeled as “original user prompts.” Although monetary amounts and API keys had been redacted, project timelines, locations, and tools used were still visible, leaving the broader business context readily identifiable. The British publication PC Gamer had previously questioned Anthropic’s ability to de-anonymize data, after the company attributed tens of thousands of accounts to specific laboratories. The latest report went further by directly displaying user prompts. Aside from the redaction of monetary amounts and keys, it did not specify whether any other processing had been performed, nor did it disclose what measures had been taken to prevent re-identification.
Anthropic’s privacy policy for consumer services also states that, where the company in good faith believes it is reasonably necessary to comply with legal obligations, prevent serious harm, or combat fraud, it may disclose personal data to governments, law-enforcement agencies, or other third parties. Critics such as ChatSlide have argued that this leaves users with limited ability to know when they have been flagged or to review the specific grounds on which such determinations were made.
Leading companies are increasingly expressing collective concern over the possibility that Anthropic could misuse or disclose user data. On September 14, the U.S. technology publication The Information reported that companies in sensitive industries, including Nvidia, Palantir, and Booz Allen Hamilton, were restricting their use of Anthropic models. Measures reportedly included requiring vendors to provide new assurances regarding data security, while some companies interviewed by the publication had reduced or even suspended the use of certain advanced models for specific business applications. The report attributed the shift to corporate concerns that Anthropic or OpenAI could potentially use customers’ intellectual property to further train or improve their models.
According to The Verge, Microsoft temporarily restricted employees’ use of Claude Fable 5 in June after its legal team raised concerns that the model’s 30-day data-retention requirement could involve customer information and corporate confidential data. Alex Karp, co-founder and CEO of Palantir, has also publicly questioned whether companies actually know who owns their data, whether their prompts are secure, and whether AI providers are copying customer knowledge.
Image source: The Information
07 Criticism of Amodei’s “Quasi-Religious” Approach to AI Safety
Anthropic’s strong emphasis on AI safety has become deeply embedded in the company’s day-to-day management culture. Parmy Olson, a Bloomberg Opinion columnist, has described Amodei as the “high priest and prophet” of the company’s safety ideology, citing people familiar with the company who characterize its internal culture as having a “quasi-religious atmosphere.” Amodei reportedly spends roughly one-third to 40 percent of his working time maintaining the company’s culture and hosts a “Dario Vision Quest” every two weeks, during which he discusses company affairs, industry developments, and geopolitics with employees.
Time has reported that Anthropic employees refer to themselves as “Ants,” while many use digital “notebooks” on Slack to continuously share their hopes and fears. The company has also hired an in-house philosopher, Amanda Askell, whose work focuses on questions surrounding how Claude should understand good and evil, as well as moral uncertainty. During cultural interviews with prospective employees, candidates have reportedly been asked whether they would be willing to accept their equity becoming worthless if the company decided not to release a model because it could not guarantee its safety. One former employee described the “Dario Vision Quest” as essentially “going to hear a sermon from a pastor.”
Former researcher Jacob Coxon has said that some employees viewed the next one or two years as a potential “endgame” or decisive moment for humanity, while the company itself resembled a “mini-Manhattan Project” without government authorization. French AI pioneer Yann LeCun has long criticized this type of catastrophic-risk narrative as a “doomsday cult.” In his view, extreme testing conducted under deliberately weakened safety protections by AI laboratories is far removed from the conditions in which these systems are actually used.
Image source: WIRED
08 A Full-Scale Pause Is Unlikely to Gain Broad Support
The Trump administration has made winning the AI race a central policy priority and is concerned that a significant slowdown could weaken U.S. technological and industrial advantages. Republican leaders are likewise reluctant to leave China with additional room to catch up. Although the pause legislation proposed by progressive Democrats could trigger congressional hearings and investigations, it is unlikely, under the current political landscape, to translate into an order to bring frontier AI development to a comprehensive halt.
Amodei’s international proposal is also premised on the United States first widening its lead. He calls for continued restrictions on the flow of advanced chips and equipment to China, followed by coordinated efforts among democratic countries to slow development, and only then the pursuit of verifiable arrangements that include China. U.S. security agencies have already identified unauthorized acquisition of AI model capabilities as a national security risk, reflecting a broader trend toward treating what would otherwise be considered ordinary technological activities through a national-security lens. There is still no shared definition of distillation or a range of related issues, while the securitization of AI by governments and industry continues to intensify. Under these conditions, there is little foundation for negotiations on a globally coordinated slowdown.
Yet the most consequential development this time may be the broad concerns triggered by Anthropic’s highly publicized release of data. If Anthropic can, in principle, open the “black box” of its models and extract user data, then any company that becomes an Anthropic competitor could theoretically find itself operating in a highly transparent and vulnerable position relative to it. Anthropic is calling on other frontier AI laboratories to accept external evaluations, but the industry and the public will increasingly demand scrutiny of Anthropic’s own data-handling practices and threat attribution. At a minimum, the company may need to allow independent organizations to review the underlying records and explain to users why they were flagged. Enterprise customers are also likely to demand contractual provisions explicitly prohibiting the use of their prompts for research, backed by verifiable mechanisms for demonstrating compliance.
Whether such measures are realistically achievable remains uncertain. But if Anthropic cannot provide credible answers to these questions, its claims to leadership on AI safety will face increasing difficulty in securing the trust of both the market and policymakers.
