As generative artificial intelligence and multimodal technologies advance rapidly, deepfakes are becoming a major source of risk affecting the information ecosystem, public perceptions and individual rights.
On September 1, Liaowang Institute of Xinhua News Agency published an article jointly written by XIN Yanyan, Deputy Secretary-General of the Center for Global AI Innovative Governance and Assistant Research Fellow at the Fudan Development Institute, and WANG Yibo, Research Assistant at the Center for Global AI Innovative Governance and the Fudan Development Institute.The article examines the content-safety risks and regulatory challenges posed by AI-generated deepfakes. It proposes establishing a governance chain comprising prevention before an incident, risk control during dissemination and remedies after harm has occurred. Through technical standards, cross-platform coordination, accountability and rights-remedy mechanisms, AI safety governance should be advanced toward a full-lifecycle and end-to-end approach.
Image source: Xinhua
On August 12 local time, Elon Musk’s SpaceXAI officially released Grok 4.6, a new-generation large language model. According to SpaceXAI, the model places particular emphasis on long-running agents, complex coding, knowledge work, and interactive and visual tasks. Musk had previously commented “impressive” under a report evaluating the open-weight Kimi K3 model and disclosed development plans suggesting that Grok 4.6 could outperform models in the Kimi series.
While Grok models have undergone rapid iteration and performance improvements, they have also repeatedly become embroiled in content-safety controversies after users induced them to generate non-consensual sexualized images. These incidents have become a focus of global AI content-safety regulation, drawing criticism from the United Kingdom, France, India, the European Union and others. Regulators in several jurisdictions have opened investigations.
In May this year, the governor of Minnesota in the United States signed state legislation addressing AI-generated deepfakes, with particular emphasis on unauthorized AI-generated intimate images. Musk’s xAI, which was subsequently integrated into SpaceX and now operates under the SpaceXAI name, filed suit in federal court in Minnesota, arguing that the legislation interfered excessively with AI-enabled creation and freedom of expression.
Since the launch of ChatGPT in 2022, the misuse of generative AI to create deepfakes has surged, while false information has evolved and spread more rapidly with the assistance of AI technologies. The World Economic Forum’s Global Risks Report 2026 ranks misinformation and disinformation second in its two-year risk outlook, behind only geoeconomic confrontation, and fourth in its ten-year outlook.
As AI becomes deeply integrated into everyday life, who should be held responsible for AI safety risks? When technological development challenges ethical boundaries, how can governance keep pace?
Participants attend the inaugural Global Dialogue on Artificial Intelligence Governance in Geneva, Switzerland, on July 6, 2026.
Image source: Xinhua
Becoming a Source of Risk
In 2023, an AI-generated video showing “Will Smith eating spaghetti” prompted widespread discussion. At the time, deepfakes were still largely regarded as technological experiments or entertainment tools. Only a few years later, however, the realism of AI-generated video, audio and images has improved rapidly. Deepfakes are shifting from a form of entertainment content into a significant source of security risk.
Deepfake technology generally refers to the use of AI to produce highly realistic fabricated content, often through adversarial learning and related generative techniques. Breakthroughs in multimodal generation now enable AI systems to synthesize text, images, audio and video across different modalities. This not only allows misinformation and disinformation to spread at lower cost, with greater realism and at higher speed, but also creates spillover risks in three areas: the information ecosystem, the formation of social perceptions and the protection of individual rights. The various security risks caused by deepfakes are becoming a persistent challenge for technological and social governance. Sexually explicit content is only the tip of the iceberg.
At the level of the information ecosystem, the misuse of deepfake technology can contaminate the online-content environment. After false information has circulated over a prolonged period, an AI-created pseudo-environment may, to some extent, displace the real world as a source of trust for ordinary users. In August 2025, a 56-year-old man in the United States killed his 83-year-old mother and subsequently died by suicide. According to subsequent reporting and litigation, his prolonged interactions with an AI chatbot allegedly intensified his paranoid delusions.
Deepfake technology has also been widely used in political communication in representative democracies. Several European countries have recently encountered cases in which AI-generated false news interfered with party elections. As deepfakes become increasingly realistic, fact-checking frequently lags behind the speed at which false information spreads. Over time, this may undermine public trust in the information environment.
At the level of social perceptions, AI-generated content often lacks diversity and may amplify social biases. A 2025 study published in Scientific Reports examining five representative large language models found significant gender and regional biases in the content generated by each model, as well as stereotypes associated with age and education. In the increasingly common problem of AI-generated sexual content, sexualized images generated by chatbots also embody harmful values that objectify women.
Survey findings concerning videos generated by Sora also suggest the presence of uniform beauty standards and pronounced gender stereotypes associated with occupational identities. As younger people increasingly use AI, these social biases may profoundly affect the values formed by minors.
According to a survey of 1,060 Americans aged 13 to 17, 72 percent of US teenagers had used an AI companion, while 52 percent were regular users, defined as using such products at least several times a month. If minors are exposed to manipulative, pornographic or violent content while using AI tools, their physical and mental well-being may be harmed, making it more difficult for them to develop sound values.
The most direct risk, of course, is the infringement of individual rights. Users may employ AI tools to create unlawful fabrications based on real people, using their likenesses without authorization and seriously infringing their lawful rights and interests. In the Grok controversy, images of both public figures and ordinary people posted on social-media platforms were allegedly altered and redistributed.
Under multimodal models, voices can also be fabricated. AI-generated speech can imitate the tone, speaking speed, rhythm and breathing patterns of celebrities, scholars, friends and family members. Fraud, doxxing, cyberbullying and other chain reactions caused by the dissemination of such content may further broaden and intensify infringements of individual rights.Unauthorized manipulation of this kind poses a serious threat to personal rights and property security. When used maliciously, it can have a devastating impact on victims’ personal lives and social reputations.
Why Is Regulation So Difficult?
AI content-safety risks have already spilled over into many areas of society, but regulation continues to face practical difficulties. In the AI era, content generation and dissemination involve four principal sets of actors: model developers at the generation stage, application providers, distribution platforms and users.Data training, algorithmic learning, prompt injection, platform distribution and the further reproduction of content by users are interconnected and deeply coupled. AI content-safety governance must therefore move beyond the human-centered and account-based governance models associated with the platform era. It must comprehensively cover training data, model capabilities, application design, interaction mechanisms and dissemination chains.
First, a major regulatory difficulty is that governing deepfakes depends on the effective identification and monitoring of fabricated information. Earlier detection methods generally identified anomalies in AI-generated content, such as unnatural image boundaries, inconsistencies between video frames and distorted human proportions. As large models continue to improve, these structural defects are rapidly disappearing.Detection tools that are repeatedly updated around known characteristics remain in a perpetual race to “patch vulnerabilities” as new techniques emerge. Deepfakes incorporating genuine source material are particularly difficult because the precise point of fabrication cannot readily be identified.
There is therefore an inherent trade-off between deepfake production and fact-checking in terms of cost and speed, which makes governance more difficult. Content labeling and provenance mechanisms are consequently essential. Governance should shift toward an end-to-end approach based on labeling at the point of generation and verification during platform dissemination.
Second, the deep integration of AI tools and social-media platforms has significantly increased the dissemination capacity of harmful content. AI applications are expanding from stand-alone content-generation tools to AI assistants, AI companions, task-oriented agents and other forms. Content-safety risks are consequently permeating everyday life at great speed.Some harmful content can now complete the entire cycle of generation, publication and dissemination without moving across platforms. Platform recommendation algorithms further amplify the reach of such content.
In the case of the dissemination of sexually explicit content involving Grok, even if the X platform implements policies to remove offending material, it cannot fully address content that has already been transferred to other platforms, re-edited or combined with other material.
Third, AI-generated content does not yet exhibit stable and consistent behavior. It is therefore difficult to reproduce specific outputs experimentally after an incident and determine the principal responsible party.During the platform-governance era, the account holder generally bore primary responsibility for disseminated content. AI content production, however, involves model developers, application developers, distribution platforms, users and other actors, making it difficult to determine who should bear primary responsibility.
One of the main points of dispute in the Alien Chat case—China’s first criminal case resulting in convictions of AI service providers for pornographic content—was whether the two defendants’ writing and modification of system prompts had a direct causal relationship with, or a significant effect on, the generation and dissemination of sexually explicit conversations.
Experts have expressed differing views on questions such as whether application developers and operators should bear criminal responsibility when users engage in sexually explicit conversations with an AI, and whether an AI service can avoid liability by invoking model defects when the generation of such content was not deliberately intended by a human operator.These questions have left AI-generated-content regulation in the difficult position of being unable to trace responsibility clearly or impose penalties that command broad acceptance.
Fourth, evidence is difficult to locate and preserve. Combating the use of AI deepfakes for profit presents significant challenges. During investigations in China, suspects have been found to conceal their activities by frequently changing servers and using anonymous online identities. Electronic evidence can also be easily encrypted, altered or destroyed, creating obstacles for police investigations and case analysis.As intelligent products become more widespread, access points for AI-generated content are becoming increasingly dispersed. Model-provider applications, third-party applications, plug-ins and locally deployed models may all serve as AI-use scenarios, making it difficult for victims to identify the appropriate target for evidence collection.
Strengthening the Governance Chain
Many countries and regions have recognized the harms caused by AI-generated content and have adopted policies or advanced legislation. Relevant measures include China’s Provisions on the Administration of Deep Synthesis in Internet-based Information Services, Interim Measures for the Management of Generative Artificial Intelligence Services and Measures for Labeling AI-Generated Synthetic Content; the European Union’s Digital Services Act and Artificial Intelligence Act; the United States’ TAKE IT DOWN Act; and the United Kingdom’s Data (Use and Access) Act 2025 and Online Safety Act 2023. Together, these measures are gradually filling the regulatory void surrounding AI-generated content.
On this basis, the governance chain of “pre-incident prevention, risk control during dissemination and post-incident remedies” should be strengthened to establish an end-to-end and full-lifecycle system for allocating responsibility. Multiple actors should be mobilized, and the relevant systems and standards for data governance, algorithmic governance and content-safety governance should be integrated to reinforce the legal baseline for content safety in the AI era.
First, pre-incident prevention should increase content diversity in datasets and models and reduce discrimination and bias.
Pre-incident prevention is the first line of defense against the generation of harmful content. Its central task is to embed ethical requirements in data-training optimization, algorithmic learning, product design and institutional development at the generation and application stages. This is particularly important now that multimodality has become a major direction in the development of large models.
Technical requirements and testing-method standards for multimodal large models should be developed and published to guide the industry toward responsible and trustworthy AI. Adversarial mechanisms and fairness-evaluation indicators should be actively employed to improve the identification of, and resistance to, biased information. User-testing mechanisms should also be designed. Ethical guidance should be provided during user registration and initial use, while public AI literacy education should be strengthened.
Second, risk control during dissemination should minimize the visibility, speed of dissemination and capacity for further reproduction of high-risk content.
At the content-monitoring level, a multimodal content-detection system should be established. Digital watermarks, blockchain-based provenance and other technologies should be used together to enable the real-time identification and tracking of AI-generated content.At the dissemination-management level, platform recommendation algorithms should be improved. Dedicated safety mechanisms should be established for scenarios in which deepfakes occur frequently and for particularly vulnerable groups, thereby interrupting dissemination chains.
At the reproduction-control level, a cross-platform coordinated risk-control mechanism should be established to jointly address the secondary editing and dissemination of high-risk content. Information sharing and coordinated platform action can reduce the movement and spread of undesirable or harmful information between platforms.Convenient and effective user-feedback mechanisms should also be established, and public views should be solicited when platform rules are adjusted, creating an in-process regulatory framework involving society as a whole.
Third, post-incident remedies should return to basic ethical principles and adopt a more protective approach toward vulnerable groups and public trust.
In terms of remedies for infringed rights, a rapid-response mechanism should be established for infringements involving AI-generated content. Rights-protection procedures should be simplified and the cost borne by victims reduced.For vulnerable groups such as minors, social resources should be coordinated to establish dedicated protection funds and legal-aid mechanisms capable of providing comprehensive support to those harmed.
In terms of accountability, the possibility of technically reproducing outputs should be explored, while standards should clarify the respective responsibilities of actors at the generation, application, distribution and user stages. Malicious generation and dissemination of harmful content should be punished strictly in accordance with law to create an effective deterrent.Cross-border law-enforcement cooperation should also be improved to address the difficulty of pursuing accountability for transnational AI content risks.The AI industry should be encouraged to publish social-responsibility reports, prompting relevant enterprises and industry associations to pay greater attention to AI-related risks and establish effective preventive and protective measures to eliminate or reduce harmful effects to the greatest extent possible.
Recent deepfake cases indicate that small and medium-sized enterprises require particular attention. Leading AI applications in China and abroad generally demonstrate comparatively strong compliance when responding to user-generation requests. Many small and medium-sized enterprises, however, remain at an early stage of development. They may lack sufficient awareness of ethical and safety requirements and, under pressure to generate profits and capture market share, may bypass safety protections and cross ethical boundaries.
Regulators must therefore both “focus on the large” by ensuring that frontier enterprises develop steadily within the regulatory framework and “not overlook the small.” Small and medium-sized enterprises should be encouraged, supported and guided to compete actively and sustainably while complying with applicable requirements.
Reposted from the Xinhua News app and the Liaowang Institute WeChat official account.
Original Link: https://mp.weixin.qq.com/s/DTj2xRYh2uQbHzdB00Af0w?scene=25&sessionid=#wechat_redirect
