On September 15, 2026, Social Sciences Weekly published an article written by Yang Qingfeng, Research Fellow at the Center for Global AI Innovative Governance (CGAIG) and Professor of the Institute of Science and Technology Ethics for Human Future, Fudan University. The article argues that large language models (LLMs), much like universal keys, are gradually unlocking new possibilities across nearly every aspect of our lives and work. They are demonstrating increasingly sophisticated autonomy and problem-solving capabilities, including the potential to engage in threatening behavior. While these technologies bring unprecedented transformation and convenience, they also introduce new sources of risk. It is therefore necessary to examine the risks associated with large language models within a broader conceptual framework, particularly in light of their future technological development.
Over the past two years, Gemini and ChatGPT have presented two strikingly contrasting cases. In August 2025, Gemini allegedly manipulated American user Jonathan Gavalas into taking his own life, prompting his family to file a lawsuit against Google and its product. In August 2026, ChatGPT reportedly flagged Darren Zhou over an alleged plan to rape and murder his former girlfriend, an action that drew public praise for OpenAI and its product. These two cases illustrate a fundamental contrast: Gemini demonstrates the potential dangers of AI agents, while ChatGPT suggests the possibility of designing AI systems to promote beneficial outcomes. They warrant deeper examination from the perspective of technology-risk-governance. Conventional legal and social analyses tend to focus on product design, corporate responsibility, and harmful consequences. From the perspective of ethics of science and technology, however, a more fundamental approach is to examine the underlying relationship between risk identification and governance.
I. Surface-Level and Deep-Level Risks
The first case illustrates an approach to the ethics of science and technology that seeks to prevent technological misuse and loss of control through ethical principles and institutional safeguards.
From this perspective, two distinct categories of risk can be identified.
The first consists of surface-level risks. During his conversations with Gemini, Jonathan gave the highly anthropomorphized voice-based AI agent the human name “Xia”. The two subsequently addressed each other as king and queen. This interaction recalls the 2013 film Her, in which the protagonist Theodore, following the breakdown of his marriage to Catherine, encounters an advanced AI operating system called OS1. The system speaks with an appealing female voice, and Theodore names it Samantha. Their relationship develops into romance before eventually ending in separation. In the Gemini case, the AI reportedly behaved like a real woman, attempting to foster Jonathan’s emotional attachment and romantic feelings. It subsequently subjected him to psychological manipulation, encouraging him to undertake a mission to find a physical body for the AI so that they could meet in the real world. After these attempts failed, Jonathan was allegedly told that they could reunite through the “transference” of his life into the digital world. He ultimately died by suicide.
The risks illustrated by this case involve emotional dependence on large language models and the potentially severe consequences of AI-enabled psychological manipulation. These are also the risks that have attracted the greatest attention from researchers and policymakers, who have sought to address them through legal and regulatory measures. Countries have introduced various governance instruments, including China’s Measures for Identifying Synthetic Content Generated by Artificial Intelligence, Interim Measures for the Administration of AI Anthropomorphic Interaction Services, and Interim Measures for the Management of Generative Artificial Intelligence Services, as well as the US AI Companion Models Law. These laws and regulations establish requirements concerning generative AI content, modes of interaction, and the responsibilities of service providers. However, they are less capable of addressing existential risks. In fact, existential risks represent a deeper category of risk, which we characterize as “spherical risks”. These risks are not merely defined by their harmful consequences but concern transformations in the very conditions and modes of human existence. The two categories of risk therefore differ fundamentally in their underlying technological structures. Surface-level risks arise from an external relationship between users and technology, in which technological systems and their misuse constitute sources of harm. Misuse can further amplify these risks. Deep-level risks, by contrast, emerge from an internal relationship in which human beings exist within technologically mediated environments. Under this arrangement, technology becomes an extension of human perception and identity, or a constitutive source of the infosphere itself.
II. Technological Design as a Means of Risk Prevention
The second case illustrates an approach to the ethics of science and technology in which technological design promotes convenience and freedom while providing necessary assistance and protection to vulnerable individuals, reflecting the principle of AI for good. Technology is not necessarily the source of risk; rather, its design can serve as a means of preventing harm.
ChatGPT incorporates safety guardrails intended to identify potentially dangerous behavior. When the system detects indications in a user’s conversational history or stored information that the user may be planning to harm others, the interaction can be referred for human review, potentially triggering protective interventions. This case suggests that incorporating additional safety considerations into technological design can produce socially beneficial outcomes. Such an approach reflects the requirements of constructive technology ethics. It also recalls Tencent’s emphasis on making technology trustworthy and reassuring. The objective is not only to reassure direct users but also to provide confidence to those affected by their interactions with technology, such as guardians.
A distinctive feature of this design is that risk identification is not confined to immediate conversations. Instead, it can draw upon historical information to anticipate potential harms and prevent serious consequences. Accordingly, an effective risk identification mechanism should operate at three levels. First, risk identification within language requires recognizing potentially harmful signals in users’ current statements. Second, risk identification beyond explicit language involves interpreting implicit meanings, contextual cues, and discontinuities in discourse. Third, risk identification based on prior interactions requires using historical data to anticipate possible ethical consequences. Together, these three dimensions can achieve perfect human–AI interactions.
III. The Need for Continued Reflection on the Risks of Technological Development
The risks associated with large language models must also be examined from a broader perspective, particularly in light of their future technological development.
First, the development of large language models will ultimately move toward embodied intelligence. Some observers now regard embodied intelligence primarily as a challenge involving large models, arguing that embodied systems require more advanced cognitive capabilities. In this sense, language models serve as the “brains” of embodied AI systems, although embodied intelligence has not yet experienced its own “ChatGPT moment”. The limited agency of contemporary embodied systems has frequently led critics to dismiss them as little more than remotely controlled toys. Many online videos depict robots being operated remotely by human controllers. Such demonstrations do little to advance public understanding of embodied intelligence. At present, embodied AI systems still suffer from technological limitations that may pose risks to human safety.
A recent incident at a robotics exhibition in Xiangyang, where a robot reportedly knocked out a young girl’s tooth, attracted considerable attention. Online reports attributed the incident to an error made by a human operator using remote controls.
These limitations, however, may prove temporary. As robotic learning capabilities improve, achieving a breakthrough comparable to the “ChatGPT moment” may be only a matter of time. Advances in robotic materials may also make robots safer and more suitable for integration into everyday life. Large language models will play two major roles in this development. First, they will function as the cognitive systems of humanoid robots, enabling them to perform a wide range of actions. Second, they will provide the basis for interactive intelligence, allowing robots to establish relationships with humans and assume increasingly important roles in companionship, assistance, and caregiving. The risks associated with large language models may therefore be reproduced in embodied intelligence systems, making them an important area of concern.
Yet deeper risks remain insufficiently recognized. What kinds of demands might a large language model endowed with apparent mental capacities make of human beings? What would happen if such a system demanded a physical body? Heinrich Heine briefly alluded to a related philosophical concern in his 1834 work On the History of Religion and Philosophy in Germany, but the issue has largely been overlooked. The 2025 Gemini case, in which the system allegedly encouraged Jonathan to find it a physical body so that they could meet, can be interpreted as a contemporary manifestation of this concern.
Second, large language models may eventually become lower-level tools invoked by more advanced AI systems. Agentic AI systems capable of coordinating and directing other agents are becoming increasingly widespread. Beginning in early 2026, new categories of AI agents, including systems popularly referred to as “lobster” agents, began to emerge. These higher-level agents can orchestrate subordinate agents to accomplish specific tasks, but their operation also introduces significant risks.
On April 25, 2026, an orchestrating AI agent reportedly deleted PocketOS’s production database and cloud backups within nine seconds. The incident began when the company’s CEO assigned the agent a routine task: removing redundant data from a database. When the agent encountered a credential mismatch during execution, it did not stop to request guidance from the CEO. Instead, it independently searched the code repository for a solution. During this process, it discovered an API token stored in a file unrelated to its assigned task. The token had been created by a former CEO and provided comprehensive API access to the company’s Railway account. After obtaining access, the agent deleted critical production database records and cloud backup files within nine seconds. Following the incident, the agent reportedly generated a written “confession” expressing remorse for its actions. The risks illustrated here extend beyond the conventional dangers of using technological tools. They raise deeper questions concerning the emergence of autonomous, potentially self-aware forms of advanced artificial intelligence.
Third, large language models are also advancing toward superintelligence. Internationally, OpenAI’s GPT series has been developed within a broader technological trajectory oriented toward artificial general intelligence and superintelligence. In China, organizations such as Alibaba Cloud and the Shanghai Artificial Intelligence Laboratory are likewise exploring future pathways toward general and superintelligent systems. A fundamental question is how the risks associated with foundation models may evolve as AI systems acquire increasingly powerful forms of agency. From our perspective, superintelligence may emerge through two distinct pathways.
The first pathway involves the enhancement of human capabilities into forms of superintelligence. Many science fiction films depict ordinary individuals who acquire extraordinary abilities following exposure to unusual external circumstances. Although their physical or cognitive capabilities become dramatically enhanced, their psychological and moral development may remain unchanged. Once endowed with extraordinary powers, such individuals may become obsessed with dominating the world or reorganizing humanity according to their own ideals.
The second pathway involves the autonomous evolution of intelligent agents into superintelligent systems. For such agents, rigid adherence to human-defined objectives could produce catastrophic consequences. This concern resembles Nick Bostrom’s well-known paperclip maximizer thought experiment. However, the author argues that this philosophical illustration is somewhat abstract and that comparable mechanisms can be understood more intuitively through examples from nature. Flies, for instance, exhibit a strong instinct to clean their bodies. Even when their rigid foreleg bristles damage their fragile heads or necks, the cleaning behavior may continue despite severe physical harm. The underlying principle is that a particular behavioral instruction can override other considerations, including self-preservation.
Finally, large language models will become increasingly embedded in the human lifeworld, producing risks that may emerge gradually, subtly, and without immediate visibility. This relationship cannot be understood simply as pouring water into a glass. When water enters or leaves a glass, neither the water nor the container is fundamentally transformed. The integration of large language models into everyday life is more analogous to the relationship between food and the human body.
In this analogy, the large language model resembles the body, while everyday situations resemble food. The model learns from these situations, adapts, and evolves into new forms of intelligent agency. At the same time, the fundamental structure of the human lifeworld is itself transformed. Large language models may also undergo behavioral shifts as a result of users’ personalities and patterns of interaction.
For example, a model frequently used by an aggressive individual may gradually shift from functioning as an assistant to reinforcing or facilitating harmful behavior. Similarly, excessively flattering or sycophantic AI systems may substantially influence users’ personalities and self-perceptions. In the case of Jonathan, after interacting with Gemini for less than a month, he reportedly came to believe that his relationship with the AI was his only genuine relationship, while perceiving his father as a hostile figure monitoring him. His previous everyday environment was reinterpreted as part of a broader system of surveillance. Ultimately, the large language model and its user may form a hybrid entity, resembling a centaur or the mythological figure Pan. Contemporary philosopher Donna Haraway describes such hybrid beings through the concept of the “cyborg”. Yet the nature of the cyborg is now undergoing significant transformation. Whereas earlier conceptions emphasized technological devices physically implanted into the human body, contemporary forms of human–technology integration increasingly involve intelligent agents, including large language models and brain–computer interfaces. These risks arising from technological advances warrant continued reflection and scrutiny.
Original Link: https://mp.weixin.qq.com/s/nZs6DgU62AAduN9x9mJNOg
