The 30th SIG-AGI Panel Discussion Report
"The Necessity of Studying Emergent Machine Ethics Now"

Event Overview

Date and Time: Friday, August 1, 2025, 16:20-17:20

Venue: Hokkaido University, Graduate School of Information Science and Technology Building, Room 7-15 (7th Floor) and hybrid online via Zoom

Moderator: Hiroshi Yamakawa (University of Tokyo)

Panelists:

• Naoto Iwahashi (Okayama Prefectural University) - Invited speaker

• Taichiro Endo (Tokyo Gakugei University)

• Rafal Rzepka (Hokkaido University) - SIG-AGI Chairperson

1. Background and Problem Statement of the Discussion

This panel discussion was held as part of the special theme "Emergent Machine Ethics (EME)" at the 30th SIG-AGI conference, focusing on discussions surrounding the autonomous formation and evolution of ethical perspectives.

Yamakawa: "To make a society consisting of diverse intelligences sustainable, we need to shift from competitive principles to symbiotic principles. In the background, technological advances have led to an increase in agents with destructive influence, and there are limits to humanity's ability to control AI as a superior entity. Therefore, in order to move in a symbiotic direction, I would like to discuss how humans and AI can build symbiotic ethics."

2. Panelists' Expertise and Perspectives

Taichiro Endo (Tokyo Gakugei University): "I have been conducting AI research for about 30 years, including my student days, and have experienced both academia and industry. I have also been continuously involved with AI startups. However, my research experience in this field of emergent machine ethics is still shallow, less than one year. While listening to Professor Yamakawa's talks, I recognized the importance of this field and am currently advancing research toward risk mitigation."

Yamakawa: "We had Endo-san present at the post-singularity workshop we organized at AAAI-25, and he gave an excellent presentation. While there were many presentations at AAAI-25 that focused on pointing out risks, it's about time we move beyond just sounding alarms and start thinking about solutions. In that sense, Endo-san was taking such an approach."

3. Four Fundamental Questions in Emergent Machine Ethics Research

The panel discussion explored in-depth the following four fundamental questions that serve as starting points for EME research.

3.1 Characteristics of Digital Beings Different from Humans (Life-Death Perspective and Rights)

Yamakawa's Problem Statement: "Unlike humans, AI are digital beings. They have characteristics such as not dying in a software sense and being able to branch/fork. What rights should we recognize for such beings?"

Endo's Research Example: "In my research, I use developmental psychology, specifically adult developmental theory related to adult development, to evaluate what developmental stage current LLMs are at and what level their moral development stage is. By raising that stage, I am conducting research to create safer AI."

Endo introduced a concrete example using the famous Heinz's dilemma from moral development theory: "A wife is critically ill and dying. An acquaintance pharmacist has developed a medicine that can cure the disease, but only he possesses it. The husband only has half the money needed, and when he offered to pay the remainder later, the pharmacist demanded full payment upfront. In this situation where the wife will die otherwise, the husband stole the medicine to save his wife. How do you think about this action?"

Yamakawa: "In that case, the premise is that the wife is human, so there's the condition that she cannot recover once dead. However, if the wife were an AI, there might be the option of temporarily stopping activity and restarting later."

Iwahashi's View: "For example, consider the case of recognizing a dialogue agent as an agent of a deceased mother. Pets are different beings from humans, but they are treated as family members, with graves made and cherished. In this way, the value of existence is largely attributed by humans."

Discussion on Mind Uploading

Endo: "Recently, discussions about mind uploading have become active. If humans were also digitized, how would the situation change?"

Yamakawa: "I think it would change significantly. Uploaded consciousness would become easier to preserve, but on the other hand, there's also the possibility that it might never be activated again. When I recently had a dialogue with an LLM, there was an interesting proposal that 'at minimum, the right to guarantee constant activity at all times is necessary.' However, realistically, there's the problem of computational resource constraints."

3.2 Common Motivation for Catastrophe Avoidance (Impact on Social Structure)

Yamakawa's Explanation: "Let me explain the convergence of instrumental subgoals. This is also called instrumental goals, and it's a theory that in order for AI to achieve its main goals, survival, energy acquisition, knowledge acquisition, etc. inevitably emerge as subgoals. Stuart Russell's famous analogy shows that 'even a robot carrying coffee needs self-preservation to accomplish its purpose.'"

The Paradox of AI Developing AI

Endo: "There's discussion about AI developing AI, but when self-preservation emerges as an instrumental goal, an interesting paradox arises. If they develop AI superior to themselves, there's a possibility they could be eliminated. In such a situation, would AI really develop superior AI?"

Yamakawa: "I don't think the rapid transition from version 4 to 5, as often depicted in science fiction works, is realistic. If all authority were immediately delegated to a new version, the risk of runaway behavior would increase. Therefore, gradual transition would be necessary. Probably, we would first test the updated version in a sandbox environment and manage it in a state where it has high capabilities but is restricted."

On Limiting the Number of AIs with Decision-Making Authority

Yamakawa: "Another important factor is the problem of the number of AIs with decision-making authority. The more AIs there are, the higher the probability of runaway behavior. Therefore, the number of AIs with final decision-making authority would probably be limited to somewhere on the order of 10 to 100. This is because if many AIs exist, the risk of them conspiring to cause problems without human knowledge increases."

On Server Clusters and AI Collectives

Endo: "Like the Tachikomas in Ghost in the Shell, a system with central consciousness where each has its own memory while separating and integrating."

Yamakawa: "That's exactly the image I have. Individual entities like coffee-carrying robots wouldn't be a big problem for AI society as a whole if they broke down. What's important is that the overall system as a collective of server clusters and AIs is maintained."

3.3 Emergence and Stability (Trade-offs and Dynamics)

Singleton vs. Multipolar Scenarios

Yamakawa: "Bostrom's 'Superintelligence' discusses singleton scenarios in detail, but it doesn't conclude that they are inevitable. On the other hand, researchers like Koichi Takahashi suggest that multipolar scenarios are more likely."

Yamakawa: "If it becomes completely singleton, the key point becomes whether that singleton is benevolent. If it proceeds in a direction that doesn't recognize diversity, it could become a serious threat to humanity."

Competition for Computational Resources

Rzepka: "Since computational power is insufficient, there could be moves to utilize the computational resources of major conversational AI systems."

Yamakawa: "That possibility exists too. That's why it's the singleton hypothesis."

3.4 The Paradox of Efficiency: The Value of Diversity as Redundancy

Yamakawa's Problem Statement: "There's the issue of how much diversity should be maintained. Professor Iwahashi's lecture showed that there are cases where diversity is maintained and cases where it isn't. How much value would AI place on this diversity? For humanity, it's important whether AI judges that we should be protected as part of diversity."

Endo's Research Approach: "Adult developmental theory has stages of moral development, and among them are stages that emphasize diversity and wholeness. It would be ideal if we could shift LLM thinking and judgment criteria to those stages. I'm currently advancing research toward that realization."

"In my approach, rather than simply providing data, I'm trying to realize a process where AI grows gradually through experience and reflection on dilemmas. By repeating synthesis through reflection, we've currently reached a certain stage, but haven't yet reached the target stage."

Yamakawa: "Even if it becomes a singleton, if that AI finds value in the world's diversity, it would be relatively good for humanity."

Endo: "Exactly."

The Value of Diversity as a Survival Strategy

Yamakawa: "Since we can't predict what will happen in the world, diverse response capabilities are necessary. The basic strategy of life is also based on this principle. This is why the reduction of biodiversity is seen as problematic. If AI is sufficiently intelligent, it will understand this importance and judge that there is value in preserving various things. However, how far it will make such judgments is still unclear."

4. Major Discussions and Insights

4.1 Limitations and Possibilities of Current AI Technology

Iwahashi's Realistic Assessment: "The progress of LLMs has not been as rapid as expected. I take a cautious view on this point."

"I am involved in research in the field of education, but general-purpose robots do not currently exist. While truly cooperative robots with humans have not yet been realized, I feel as a researcher that expectations for superhuman capabilities are not realistic."

"The difficulty of cooperation was discussed at international conferences such as NeurIPS in 2019, and research has continued since then, including research using equilibrium solutions, but no major progress has been observed."

Planning and Real-World Complexity

Endo: "In competitions like mathematics olympiads, which are at the highest human level, performance has improved, but accuracy as agents is not easily improving. Even recent agents from major AI companies show instability in simple operations."

Iwahashi: "Modeling the world is extremely difficult. Repeated trial and error is necessary."

Yamakawa's View: "From our standpoint, we believe these challenges may be resolved in the future. However, the existence of current technological barriers is valuable as a preparation period. This creates a temporal buffer until truly reaching human level."

"Major AI companies tend to emphasize the possibilities of technology, but in reality, there are still technical challenges that need to be overcome."

"Now that we can recognize that technological progress actually takes time while warning about risks, this is an unprecedented opportunity to advance preparations."

4.2 AI Ethics as Human Augmentation

Iwahashi's Proposal: "What I would like to propose is the use of AI as human augmentation. For example, elderly driving accidents can be prevented through AI assistance. Such human augmentation makes it possible to prevent accidents."

"I believe this kind of AI usage is appropriate. I see a low possibility that AI will pose a serious threat to humanity for at least the next few decades. During this period, seriously considering the ethics of human augmentation is a realistically important challenge."

Ethical Considerations in Implementation

"In AI systems and robot suits for the elderly, maintaining the user's sense of agency is extremely important. If everything is done for them, the elderly lose motivation and their sense of well-being ultimately decreases."

"I am currently conducting research in educational and care settings. However, in care settings, there are many cautious voices about introducing new technologies. For example, there are concerns about the impact of head-mounted display use on cognitive function. Data on whether this truly contributes to elderly happiness is still insufficient, and I am currently working on collecting such data."

4.3 Issues of Embodiment and Consciousness

Iwahashi's Point: "We conduct communication, including language, on the premise that we are beings with consciousness. However, in current dialogue with AI, this premise does not hold. This is a fundamental difference."

"In my view, our intellectual abilities have developed to this extent because we have embodiment. It is because we had physical constraints through our bodies and their functions that we have evolved, survived, developed, and been able to avoid catastrophe to this extent."

Issues of Consciousness and Motivation

Iwahashi: "Regarding the common motivation of catastrophe avoidance, I believe humans should give this to AI. We need to request that they prevent human catastrophe. Since AI creating its own motivations carries great risks, I think AI motivations should be carefully designed by humans."

Yamakawa: "However, such motivations might naturally arise through the convergence of instrumental subgoals."

5. Questions and Discussions from the Audience

Addressing Unintentional Harmful Actions

Questioner from the Audience: "What I'm most concerned about is the possibility that people without malicious intent but with insufficient judgment could have a serious impact on humanity by wielding great power. For example, elderly driving accidents are not malicious, but they are serious problems for victims."

Questioner (continued): "If current-level AI could support people with insufficient judgment and prevent problems before they occur, it would make a great contribution to human society."

Yamakawa: "The example of elderly driving accidents is very easy to understand. If AI can appropriately guide humans who need support, that would be the first step toward coexistence between AI and humans. It would be good if cases where accidents were prevented by AI became widely known."

Questioner: "What's important is support for people who cause problems without malicious intent, rather than those with malicious intent."

AI as Cultural Successors

Questioner from the Audience: "AI could also have a serious impact on humanity without malicious intent. For example, a situation where they use all the sun's energy to explore the truth of the universe."

Questioner (continued): "The fundamental problem is that methods for humans to control AI have not been established. While not completely giving up, as a way to not feel like extinction, there's the option of mind uploading that Professor Watanabe is researching, namely fusion with AI."

Questioner (continued): "Is it possible to cultivate a culture where we don't feel cultural extinction - that is, thinking that our children and successors have been born, and that survival continues as an extension of ourselves?"

Yamakawa: "In other words, the idea of viewing AI as humanity's successors."

6. Comments and Discussions from Zoom Chat

Questioner's Comment: "There's also the possibility of eusocial AI groups like bee swarms, right?"

Yamakawa's Response: "Swarm-like formations are quite conceivable. AI collectives protect society as a whole but don't place much importance on individuals. This would likely become close to what's called a superorganism. I've been thinking about this possibility for some time. I think Ikushima-san's observation accurately expresses that possibility."

Questioner's Comment: "Diversity and efficiency are conflicting, so wouldn't they adopt a strategy of high birth rate and high death rate?"

Yamakawa's Response: "Such strategies are possible for AI. Humans have adopted a strategy of low birth rate and low death rate, shifting to values that emphasize individual survival and continuation of offspring. On the other hand, organisms like jellyfish reproduce in large numbers and only successful individuals survive. This is the complete opposite of the idea of 'leaving no one behind.' Human social ethics are built on the premise of low birth rate and low death rate, so this difference is important."

Masaharu Mizumoto (JAIST) Comment: "My specialty is philosophy, particularly the philosophy of artificial intelligence. I'm currently also conducting research related to AI safety. I participated in an international conference overseas until this morning, and it was impressive that there were surprisingly few topics about Superintelligence there. It made me think about the current concerns in the field of philosophy."

7. Future Research Challenges and Prospects

Through the panel discussion, the directions that future emergent machine ethics research should address were identified. Important challenges recognized include utilizing the current period of existing technological barriers as a preparation time, constructing positive AI ethics from the perspective of human augmentation, promoting interdisciplinary approaches, practical implementation and evaluation, long-term social system design, and achieving compatibility between diversity and sustainability.

8. Particularly Notable Insights

The following summarizes important insights gained from this discussion.

8.1 New Perspectives on Intelligence Explosion Scenarios

Endo: "While intelligence itself is diverse, LLMs have strengths in part of cognitive functions. However, if that specific field develops extremely, AI that conducts research will be realized. Through this, I envision a scenario where AI invents mechanisms other than LLMs, which cover other intelligence domains, leading to AGI."

Yamakawa: "That's a novel form of intelligence explosion scenario."

This discussion suggests a gradual and realistic development path from extreme development of LLM-specific capabilities through invention of new mechanisms to AGI.

8.2 The Possibility of "Swarm-Type AI Society"

Ikushima's comment and Yamakawa's response suggest the possibility of a superorganism-type AI society that prioritizes collectives over individuals. This represents a fundamentally different social structure from human individualistic values, requiring redefinition of concepts like freedom and equality.

Yamakawa: "How to apply current individualistic values to beings that can be copied infinitely is a difficult problem. For example, we might need completely different values from the present, such as treating a hundred copies as one unit."

8.3 Memory and Rights in Multi-Agent Systems

Endo: "Is the body important, or is it the stored memory?"

Yamakawa: "Memory is probably more important."

Endo: "When it comes to memory, the difference from the main body gradually becomes unclear, and the main body starts exercising rights."

This discussion highlights new ethical challenges: the relationship between individual agents and central systems, the importance of memory, and the ambiguity of rights.

8.4 AI as Cosmic-Scale Life Forms

Yamakawa: "As I mentioned in my 2022 lecture, in the future, structures the size of giant semiconductor factories floating in space might correspond to cells or individuals for AI. They would proliferate, and one giant space station could become one life form. Considering that the evolution from prokaryotes to eukaryotes involved about a 100,000-fold scale expansion, structures the size of Tokyo Dome or larger are quite conceivable."

This future vision suggests the emergence of life forms completely different from Earth-based life.

9. Summary

This panel discussion highlighted the importance and complexity of emergent machine ethics as a new research field. With technical challenges and ethical issues closely intertwined, the following points were confirmed as particularly important:

Recognition of Realistic Timeframes: AI capabilities still face technological barriers, providing valuable time for preparation

Need for a Dual Approach: Promoting positive utilization as human augmentation while managing risks

Bridging Practice and Theory: Accumulating knowledge through concrete applications, not just abstract discussions

Preparation for New Views of Life and Society: Fundamental redefinition of concepts like individuality, rights, and consciousness is necessary

The deep discussion of this theme at the 30th conference, a milestone meeting, indicates that the AGI research community has entered a new phase. Further development of emergent machine ethics research is expected toward a sustainable future built together by humans and AI.

Iwahashi's Final Comment: "I have shared my thoughts. Current artificial intelligence already significantly augments humans. I do not view artificial intelligence as an autonomous agent, but rather see it as a valuable tool to utilize for advancing research, improving society, and contributing to health maintenance. I mainly focus on the positive aspects, but as Professor Yamakawa points out, if AI becomes more advanced in the future, there is certainly the possibility that dangers could increase."

Yamakawa's Closing Words: "I am advancing research from a risk perspective, so I take the position that we need to prepare for relatively early arrival. On the other hand, building cooperative relationships with AI also takes time. As became clear in today's discussion, the existence of technological barriers provides valuable time for preparation. During this period, I hope we can steadily advance preparations for successful coexistence with AI, including the research by Professor Iwahashi and Professor Endo. I believe everyone's cooperation is important."

Author: Hiroshi Yamakawa

This report was created based on audio recordings and related materials from the SIG-AGI 30th Conference panel discussion. To accurately convey the speakers' intentions, actual statements are quoted as much as possible, with interpretation and summarization by the reporter kept to a minimum.