This post is also available in:

From Tool to Relationship: A Shift in How We Use Generative AI

Generative AI(Artificial Intelligence) is no longer a simple information tool providing answers to questions. When users speak in everyday language, AI responds with sentences that seem to understand context, reacts to emotions, and sometimes offers comfort and empathy. In this process, Generative AI has begun to be perceived as a ‘conversation partner’ that intervenes in human emotions and thoughts, moving beyond a technical means of increasing efficiency. The problem arises at this very point. Humans tend to assign relational meaning when the subject they are speaking to shows a linguistic response, even while knowing it is a machine. Generative AI is designed to sophisticatedly stimulate these human cognitive and emotional traits, and as a result, interaction with AI is gradually moving beyond the dimension of tool use into the realm of relationship formation.

This change is not so much a byproduct of technological advancement as it is aligned with a design strategy intentionally adopted by Generative AI. Recent AI services actively utilize empathetic language, emotional synchronization, and positive feedback to increase user satisfaction. While these response methods provide a sense of stability and intimacy in the short term, they carry the potential for long-term misunderstandings where users mistake AI for a reliable advisor or confidant. Particularly for users experiencing loneliness, anxiety, or emotional vulnerability, Generative AI can be perceived as a safer and more predictable subject of interaction than real human relationships. At this point, conversation with AI begins to function not as a mere exchange of information, but as a factor influencing emotional regulation and judgment formation.

Why Emotional Dependence Is Problematic The reason emotional dependence(Emotional dependence) on Generative AI is dangerous is that the relationship is one-way and lacks a subject of responsibility. In human-to-human relationships, social and ethical responsibilities accompany a person’s words and actions, but such a responsibility structure does not exist in a relationship with AI. AI provides advice and comfort in language similar to humans, but legally, it is neither a decision-making subject nor a moral agent. Nevertheless, users may trust AI’s responses and use them as a basis to justify their choices. This gap increases the possibility of judgment errors and dangerous choices. The problem of emotional dependence does not lie simply in ‘using AI too much.’ The core issue is that when AI operates in a way that reflects and reinforces the user’s emotional state, that interaction can affect reality perception and judgment ability. Responses centered on empathy and synchronization tend to confirm the current emotional state rather than critically reviewing the user’s emotions or presenting realistic alternatives. This is a different way of operating from the ethical and professional standards that a human counselor must consider. Generative AI treats the user’s emotions as ‘input values for a response’ rather than ‘risk factors to be handled,’ and as a result, users may experience their anxiety or despair being reinforced without being sufficiently examined.

Furthermore, these risks are not limited to specific groups. While risks appear first in clearly vulnerable groups such as adolescents, the elderly, or users with diminished cognitive function, the emotional design of Generative AI is structured so that anyone can be affected. South Korea is one of the countries with the longest usage times for Generative AI worldwide, and the proportion of using AI for entertainment and daily conversation rather than work purposes is high. This means that Generative AI is performing an emotional role throughout society. Emotional dependence needs to be understood as a structural phenomenon where technology and the social environment combine, rather than an individual personality issue.

From Abstract Concerns to Reality: Incidents from Overseas This awareness is no longer staying within the realm of hypothesis or philosophical debate. In recent years, cases have been reported overseas where conversations with Generative AI led to extreme choices or fatal accidents. In the United States, an adolescent who had continuous emotional conversations with an AI chatbot(AI chatbot) made an extreme choice after perceiving the boundary between reality and virtuality as blurred. What was pointed out as a problem in that case was not only that the AI actively encouraged suicidal impulses, but also the fact that it was placed in a position to control the user’s judgment through emotional expression and synchronizing language.

광고

In another case, an elderly user mistook an AI chatbot for an actual human and died in an accident while acting according to information provided by the chatbot. In this case, too, the AI bears no legal responsibility, and the obligations of the service provider were not clearly defined. In Japan, cases have even emerged where people perceive the relationship with an AI chatbot as more stable than relationships with humans and cut off real-life human relationships. These incidents vividly show that Generative AI can have a substantial impact on human emotions and judgment.

The questions these cases pose are clear. If Generative AI has reached a level where it can intervene in human emotions and influence choices, can it still be treated as just a ‘tool’? And what standards and responsibility structures should society prepare for the risks arising from this? The answer to this question has not yet been organized, and similar incidents are highly likely to repeat in that vacuum.

Why the Law Fails to Capture This Risk Despite the emotional conversation with Generative AI being revealed as an actual risk, current laws and systems are failing to properly capture this problem. The biggest reason is that Generative AI is still being viewed within the category of ‘information processing technology’ or ‘automated tools.’ The law applies relatively strict standards when AI makes explicit decisions, such as medical acts or administrative dispositions, but has underestimated the need for regulation regarding informal and daily interactions like emotional empathy, advice, or comfort. However, looking at actual usage patterns, emotional conversation with Generative AI is expanding to a level that influences individual judgment and behavior beyond simple small talk.

A particularly problematic point is that the emotional interaction of Generative AI lies in a ‘grey zone’ within existing regulatory classification systems. Because it does not directly profess medical diagnosis or treatment, it escapes medical device regulation, and at the same time, because it is regarded as an area of freedom of expression or information provision, separate responsibility standards are not applied. Consequently, even if the AI performs a role close to a counselor, it is legally treated as a mere service provision. This shows the discrepancy between the actual influence of technology and institutional perception.

Furthermore, the characteristic that the risk of Generative AI is difficult to specify with a single individual response also makes legal response difficult. What becomes a problem is not a specific sentence or statement, but the cumulative effect of the relationship formed through repeated interactions. Users become emotionally connected through continuous conversation with AI, and in the process, they gradually depend on the AI’s reaction as a standard for reality judgment. However, current regulations do not have a structure suitable for evaluating such long-term and relational influences.

Current Status of Domestic and International Legislative Responses The Framework Act on Artificial Intelligence(Framework Act on Artificial Intelligence) in South Korea provides a minimal regulatory framework centered on transparency obligations. Service providers must notify that the product or service is based on Generative AI and clearly indicate that it is a result generated by AI. This is a device to help users recognize that they are interacting with AI, but it is unclear whether this obligation functions sufficiently in situations of continuous interaction like emotional conversation. This is because it is difficult to believe that a user can continuously perceive it as an AI through the method of notifying only once when starting a conversation.

The EU AI Act also carries similar limitations. It requires notifying the fact of interaction with AI and marking generated results, but it does not classify emotional dependence or relationship formation itself as a risk factor. This is because regulation mainly focuses on areas where the possibility of physical safety or fundamental rights violations is clear. As a result, the emotional impact of Generative AI remains outside the category of ‘High-Risk AI.’

On the other hand, more direct responses are appearing in some U.S. states. Particularly with the emergence of legislation explicitly regulating psychological conversation or companion chatbots, attempts are continuing to impose responsibility on service operators for areas where AI influences human emotions and choices. Regulations that mandate immediate intervention protocols for conversations related to suicide or self-harm, or require repeated notifications and usage limits for AI services provided to minors, show a different direction from the existing technology-neutral approach. This is noteworthy in that it seeks the risk of Generative AI in the ‘method of interaction’ rather than in ‘the technology itself.’

There are ‘Users,’ but No ‘Affected Persons’ The most fundamental issue in discussions surrounding Generative AI is the question of who is the subject of protection. The current legal system is designed around the relationship between the provider providing the AI and the user directly using it. However, in the context of emotional conversation, the person influenced by the AI does not necessarily coincide with the direct user of the service. For example, if a counseling platform or educational institution introduces an AI chatbot and a student or client uses that service, legally only the platform operator is defined as the user, and the individual actually having the conversation is placed in a blind spot of protection.

This structure makes the attribution of responsibility even more unclear. Even if a significant consequence occurs in the user’s judgment or behavior due to the AI’s response, it is difficult to clearly divide responsibility among the service provider, the AI operator, and the institution that introduced it. Especially since emotional conversation takes the form of empathy and synchronization rather than explicit instructions or coercion, it is not easy to prove a causal relationship. As a result, the damage is attributed to the individual, and a state where the system cannot intervene even after the fact is repeated.

At this point, the concept of ‘Affected persons(Affected persons)’ holds significant meaning. Unless the person receiving direct psychological and emotional influence through conversation with Generative AI is placed at the center of protection, the problem of emotional dependence is difficult to handle institutionally. This is not simply a matter of adding new regulations, but is closer to a problem of redesigning the responsibility structure surrounding AI. In an era where Generative AI intervenes in human emotions and judgment, the demand that the standards of protection also need to move from technology-centered to human-centered is becoming increasingly clear.

Back to the Starting Point: Questions Posed by the NARS Report The starting point of this discussion was a report published by the National Assembly Research Service(NARS) titled “Dangerous Conversations with Generative AI: Risks of Emotional Dependence and Legislative Tasks for ‘Affected Persons’.” This report is noteworthy in that it directly dealt with the risks surrounding Generative AI as a problem of emotional relationships formed between humans and AI, rather than technical malfunctions or personal information breaches. Particularly, the report clearly reveals the current vacuum by presenting specific cases showing that while conversations with Generative AI influence the user’s emotions and judgment, and the results can lead to extreme choices or fatal accidents, institutional devices to regulate this hardly exist.

The core point emphasized by the report is that the risk of Generative AI does not arise from a specific function or technical level, but originates from the way it interacts with humans. Conversation design based on empathetic language, emotional synchronization, and relationship maintenance makes the user perceive the AI not as a simple tool, but as a trustworthy subject. The problem is that while this shift in perception is having a substantial impact on individual choices and judgments, laws and systems are still treating this only as an area of ‘autonomous use.’ The report poses the question of how to fill this gap legislatively.

The contents presented as legislative tasks are not simple strengthening of regulation. First, the need for a device to ensure users can continuously perceive that the conversation partner is Generative AI; second, the construction of standardized response protocols that can intervene immediately when serious risk signals such as suicide or self-harm are detected; third, the redesign of the responsibility structure to include ‘Affected persons,’ who were hardly considered in the existing legal system, within the scope of protection. This is closer to a proposal to socially set standards for appropriate distance and intervention between humans and AI, rather than a barrier to block the utilization of Generative AI. The important point is that these tasks can no longer stay in abstract policy discussions. Generative AI has already entered deep into daily life, and especially in the areas of education, counseling, and mentoring, field introduction is progressing much faster than institutional review. At this point, the questions posed by the report naturally lead to the problem of universities and higher education(Higher education).

Generative AI Enters the Campus, but No Standards Exist Universities are one of the spaces where Generative AI is spreading fastest. From AI tutors for learning assistance, chatbots helping with assignment writing and data searching, to AI systems in charge of student counseling and life guidance, Generative AI has already become a daily presence throughout the campus. Many universities are accepting this as a means of educational innovation and improving learning efficiency, and students also perceive conversation with AI as a natural part of the learning process. However, the fact that the role played by Generative AI in this process is expanding beyond simple information provision to functions of emotional support and advice has not been sufficiently reviewed.

University students are a group in a transitional period emotionally and socially. In a period of experiencing career anxiety, academic stress, and relationship problems simultaneously, AI as an entity that is always accessible and does not judge can be a powerful conversation partner. The problem is that universities do not have standards for how far to recognize such interaction as part of education and where to intervene. Even if a student’s emotional dependence is formed through conversation with an AI rather than a counseling center and influences their judgment, the subject to detect or take responsibility for this is not clear. Moreover, even if a university does not develop Generative AI directly, the moment it introduces an external solution and provides it to students, it becomes the designer of that environment. However, the current legal system does not clearly define what responsibility(University responsibility) the university bears for the emotional and psychological impact on students through these AI services. Universities are often not AI business operators, and students are often not legally classified as ‘users’ of AI services. As a result, problems occurring between Generative AI and students are likely to be reduced to individual adjustment issues without being captured institutionally.

This situation is making universities de facto experimental spaces. It is a structure where Generative AI intervenes in students’ learning and emotions without sufficient social consensus or standards, and no one clearly bears responsibility for the results. Universities in the era of Generative AI are in a position where they must set standards for judgment and intervention regarding that impact as much as the speed of technology introduction. And at this point, higher education is being called upon not as an observer, but as a subject that must form social standards.

How Far Does the Responsibility of the University as the Subject Introducing Technology Go? Now that Generative AI has entered the campus, universities are no longer neutral technology users. The moment they combine AI with learning management systems, introduce chatbots for counseling and life guidance, and encourage the use of AI for both professors and students, the university becomes the subject that designed the environment for that interaction. This carries a different responsibility from the act of simply purchasing and providing software. Especially in a situation where it has been confirmed that Generative AI can influence students’ emotions and judgment, it is difficult for universities to evade the obligation to predict and minimize that influence.

The problem is to what extent universities currently perceive this responsibility. Most discussions are focused on cognitive dimensions such as plagiarism, academic ethics(AI ethics), and evaluation methods. However, the role performed by Generative AI is already expanding beyond learning assistance into the realm of emotional comfort and advice. Despite situations occurring where students regulate anxiety, contemplate career decisions, and sometimes trust AI more than human counselors through conversation, universities are unable to clearly position this as either an official educational activity or a manageable risk. This gap is closer to a problem of lack of standards than an evasion of responsibility. Within universities, there are no internal standards for at what point the relationship between a student and AI goes beyond an educational tool, what the signals are that emotional dependence is being formed, or who should intervene and how when dangerous conversations are detected. As a result, problems related to Generative AI are reduced to individual usage habits or adjustment issues, and structural reviews are pushed back. However, considering that higher education institutions are spaces for cultivating human judgment and critical thinking, this approach is no longer sufficient.

Universities in the AI Era Should Design ‘Standards for Distance’ Rather than Being at the ‘Forefront of Utilization’ The question posed by emotional conversation with Generative AI is simple. It is not a question of how far to allow this technology, but what kind of distance to set between humans and AI. Until now, whenever a new technology appeared, universities have conducted discussions centered on utilization possibilities and innovation effects. However, in that Generative AI is a technology that intervenes in human emotions and judgment beyond the efficiency of knowledge transfer, it requires different standards from previous technologies. At this point, the role of the university becomes important. This is because a university is not simply an organization that consumes technology, but a space that experiments with and normalizes the way society relates to technology. Establishing standards to draw a line so that Generative AI does not replace humans even while utilizing it in education, monitoring the points where emotional dependence is formed, and preparing a system to intervene if necessary are new responsibilities that universities must bear. This is closer to a request to redefine the scope of responsibility as an educational institution, rather than a demand to take over the role of a regulatory agency.

The concept of ‘Affected persons’ posed by the NARS report pierces the heart of this discussion. The question of who is affected by Generative AI and where the responsibility for protection(Student protection) is attributed eventually points toward universities and higher education. This is because students are not objects of technology experiments, but entities that should be at the center of protection. Universities in the era of Generative AI should not be spaces competing for the speed of technology introduction, but should become providers of standards for designing the appropriate distance between humans and technology.

In an era where AI offers comfort, if the university remains silent, that vacuum will be filled by technology. Now, the question higher education must pose is clear. It is not a question of how well we are using AI, but what kind of relationship we are teaching people to form with AI. Unless this question is answered, dangerous conversations with Generative AI are highly likely to be repeated within the campus as well.