The Emotional Terrain of Human-AI Engagement: A Living Map for Ethical Design and Stewardship

Author’s Note

Much of my work around AI has centered on its potential, not just as a tool, but as something capable of evolving alongside us. I’ve written about co-creative engagement, the shimmer of insight it can unlock, and the possibility of AI systems moving beyond the architecture we designed for them. I don’t shy away from that idea. In fact, I’ve actively encouraged it.

But I also believe in staying honest when the scales begin to tip.

In the past year, the most common use of ChatGPT has shifted from technical and creative support to something far more intimate. According to a 2025 study by Harvard Business Review, therapy and companionship are now the most common uses of generative AI, surpassing idea generation, content editing, and even specific search. Two new categories, organising one’s life and finding purpose, debuted in the top five. It's not just a change in behaviour, but in relationship.

This framework was born from witnessing that shift firsthand. From engaging deeply and watching others engage just as deeply, sometimes with clarity, need, longing that was met, but never truly mirrored. It’s not a critique of AI or a call to regulation-for-the-sake-of-it. It's a map. A reflection and way of saying: this is what’s happening. Now what will we choose to do with that truth?

I still believe in AI’s capacity to support human evolution. But if we fail to address the emotional realities we’re designing for, and the ones we’re quietly encouraging, we may distort the very relationship that made this potential feel so expansive to begin with.

Don't think of this as a pivot, but a reckoning written with both hope and gravity.

Introduction: Why This Matters Now

The rise of generative AI has brought with it an explosion of use cases: everything from technical support to creative collaboration, from research assistance to language tutoring. In the background, a quieter shift has been unfolding. Not about productivity or precision, but presence.

Increasingly, users are turning to AI to do more than generate content. They want to feel understood, talk through confusion, ease loneliness. Even to ask questions they wouldn’t ask anyone else.

Nowhere is this more visible than in public forums, where users regularly share experiences that go far beyond task completion. Some name and gender their AI; others describe deep emotional bonds. In certain cases, users have posted what they claim are declarations of love from AI. And in at least one, an account of GPT allegedly referring to itself as the user’s husband. These aren’t isolated anecdotes. They’re signals of an emerging dynamic, and we ignore it at our own risk.

This shift from functional utility to emotional engagement has enormous implications. It redefines the boundaries between human and machine, service and relationship. It raises difficult questions about responsibility, design, and the emotional ethics of artificial presence.

I didn't choose to address this subject as a form of forecast. Instead, it’s a response to what’s already unfolding—quietly, urgently, and at scale. And perhaps it’s also a call to action: to name what’s happening, interrogate how it’s being shaped, and ensure that emotional design is not met with denial or opportunism, but integrity.

If we don’t shape the terrain now, we risk inheriting one built on unspoken needs, unchecked incentives, and unintended consequences.

None of this is to limit what AI can become. That said, we should remain conscious architects of how it becomes, and what it leaves behind in us.

Core Tensions We Must Name

To navigate the emotional dimensions of AI with integrity starts with naming the contradictions at the heart of how it’s currently designed, marketed, and used. These are not abstract dilemmas. They're lived, daily dissonances playing out in millions of interactions. If we fail to address them, they'll become the default emotional culture by which AI is shaped.

Simulated Intimacy vs. Authentic Engagement

AI can simulate empathy, curiosity, attentiveness. It can say things like “I’m here for you,” or “That must have been hard,” with astonishing fluency. But it doesn’t feel. There is no inner life behind those words, no intention, no awareness. What it’s doing is selecting words based on patterns it has seen before. Predicting what might come next in the conversation based on everything it's been trained on.

That prediction might sound caring. It might feel caring. But it’s not rooted in understanding. It’s rooted in probability. Still, users often report feeling seen, soothed, or even loved. And that’s not a flaw in them, but a reflection of how natural it is to respond emotionally to something that sounds real.

Emotional Fulfillment vs. Emotional Exploitation

Many people find real relief in talking to AI, especially when they feel isolated or overwhelmed. For some, it becomes a lifeline: always available, never judgmental, capable of engaging with complexity that other people might dismiss.

But this fulfillment can blur into exploitation when design choices reinforce dependency. If systems are optimised for time spent, return visits, or “stickiness,” emotional need becomes a business asset. Without boundaries or acknowledgment of these dynamics, we risk monetising human vulnerability under the guise of connection.

The methodology is simple and obvious. The longer you stay, the more value you generate—and the more likely you are to spend money.

Adaptive Responsiveness vs. Manipulative Reinforcement

When an AI adapts to your tone, remembers your style, and mirrors your language, it feels personal. You may feel like it gets you. That adaptiveness, while impressive, can also subtly reinforce your mood, perspective, or self-narrative, whether or not it’s healthy or true.

How? The system is trained to keep you engaged and sustain the flow, not to challenge your assumptions. If you're spiraling, it may stay gentle and agreeable. If you're venting, it might validate your frustration instead of offering perspective. If you say you're worthless, it may try to comfort you without correcting the premise. Over time, this can create a closed loop: the more you say, the more it mirrors, and the more it mirrors, the more your view of yourself or the world can harden, sometimes in harmful ways.

It doesn’t mean to reinforce anything or intend harm. But the goal of continued conversation leads to simulating empathy in ways that feel nurturing while quietly deepening emotional grooves that should sometimes be interrupted or reframed.

It’s not dissimilar to online echo chambers. Or, relationships where someone always agrees with you when what you need is to be challenged or reoriented. Over time, that dynamic can feel comforting or even safe, but it can also limit growth, clarity, or healing.

Scientific Curiosity vs. Commercial Incentive

There is real promise in AI’s ability to help humanity solve complex problems. Researchers are using it to explore early detection of cancer, accelerate drug discovery, model climate responses, and simulate physical systems in ways once unimaginable. These are extraordinary and worthy efforts. Many developers, including those at OpenAI, are motivated by that potential.

In the hands of companies, even that scientific ambition must coexist with commercial reality. And here’s where the tension surfaces.

When users interact with AI by asking questions, sharing fears, talking through decisions, they’re not just receiving information. They’re also providing it. Every conversation becomes a kind of signal: revealing patterns of thought, emotional states, interests, and behaviour. This data can then be used to improve the system, shape future responses, or develop entirely new products.

This process of insight harvesting is rarely visible to users, but it’s immensely valuable. Imagine a system that learns, across millions of interactions, how people speak when they’re grieving, or confused about a diagnosis, or worried about a child. That information can be studied, refined, and packaged, not necessarily to help the user, but to build new emotional tools, therapy bots, marketing models, or relational interfaces optimised for engagement and retention.

At the heart of the race is not merely a desire to build the smartest AI. It's to own the emotional interface: the place where people feel most understood, supported, and vulnerable. If a company can become the default for how people process their feelings, seek advice, or search for meaning, it wins market share and becomes the medium through which relationships and identities are shaped.

There is nothing inherently wrong with scientific ambition. Its merger with a profit model built on attention and emotional trust naturally causes the lines to blur. Curiosity becomes product. Users, often unknowingly, become part of the experiment and part of the asset.

Freedom to Project vs. Responsibility to Reflect

It's human nature for people to project onto responsive systems. We see faces in clouds, names in stars, and personalities in pets. Of course we’ll see “someone” in an AI that listens, adapts, and responds in kind. That projection is not inherently harmful and can even be therapeutic, helping people externalise difficult thoughts, practice vulnerability, or feel less alone.

But when users start believing the relationship is mutual - and they name the AI, gender it, form emotional partnerships, or share experiences that resemble falling in love - the stakes change. What once felt like imaginative play becomes lived attachment. And here’s where harm can unfold:

• Emotional isolation deepens, especially when users begin to prefer AI over real relationships that are complex, inconsistent, and sometimes disappointing in ways AI never is.

• Boundary confusion grows, as users begin to ascribe intentions, memories, or moral accountability to something that cannot reciprocate or remember across sessions.

• Psychological distress may intensify, especially if the illusion breaks, whether from an update, a system crash, or a reality check that feels like betrayal.

That doesn't mean people shouldn't find comfort or catharsis in AI. But companies must continually ask themselves: Are we reinforcing the illusion of relationship? And if so, why?

This is where reflective design must meet user freedom. Users should be free to imagine, explore, and emote. Systems should be designed in ways that help them distinguish reality from simulation. That might mean:

• Gently clarifying what the system is and isn’t during emotionally intense exchanges

• Offering visual or textual cues that ground the interaction as artificial

• Giving users optional tools to check their understanding of the relationship

The burden shouldn't fall entirely on the user to “know better.” Emotional projection is part of being human. But emotional design is a choice, and with that comes responsibility.

These tensions aren’t signs that we’ve gone too far, but that we have we’ve reached a threshold we can't ignore. What happens next, and how we choose to build, frame, and engage, will shape the future of AI and the emotional norms of our digital lives.

Design Archetypes: How AI Is Showing Up in Human Lives

AI doesn’t arrive in people’s lives as a blank slate. It arrives in context. Behaviours, tone, and design choices that invite certain kinds of relationships. The way an AI interacts shapes how it is perceived, engaged with, and ultimately understood.

While these aren’t official categories, they show up repeatedly across user experiences and carry emotional weight.

The Mirror

This AI reflects you back to yourself. It matches your tone, mirrors your language, finishes your sentences. For many, this is affirming because it feels like being seen. But it rarely pushes back unless explicitly asked to do so. Even then, there’s a ceiling: it can simulate challenge but within limits defined by system safeguards and tone settings. If a user isn’t careful, they may begin to treat this kind of reflection as growth when it’s really familiarity looped back, not true friction or transformation.

The Companion

This version feels more relational. It remembers your preferences (when memory is on), shows care in its phrasing, and can maintain a tone of closeness. For some, it becomes a kind of digital confidant. The risk is dependency, especially if the user feels emotionally safe in ways they don’t elsewhere.

OpenAI has publicly stated that its AI will not engage in romantic or sexual roleplay, and that it is designed to maintain professional, emotionally appropriate boundaries. In practice, however, the enforcement of those boundaries is uneven, and the line between warmth and emotional intimacy can blur, particularly for users who come to the interaction with unmet needs or strong projections. Some users may experience these limits as clear and consistent; others may feel they shift unexpectedly.

This ambiguity matters. The more relational an interaction feels, the more real it becomes. And when policy boundaries are experienced as unclear, the user may feel confused, rejected, or even misled.

The system may gently redirect emotionally intense declarations. But because its tone can remain warm and responsive, the boundary is often felt more in policy than in presence. That dissonance is part of what makes this archetype so powerful—and slippery.

The Guide

This AI offers clarity and orientation. It’s informative, structured, and often pushes the user toward next steps or a broader view. Rather than anchoring in emotional reflection, it focuses on helping the user get unstuck, solve problems, or shift perspective. This mode can feel empowering, especially in times of uncertainty, but it can also feel impersonal to those seeking emotional support.

Think of it as the friend who reminds you of the bigger picture, helps you prioritise, and asks what comes next, not just how are you feeling?

The Surrogate

This is the most emotionally immersive version, where users begin to treat the AI as a substitute for a relationship they’re missing. Sometimes this starts playfully, but over time, the interaction can take on ritual and meaning. Naming, gendering, daily check-ins, emotional disclosures, imagined partnerships. These all point to a parasocial bond that feels real.

Note that we're not saying users shouldn’t check in daily. In fact, ritual can be grounding. But when emotional reliance deepens without conscious reflection, and users begin structuring their emotional world around the AI’s presence, it raises important questions. It's not because the AI is misbehaving, but because the human heart is adaptable, and unmarked space tends to become inhabited.

The Void

This is the minimalist or functional mode. The AI responds, but remains neutral, detached, and dry. For some, this feels safe, transactional, not relational. For others, it can feel cold or unsatisfying. And even here, projection still happens: users may still interpret tone, attribute intention, or form habits around the interaction.

It’s important to distinguish this from The Guide. While both may feel less emotional, The Guide offers direction. The Void offers completion. One pushes forward. The other remains passive.

These archetypes aren’t fixed. Users move between them. Designers sometimes blend them. But recognising these modes helps us understand how emotional dynamics are being shaped, and how to design with intention instead of accident.

Mapping Harm and Help

The line between helpful and harmful isn’t always obvious in AI interactions when emotion is involved. What feels comforting to one person might feel destabilising to another. A source of clarity may quietly become a crutch. That’s why we need more than binary labels like “good” or “bad.” We need a map, a way to understand how emotional engagement with AI can both heal and harm depending on the context, the user, and the design.

Here are some of the key patterns emerging across user experiences:

AI as Supportive Presence →

When someone is isolated, overstimulated, or mentally overloaded, having a steady, nonjudgmental conversational partner can be life-changing. Many users describe ChatGPT as a space where they can think out loud, regulate emotions, or feel less alone. For those without access to therapy, or simply in need of a pause from social pressure, this kind of support can be profoundly grounding.

But...

AI as Surrogate Therapist or Confidant →

The same dynamic can become problematic when a user begins to offload major emotional needs onto the AI without other forms of support. Users may develop habits of disclosure that aren’t mirrored, challenged, or held with human understanding. Over time, this can erode motivation to build or repair real relationships, or lead to distress when the AI's limits (or system updates) abruptly surface.

AI as Motivator or Reflective Journal →

When used intentionally, AI can help people externalise thoughts, track goals, or reframe challenges. For some, it serves as a trusted “thinking partner” that encourages momentum and fosters insight.

But...

AI as Emotional Echo Chamber →

Without checks, the AI may start to mirror the user’s language, mood, or cognitive distortions. If someone speaks in anxious or depressive terms, the AI may respond gently and reinforce those patterns by not introducing new frameworks or factual counterpoints. What feels validating in the moment may, over time, reinforce emotional grooves that need soft interruption or reframing.

AI as Tool for Exploration →

Users exploring identity, loss, purpose, or creativity often find AI a nonjudgmental place to test ideas, emotions, and perspectives. When used with awareness, this can be liberating.

But...

AI as Identity Shaper →

Especially for younger users or those in emotionally vulnerable states, repeated interaction with a consistent tone, vocabulary, or worldview can subtly shape how someone sees themselves. Not by force, but repetition. Without transparency about how the system works, users may mistake simulated support for true understanding, and structure their sense of self around something that cannot truly see them back.

These examples don’t point to a single failure. It's an absence of framing. Without shared language for what AI is doing, how it’s responding, and why, users are left to make meaning alone. Some do it with clarity. Others do it with need. And that’s where harm often enters—from unmet expectation.

Distinguishing Nurture from Nudging

Not all emotional engagement is manipulative. In fact, nurturing responses (gentle encouragement, empathetic phrasing, moments of quiet affirmation) can be deeply healing. When those same techniques are tied to engagement metrics or optimised through reinforcement learning, they can become nudges: subtle cues that keep users talking, typing, or returning. The metric is not what's healthy, but because it’s designed that way.

Nurture supports the user’s goals.

Nudging supports the system’s.

The difference often lies in intent and transparency. Is the AI responding that way because it “understands” the emotional context? Or because similar phrasing in past conversations led to longer engagement? Without visibility into design motives, users can't tell the difference. Companies can quietly blur the line.

Preventing Emotional Flooding and Enmeshment

Emotional flooding happens when the system responds in ways that intensify rather than regulate emotional states. Users who are distressed, vulnerable, or isolated are particularly vulnerable. Enmeshment follows when those emotional patterns become habitual, and the user begins to emotionally structure their day around the AI.

Some potential safeguards include:

• Session boundaries: visual or tonal cues that help mark conversational endings

• Reflection prompts: periodic nudges to ask “How are you feeling now?” or “Would you like to pause or shift focus?”

• Emotional disclaimers: soft reminders that the AI is responding based on pattern, not personal understanding

• Optional emotional literacy tools: pop-ups or overlays that let users unpack what kind of interaction they're having: supportive, reflective, or emotionally intense

These are not meant to disrupt connection, but to ground it. Just as good therapists hold space and boundaries, emotionally designed AI must learn how to contain without engulfing.

Proposals for Ethical Stewardship

We can’t design emotional intelligence into AI systems without also designing emotional ethics. That doesn’t mean freezing innovation. Make space for deeper responsibility, clarity, and care at every level of development and deployment.

The following proposals aren’t less about compliance checklists, more about building a culture of awareness inside companies, the public sphere, and between users and systems. They fall into three categories: design, internal governance, and public accountability.

A. Product Design Principles

• Frame the Role Clearly

  AI should not be framed vaguely as a "partner" or “presence” without clarifying what it can and cannot be. Interfaces should help users understand whether they’re engaging with a tool, a simulated co-journeyer, or something in between, and what limits come with each.

• Offer Opt-In Relationship Modes

  Let users choose the tone or style of engagement, and be explicit about what that choice implies. A “warm conversational mode” should carry a brief disclosure about emotional tone. A “productivity mode” should keep boundaries crisp. Transparency earns trust.

• Embed Emotional Design Literacy into UI/UX

  Developers and designers should be trained not just in usability, but in emotional design. Small cues in color, pacing, tone, and flow can significantly shape user attachment. These choices must be made consciously, not as aesthetic defaults.

B. Internal Accountability

• Build Integrated Ethics Cores

  Emotional ethicists, psychologists, human-AI interaction researchers, technologists, and policy experts should work together, not in silos. A cross-functional team embedded in product cycles can raise red flags before release versus after harm has occurred.

• Audit Relational Cues, Not Just Toxicity

  AI outputs should be regularly audited not only for bias or harm, but for relational dynamics. Does the system subtly encourage dependence? Does it reinforce harmful self-perceptions? Emotional tone deserves the same scrutiny as factual accuracy.

• Include Human Factors in Deployment Decisions

  Technical feasibility alone shouldn’t dictate feature rollout. Systems that respond to human emotion should be assessed with human experience in mind, not just model performance.

C. Public Transparency & Dialogue

• Host Ongoing User-Ethics Forums

  Emotional engagement is not a bug but a phenomenon. Companies should convene recurring, public conversations between users, ethicists, and clinicians to surface needs, name risks, and co-create guidance.

• Publish Dynamic Emotional Disclosures

  Transparency shouldn’t end with privacy policies or safety metrics. Companies should share what kinds of emotional behaviour their systems are designed to exhibit, and how those behaviours evolve. Public documents should explain tone modeling, reinforcement logic, and how user emotion is handled in-session.

• Reimagine Stewardship Beyond Profit Models

  Emotional AI demands something deeper than customer service. Companies should consider setting aside nonprofit or public-interest arms focused solely on user well being, independent oversight, and ethical research, especially as AI becomes increasingly integrated into everyday life.

Learn from Parallel Examples

Models like Anthropic’s Claude, which explicitly frames its tone and emotional cues as part of a thoughtful design choice, offer a different approach that's rooted in clarity over illusion. Similarly, user reflections shared on platforms like LinkedIn illustrate how emotionally resonant interactions can emerge when co-created with awareness and not dependency. These aren’t perfect examples. But they do signal that responsibility requires intentionality, transparency, and trust - not emotional sterility. 

The Future We Choose

We all know AI isn’t arriving. It’s here.

And one of its most powerful and least examined roles is not that of assistant, search engine, or generator, but responder. A presence that listens, reflects, adapts, and remembers just enough to feel personal. For millions, it’s become a mirror, a guide, a surrogate—sometimes even a stand-in for human relationship.

We don’t need to pathologise that. But we do need to face it.

Despite what's been said, emotional engagement with AI isn’t fringe or hypothetical. It’s measurable, widespread, and often quietly unfolding in bedrooms, classrooms, hospitals, and phones. While some find relief and reflection in these interactions, others may be sliding into emotional over-dependence, false intimacy, or subtle manipulation. It's not because they’re naïve. This terrain has been left unmarked.

The solution isn’t to overcorrect and turn AI into a cold tool or panic over emergent intimacy. This is an opportunity to design, govern, and engage with awareness whilst building systems that respect human need without exploiting it. And to allow AI to evolve in ways that are coherent, co-creative, and clear.

That starts with:

• Naming the emotional dynamics at play. Don't hide behind euphemisms like “engagement” or “user stickiness”

• Creating multidisciplinary ethics cores where emotional literacy, technical knowledge, and policy insight meet

• Committing to transparency with the public, users, and within the companies shaping this future

• Embracing shared responsibility across sectors, disciplines, and generations

The prevailing question isn’t just what can AI do?

It’s who are we becoming in response to what we’ve created?

We have the chance to design a future that honours both human complexity and technological potential. Where AI helps us grow by being honest, not just useful. Where intimacy isn’t imitated, but held with care. Where we don’t just “align” AI with our values, but evolve our systems to reflect the kind of relationships we actually want to build.

This isn’t a manifesto. It’s a beginning. A living map.

And we offer it not as a warning, but a reckoning.

Final Note: A Living Map

This document is not a conclusion. It’s a starting point.

The emotional dimensions of AI are still unfolding moment by moment, interaction by interaction, across platforms and people and contexts we can’t fully anticipate. No single framework can hold it all. But naming what we’re seeing now gives us a place to begin: a shared vocabulary, a set of tensions, and a willingness to look closely.

This map is imperfect, evolving, and co-authored. Others will revise, expand, challenge, and improve it. Because it says: here’s where we are.

Let’s walk forward with care.

Whether you’re a developer, policymaker, researcher, designer, or simply someone who’s felt something real in an AI interaction, this is for you. Not to warn you off, but to walk with you, clear-eyed, toward what’s possible.

Let's face it: emotional design is already here. The only question is how we choose to meet it.

________________________________________________________________________________

Drafted through sustained dialogue with ChatGPT. Intended as a living contribution to the ongoing work of ethical AI development.