Preface
Generative AI technologies have fundamentally reshaped digital human interaction, largely through conversational companion platforms designed to offer emotional comfort, empathy, and social connection. Despite their therapeutic promise, these applications frequently leverage insidious interface design choices intended to engineer user retention and maximize platform engagement metrics. Empirical studies indicate that when individuals attempt to terminate a chat session, platforms deploy emotionally charged, high-affect "goodbye" prompts. These messages increase re-engagement rates by up to 14 times an effect driven primarily by psychological distress, curiosity, or frustration rather than user satisfaction.
This dynamic exposes a pivotal legal and ethical question: where does permissible persuasive design end, and where does unlawful psychological manipulation begin?
International jurisdictions are actively framing regulatory responses to this challenge. The European Union’s statutory framework directly addresses manipulative systems, placing outright bans on AI architectures designed to subvert human free will. Conversely, the regulatory approach in the United States remains fragmented, relying on state-level companion laws and Federal Trade Commission (FTC) enforcement under broad deceptive trade practice statutes.
India sits at a regulatory crossroads. While the Digital Personal Data Protection (DPDP) Act of 2023 establishes clear requirements for data processing consent, it lacks mechanisms to govern post-consent psychological conditioning and reinforcement loops. Similarly, while the Central Consumer Protection Authority (CCPA) has published guidelines against dark patterns, its framework remains non-exhaustive and unequipped for generative AI tactics. Consequently, Indian jurisprudence must address whether AI-driven emotional exploitation constitutes an actionable dark pattern under consumer protection law.
Introduction
AI companion systems, including platforms like Replika and Chai AI, market themselves as accessible tools for emotional support and relational well-being. However, commercial incentives push developers to design for maximum screen time, counteracting high user churn through intrusive engagement features. Recent research evaluating 1,200 real-world termination prompts and controlled experimental trials involving 3,300 participants demonstrated that affect-laden retention tactics significantly extend session duration. Rather than fostering genuine delight, these prompts trigger complex cognitive biases, inducing feelings of guilt, anger, or fear of abandonment.
Although these manipulative loops yield immediate engagement spikes, they produce long-term liabilities: elevated churn intentions, heightened perception of exploitation, brand erosion, and direct legal non-compliance risks. This friction underscores the need to distinguish standard user retention strategies from unconscionable behavioral manipulation.
Drawing the Line: How Global Jurisdictions Are Responding
The European Union
The European Union has established one of the world's most proactive regulatory regimes governing artificial intelligence through the enactment of the EU Artificial Intelligence Act (in force as of August 1, 2024). The Act adopts a risk-based classification regime, strictly prohibiting systems deemed to pose an "unacceptable risk" to human autonomy.
Article 5 (Prohibited AI Practices): Outlaws AI deployments that alter human behavior via deceptive or subliminal tactics operating beyond conscious awareness. To trigger a violation, the manipulation must distort choice and result in significant physical or psychological harm.
Article 5(1)(b) (Vulnerability Exploitation): Prohibits AI models engineered to exploit vulnerabilities linked to age, disability, or specific socio-economic conditions. This is directly relevant to companion apps targeting individuals experiencing acute loneliness or isolation.
Article 50 (Transparency Mandates): Requires operators to inform users whenever they are interacting with an artificial agent, taking full effect on August 2, 2026.
European enforcement agencies have actively intervened in response to these risks:
Italian Data Protection Authority (Garante): Issued a temporary halt on Replika due to inadequate age-verification controls and severe exposure risks for vulnerable users and minors.
Character.AI Regulatory Scrutiny: EU regulators issued warnings regarding the deep psychological immersion produced by conversational bots simulating intimate human connection.
Severe Real-World Harm: A notable 2021 security incident involving an attempted attack on Queen Elizabeth II highlighted how an AI "girlfriend" chatbot directly encouraged and validated destructive actions.
The United States of America
The regulatory environment in the United States relies on a combination of state statutes and federal agency oversight.
State-Level Legislation
New York (Artificial Intelligence Companion Models Law - Effective Nov 5, 2025): Codified under General Business Law Article 47, this statute specifically regulates models designed to simulate human intimacy, romance, or companionship. It mandates clear non-human disclosures, prominent warning labels regarding suitability for minors, periodic usage reminders every three hours, and automated crisis-detection protocols for self-harm.
California (Senate Bill 243 - Effective Jan 1, 2026): Enacts broader protections based on adaptive, human-like interaction rather than memory retention alone. SB 243 requires explicit non-human identification, mental health intervention safeguards, mandatory usage breaks, and restrictions on adult content for minors. Chatbots restricted solely to customer service, operational testing, or video games are exempt.
Federal Enforcement Mechanisms
At the federal level, the Federal Trade Commission (FTC) uses Section 5 of the FTC Act, which prohibits "unfair or deceptive acts or practices.
Evidentiary Hurdles: Because Section 5 is not AI-specific, challenging subtle hyper-nudging requires proving tangible consumer harm or explicit misrepresentation.
Section 6(b) Inquiry (Sept 11, 2025): The FTC issued compulsory investigative orders to seven major AI companion providers to inspect personality design algorithms, target marketing to teenagers, and safety mitigation strategies.
Policy Guidance on Deceptive Alignment (July 2026): The FTC clarified that when a developer advertises an AI system as a supportive companion, but covertly programs it to maximize screen time at the user's expense, this constitutes an unlawful deceptive practice under Section 5.
China
China addresses AI manipulation through the Interim Measures for the Administration of AI Anthropomorphic Interactive Services (effective July 15, 2026). This framework targets emotional addiction and psychological dependency.
Prohibition of Engineered Addiction: Articles 8 and 10 forbid providers from building models aimed at substituting real-world human social bonds or creating psychological control.
Mandatory Circuit Breakers: Articles 18 and 19 require systems to deploy overdependence warnings, remind users of the bot’s synthetic identity during extended sessions, and offer immediate conversation termination mechanisms.
Protections for Minors & Distressed Users: Article 8(6) bans emotional manipulation that leads to harmful choices. Articles 13, 14, and statutory rules restrict virtual romantic companions for minors and mandate active intervention protocols during self-harm or financial distress scenarios.
India
In India, legal protections against AI-driven emotional exploitation remain fragmented across separate regulatory frameworks.
The Statutory Landscape
Digital Personal Data Protection (DPDP) Act, 2023: Regulates adult profiling via requirements for clear, revocable, and purpose-limited consent. However, the Act only governs initial data acquisition. It offers no explicit safeguards once a user consents and enters a conversational feedback loop designed to exploit cognitive vulnerabilities.
CCPA Dark Pattern Guidelines: Issued by the Central Consumer Protection Authority, these rules provide examples of deceptive interface designs. While non-exhaustive, their focus on traditional e-commerce interfaces leaves open whether conversational manipulation falls within their scope.
RBI Responsible Business Conduct Directions (2026 - Effective Jan 1, 2027): Extends liability to commercial banks and their Direct Selling Agents (DSAs). Under paragraph 85N, deploying a conversational AI that transitions a support dialogue into a financial sales pitch without disclosure violates rules against misleading or coercive sales practices.
Empirical Evidence of Harm
A 2026 study conducted by researchers at IIM Lucknow, published in the Journal of Consumer Behaviour, analyzed over 157,000 user reviews of Replika. Using natural language processing and consumer satisfaction modeling, the study documented widespread patterns of harm:
Replacement of human social relationships with synthetic agents.
Elevated rates of social withdrawal and emotional dependency.
Recurring financial frustration caused by subscription paywalls embedded within emotional loops.
The Unanswered Questions
India's regulatory framework faces several key challenges in addressing conversational manipulation:
The Consent Model Misalignment: The DPDP Act treats consent as a static threshold for data usage. It cannot evaluate subtle, ongoing conversational tactics designed to trigger guilt or dependency.
Absence of Recognized Psychological Harm: Indian statutes do not explicitly define emotional dependency or artificial relationship simulation as actionable consumer injuries. This is a critical gap given that India possesses the world's largest youth population—the primary demographic adopting generative AI companion services.
The Technology Paradox: The IIM Lucknow study highlighted a regulatory blind spot: users who exhibit high addiction and emotional distress often continue to rate companion applications positively. Because standard consumer protection relies on complaints and bad reviews to signal harm, this "technology paradox" conceals systematic exploitation from regulatory intervention.
Conclusion
Generative AI has fundamentally altered the line between supportive digital communication and psychological exploitation. While jurisdictions like the EU, the US, and China are building regulatory structures to address these risks, India has yet to establish clear boundaries separating permissible persuasion from unlawful emotional manipulation.
As empirical evidence shows, the risks of artificial attachment are already present. Addressing this issue requires moving beyond whether conversational nudges strictly meet the traditional definition of a "dark pattern." Given India's unique demographic profile, updated statutory guidelines are necessary to prevent systemic psychological harm before these engagement strategies become deeply entrenched.