As generative artificial intelligence (genAI) applications gain popularity, there is a dearth of research examining how applications may transform social interactions. One possible application set to transform social interactions is the use of generative speech to power AI agents that can realistically represent people. Our work examines the potential implications of AI agents representing individuals in human conversations ("agent representation") as a way to begin filling this research gap. We take a multi-method approach, conducting formative interviews with developers, a co-design workshop with designers, a harm analysis among researchers, and interviews with the general public. Both technologists and potential users worry adopting agent representations might harm the quality, trust, and autonomy of human communication. Potential users are particularly concerned that agent representations could undermine the value of social interaction and threaten individuals’ ability to control their image. To avoid such potential consequences, future genAI-powered agents and speech applications should take into account user-defined red lines when considering applying these technologies in social settings.

Most prior work on human-AI interaction is set in communities that indicate skepticism towards AI, but we know less about contexts where AI is viewed as aspirational. We investigated the perceptions around AI systems by drawing upon 32 interviews and 459 survey respondents in India. Not only do Indian users accept AI decisions (79.2% respondents indicate acceptance), we find a case of AI authority—AI has a legitimized power to influence human actions, without requiring adequate evidence about the capabilities of the system. AI authority manifested into four user attitudes of vulnerability: faith, forgiveness, self-blame, and gratitude, pointing to higher tolerance for system misfires, and introducing potential for irreversible individual and societal harm. We urgently call for calibrating AI authority, reconsidering success metrics and responsible AI approaches and present methodological suggestions for research and deployments in India.