We investigated the presence and acceptance of hallucinations (i.e., accidental misinformation) of an AI-generated persona system that leverages large language models for persona creation from survey data in a 54-user within-subjects experiment. After interacting with the personas, users were given a task to ask the personas a series of questions, including an unanswerable question, meaning the personas lacked the data to answer the question. The AI-generated persona system provided a plausible but incorrect answer half (52%) of the time, and more than half of the time (57%), the users accepted the incorrect answer, and the rest of the time, users answered the unanswerable question correctly (no answer). We found that when the AI-generated persona hallucinated, the user was significantly more likely to answer the unanswerable question incorrectly. Also, for genders separately, when the AI-generated persona hallucinated, it was significantly more likely for the female user and the male users to answer the unanswerable question incorrectly. We identified four themes in the AI-generated persona's answers and found that users perceive AI-generated persona's answers as long and unclear for the unanswerable question. Findings imply that personas leveraging LLMs require guardrails to ensure that personas clearly state the possibility of data restrictions and hallucinations when asked unanswerable questions.

Large Language Models (LLMs) are emerging as a powerful tool for AI-generated personas. This study evaluates the usability of AI-generated personas, comparing chat and profile formats. The findings indicate chat personas tend to be perceived more favourably, and profile personas exhibit greater variability in user perception. The increased difficulty and longer dwell time experienced by users with the profile persona, despite negative usability metrics, paradoxically resulted in better task performance. Usability issues indicate that many current limitations of AI, including verbosity, hallucinations, and empty rhetoric which was described as the persona having ‘no soul’, are inherited in AI-generated chat personas. However, there are also new issues. For one, the risk of information overload in an AI-generated profile persona implies that the AI does not consider human users’ cognitive limitations when designing the persona (but usability scores for profile personas increase with dwell time, implying that users get used to the longer format the more time they spend). Another is the ‘fourth wall’ effect of AI-generated chat personas in which the user feels they are talking to someone describing the persona rather than the persona itself. Future work could address the usability paradox and the fourth wall effect of using personas.CCS CONCEPTS Human-centered computing Human computer interaction (HCI)

Demographics are considered foundational information in most persona profiles. However, the effect of persona ethnicity and gender on designers’ identification with the persona has limited evaluation in the human-computer interaction literature. We conducted a study with 64 professional designers from the United States, Indian, Korean, and Mexican nationalities to investigate the effects of AI-generated persona ethnicity and gender on persona identification. The personas were created using Generative AI in the persona narratives and the persona video creation. The contribution of this work is that, against assumptions, neither persona ethnicity nor gender play a major role in persona identification among designers with different ethnic backgrounds. While there were some insinuations of ethnicity and gender in the open-ended feedback from the designers, the emergent qualitative themes describing persona identification were overwhelmingly universal and applicable regardless of ethnicity or gender. This implies that professional designers can effectively use personas with different demographic backgrounds, and effects of demographic attributes in personas leading to stereotyping are less impactful than presumed.

Large language models (LLMs) can generate personas based on prompts that describe the target user group. To understand what kind of personas LLMs generate, we investigate the diversity and bias in 450 LLM-generated personas with the help of internal evaluators (n=4) and subject-matter experts (SMEs) (n=5). The research findings reveal biases in LLM-generated personas, particularly in age, occupation, and pain points, as well as a strong bias towards personas from the United States. Human evaluations demonstrate that LLM persona descriptions were informative, believable, positive, relatable, and not stereotyped. The SMEs rated the personas slightly more stereotypical, less positive, and less relatable than the internal evaluators. The findings suggest that LLMs can generate consistent personas perceived as believable, relatable, and informative while containing relatively low amounts of stereotyping.