Usability studies often overlook valuable insights from moderator-to-participant interventions. This study proposes that these interventions, moderator-provided “nudges”, are a rich source of insights on user needs, cognitive load, and system design gaps. We analyzed transcripts from 86 sessions involving two digital library systems and 56 blue-collar (BC) and 30 white-collar (WC) participants, and identified 962 instances of moderator interventions (i.e., nudges). Findings show significant disparities in both the volume and composition of nudges across groups. BC participants required 21 times as many nudges (N=919) as WC participants (N=43). Of the BC participants’ nudges, 737 (80.2%) were system-related, and 182 (19.8%) were user-related. Treating nudges as usability insights for HCI research enables researchers to identify where system scaffolding, such as explicit language, progressive guidance, or simplified workflows, is necessary. The study contributes to inclusive design theory by demonstrating how occupational background influences interventions in user studies and provides design implications for making digital information services more inclusive across occupational groups.

The proliferation of artificial intelligence (AI) technologies has led to the rise of conversational decision-making support systems, such as dialogue persona systems that provide conversational access to various user segments. For example, product managers can ask personas about features before implementing them, politicians can learn about the needs of local communities through personas, and so on. Nascent research has looked at challenges when users interact with AI personas, but has not framed it as a principal–agent problem, in which the AI represents a persona that itself represents real people in the data. This setting exposes unique interaction challenges that decision makers face when engaging with AI-generated conversational personas, which we examine through a user study with 56 participants using AI-generated conversational personas. Our results indicate seven interaction challenges: (1) Hidden Information, (2) Hidden Personas, (3) Hidden UI, (4) Lack of AI Agency, (5) AI’s Selective Attention, (6) Confusing Distributional Information, and (7) Conversational Cold Start that we conceptually link with agency theory. We discuss how the interaction challenges could be alleviated and suggest directions for future work. • This article explores how decision makers interact with AI-generated conversational personas derived from real survey data. • It conducts a comparative study between conversational personas and traditional profile personas in decision-support contexts. • The study employs a think-aloud user study with 56 participants to capture interaction experiences and challenges. • It identifies seven specific interaction challenges unique to conversational personas that may hinder effective decision making. • The article provides insights and recommendations for designing conversational decision support systems using AI-generated personas.

We examined the accessibility of online information services (OISs) for underserved communities through a user study involving 56 blue collar participants interacting with a website and an app for five tasks. The blue collar participants were generally unsuccessful on both platforms, with 12.7% (n=7) unable to successfully complete any tasks; a hundred percent required at least minor assistance. Participants were also inefficient, taking 28.62 more steps than optimal (143.1%) on the website and 10.41 more steps (47.3%) on the app. Time inefficiency was also noteworthy, with 535.76 more seconds than optimal (248.0%) on the website and 266.55 more seconds (142.4%) on the app. Though still poor, the app yielded better outcomes with higher success rates and usability ratings. Digital proficiency correlated with success on both platforms, which is good news as this is addressable by OIS providers. Qualitative analysis revealed that many in this underserved population were unaware that these valuable OISs were available to them. Findings underscore the need for OIS providers to prioritize targeted outreach to inform underserved communities that OISs are open and welcoming. Designing OISs with accessibility and simplicity targeted for mobile devices is crucial for bridging the digital literacy gap and empowering underserved communities to engage effectively with OISs.

The development of persona systems provides a possibility for end users to interact with different persona modalities. In a 54-participant randomized controlled experiment, we compare two persona interaction modalities, document and dialogue personas, both generated using AI approaches from survey data. Overall, dialogue personas appear to be perceived more favorably than document personas. However, document personas exhibit a wider range of perceptions, suggesting that experiences with document personas are more polarizing among users. The document personas had higher transparency and were perceived as more complete, but the task completion was perceived as more difficult, although the task success rate was higher. The dialogue personas were perceived as more usable, with a higher System Usability Scale score, and more enjoyable. Our findings provide critical insights into the increasingly important area of persona interaction modalities and the broad paradigm of human-persona interaction.

We investigated the presence and acceptance of hallucinations (i.e., accidental misinformation) of an AI-generated persona system that leverages large language models for persona creation from survey data in a 54-user within-subjects experiment. After interacting with the personas, users were given a task to ask the personas a series of questions, including an unanswerable question, meaning the personas lacked the data to answer the question. The AI-generated persona system provided a plausible but incorrect answer half (52%) of the time, and more than half of the time (57%), the users accepted the incorrect answer, and the rest of the time, users answered the unanswerable question correctly (no answer). We found that when the AI-generated persona hallucinated, the user was significantly more likely to answer the unanswerable question incorrectly. Also, for genders separately, when the AI-generated persona hallucinated, it was significantly more likely for the female user and the male users to answer the unanswerable question incorrectly. We identified four themes in the AI-generated persona's answers and found that users perceive AI-generated persona's answers as long and unclear for the unanswerable question. Findings imply that personas leveraging LLMs require guardrails to ensure that personas clearly state the possibility of data restrictions and hallucinations when asked unanswerable questions.

Large Language Models (LLMs) are emerging as a powerful tool for AI-generated personas. This study evaluates the usability of AI-generated personas, comparing chat and profile formats. The findings indicate chat personas tend to be perceived more favourably, and profile personas exhibit greater variability in user perception. The increased difficulty and longer dwell time experienced by users with the profile persona, despite negative usability metrics, paradoxically resulted in better task performance. Usability issues indicate that many current limitations of AI, including verbosity, hallucinations, and empty rhetoric which was described as the persona having ‘no soul’, are inherited in AI-generated chat personas. However, there are also new issues. For one, the risk of information overload in an AI-generated profile persona implies that the AI does not consider human users’ cognitive limitations when designing the persona (but usability scores for profile personas increase with dwell time, implying that users get used to the longer format the more time they spend). Another is the ‘fourth wall’ effect of AI-generated chat personas in which the user feels they are talking to someone describing the persona rather than the persona itself. Future work could address the usability paradox and the fourth wall effect of using personas.CCS CONCEPTS Human-centered computing Human computer interaction (HCI)

Demographics are considered foundational information in most persona profiles. However, the effect of persona ethnicity and gender on designers’ identification with the persona has limited evaluation in the human-computer interaction literature. We conducted a study with 64 professional designers from the United States, Indian, Korean, and Mexican nationalities to investigate the effects of AI-generated persona ethnicity and gender on persona identification. The personas were created using Generative AI in the persona narratives and the persona video creation. The contribution of this work is that, against assumptions, neither persona ethnicity nor gender play a major role in persona identification among designers with different ethnic backgrounds. While there were some insinuations of ethnicity and gender in the open-ended feedback from the designers, the emergent qualitative themes describing persona identification were overwhelmingly universal and applicable regardless of ethnicity or gender. This implies that professional designers can effectively use personas with different demographic backgrounds, and effects of demographic attributes in personas leading to stereotyping are less impactful than presumed.

Cipherbot, an educational chatbot using large language models to answer student questions concerning learning materials uploaded by the educator, was pilot tested in a classroom setting. Forty-four students used Cipherbot for seven weeks, sending 8077 messages. The average number of messages sent per student was 184 (SD = 80), with an average length of 98 characters (SD = 80). The engagement followed a non-normal distribution, with few power users, implying that most students are still hesitant to adopt tools like Cipherbot. Cipherbot was able to answer 82.5% of the student questions, demonstrating a scalable ability to address students' learning queries, with some room for improvement.

In this article, we present METRIC. Measuring Engagement Through Remote Interactions of Customers (METRIC) (https://metric.qcri.org/) is a tool for collecting, measuring, analyzing, and reporting the engagement of online systems through actual interactions of customers or users, either remote or in the lab. METRIC enables system stakeholders to enhance their understanding of their audience, customer, or users' actual behavior on pages, images, videos, interfaces, and online systems, including the gaze and interaction with sub-elements on a page within a system or comparisons via A/B testing. Along with eye-tracking devices, METRIC uses a webcam-based eye-tracking JavaScript library for the ability to monitor the users' real visual attention during interaction with the online system. METRIC provides sophisticated reporting features throughout the collecting, measuring, and analyzing process. METRIC can also be deployed in user experiments toward the design of better cooperation technologies, primarily due to its online nature.