Chatbots possess great potential benefits, yet concerns persist regarding users adopting inappropriate, offensive language. This research delved into the influence of user characteristics on verbally aggressive behaviours towards social chatbots. Employing a mixed-method study, we examined individual characteristics such as personal dispositions, offensive language patterns, academic majors, and prior experiences with conversational agents. Findings from a ten-day field experiment involving 33 participants using a real-world Telegram-based chatbot app unveiled that users' anthropomorphism, computer-related major, and gender significantly impact their moral emotions and evaluations of the chatbot's capabilities. Moreover, employing offensive language towards the chatbot detrimentally impacted users' perceptions of its abilities, helpfulness, and likability. The research findings advocate for ongoing monitoring and effective resolution of users' behaviours regarding the use of offensive language in their interactions with a chatbot. Additionally, the results underscore the importance of incorporating diverse perspectives into chatbot design to address biases and offensive utterances.

While detecting offensive language in online spaces remains an important societal issue, there is still a significant gap in existing research and practial datasets specific to chatbots. Furthermore, many of the current efforts by service providers to automatically filter offensive language are vulnerable to users’ deliberate text manipulation tactics, such as misspelling words. In this study, we analyze offensive language patterns in real logs of 6,254,261 chat utterance pairs from the commercial chat service Simsimi, which cover a variety of conversation topics. Based on the observed patterns, we introduce a novel offensive language detection method—a contrastive learning model that embeds chat content with a random masking strategy. We show that this model outperforms existing models in detecting offensive language in open-domain chat conversations while also demonstrating robustness against users’ deliberate text manipulation tactics when using offensive language. We release our curated chatbot dataset to foster research on offensive language detection in open-domain conversations and share lessons learned from mitigating offensive language on a live platform.

Natural language processing is enabling machines to communicate with humans naturally, yet the dynamics of extended user-chatbot interactions remain much unexplored. This study characterizes the conversational styles, demographics, psychologies, and emotional tendencies of the most active users (i.e., top 1% by message count) of a commercial chatbot platform (SimSimi.com), whom we refer to as superusers. We analyzed the linguistic patterns and topics of 1,988,971 messages written by 1,994 superusers over a period of three years. We further surveyed 76 users to observe their emotional dispositions and perceptions towards the chatbot. We find that SimSimi superusers empathize and humanize the chatbot more than less active users, and they show a higher tendency to share personal and negative feelings. Our findings suggest that chatbots require new design considerations for users who are vulnerable due to their high anthropomorphism and openness toward machines. Our work also shows that chatbots should have functions to offer social support when necessary.

Conversational agents (CAs) offer new functionality and convenience. While their sales have been soaring, they have also rapidly become victims of verbal abuse by their users. Without proper handling of abusive usage, abusers’ actions can be reinforced and transferred to real life. This study investigates whether alternative response styles of empathy orientation and emotional expressivity of voice-activated virtual assistants influence users’ moral emotions found to reduce verbal aggression as well as whether they affect user perceptions of the agent’s capability. Ninety-eight participants were assigned to one of the three emotional expressivity conditions (no-facial expression, fixed-facial expression, varied-facial expression) and interacted with two alternative empathy orientation conditions (other-oriented, self-oriented) of agents. The experimental results show that, regardless of the emotional expressivity types, the agent’s empathy orientation has a significant effect on the moral emotions and agent capability perceptions. Overall, an agent that employed other-oriented empathy style elicited most positive responses from the users. However, the preference was not across the board, as about one-third of the participants showed preference to the self-oriented CA. Users valued agents’ verbal contents and vocal characteristics above their facial expressions. Based on the study findings, we draw several design guidelines and suggest avenues for future research.

Fact-checking has become the de facto solution for fighting fake news online. This research brings attention to the unexpected and diminished effect of fact-checking due to cognitive biases. We experimented (66,870 decisions) comparing the change in users’ stance toward unproven claims before and after being presented with a hypothetical fact-checked condition. We found that, first, the claims tagged with the ‘Lack of Evidence’ label are recognized similarly as false information unlike other borderline labels, indicating the presence of uncertainty-aversion bias in response to insufficient information. Second, users who initially show disapproval toward a claim are less likely to correct their views later than those who initially approve of the same claim when opposite fact-checking labels are shown — an indication of disapproval bias. Finally, user interviews revealed that users are more likely to share claims with Divided Evidence than those with Lack of Evidence among borderline messages, reaffirming the presence of uncertainty-aversion bias. On average, we confirm that fact-checking helps users correct their views and reduces the circulation of falsehoods by leading them to abandon extreme views. Simultaneously, the presence of two biases reveals that fact-checking does not always elicit the desired user experience and that the outcome varies by the design of fact-checking messages and people’s initial view. These new observations have direct implications for multiple stakeholders, including platforms, policy-makers, and online users.

With the popularity of AI-infused systems, conversational agents (CAs) are becoming essential in diverse areas, offering new functionality and convenience, but simultaneously, suffering misuse and verbal abuse. We examine whether conversational agents' response styles under varying abuse types influence those emotions found to mitigate peoples' aggressive behaviors, involving three verbal abuse types (Insult, Threat, Swearing) and three response styles (Avoidance, Empathy, Counterattacking). Ninety-eight participants were assigned to one of the abuse type conditions, interacted with the three spoken (voice-based) CAs in turn, and reported their feelings about guiltiness, anger, and shame after each session. The results show that the agent's response style has a significant effect on user emotions. Participants were less angry and more guilty with the empathy agent than the other two agents. Furthermore, we investigated the current status of commercial CAs' responses to verbal abuse. Our study findings have direct implications for the design of conversational agents.