ChatGPT and other large language models (LLMs) have proven useful in crowdsourcing tasks, where they can effectively annotate machine learning training data. However, this means that they also have the potential for misuse, specifically to automatically answer surveys. LLMs can potentially circumvent quality assurance measures, thereby threatening the integrity of methodologies that rely on crowdsourcing surveys. In this paper, we propose a mechanism to detect LLM-generated responses to surveys. The mechanism uses ''prompt injection,'' such as directions that can mislead LLMs into giving predictable responses. We evaluate our technique against a range of question scenarios, types, and positions, and find that it can reliably detect LLM-generated responses with more than 98% effectiveness. We also provide an open-source software to help survey designers use our technique to detect LLM responses. Our work is a step in ensuring that survey methodologies remain rigorous vis-a-vis LLMs.

Learning to operate a complex system, such as an agile production line, can be a daunting task. The high variability in products and frequent reconfigurations make it difficult to keep documentation up-to-date and share new knowledge amongst factory workers. We introduce CLAICA, a Continuously Learning AI Cognitive Assistant that supports workers in the aforementioned scenario. CLAICA learns from (experienced) workers, formalizes new knowledge, stores it in a knowledge base, along with contextual information, and shares it when relevant. We conducted a user study with 83 participants who performed eight knowledge exchange tasks with CLAICA, completed a survey, and provided qualitative feedback. Our results provide a deeper understanding of how prior training, context expertise, and interaction modality affect the user experience of cognitive assistants. We draw on our results to elicit design and evaluation guidelines for cognitive assistants that support knowledge exchange in fast-paced and demanding environments, such as an agile production line.

Emoji have become an essential part of modern communication, helping to convey emotions and tone quickly and concisely. Emoji used by humans and Intelligent Agents (IA) have been shown to affect people's decision making intentions, suggesting they could be used to manipulate users to follow their advice. We present a mixed-methods crowdsourcing study (N = 194) that shows that adherence to an IA's recommendation and user experience are not affected by emoji when used in a positive, collaborative way. However, we demonstrate that explanations provided by an IA do increase adherence to its recommendation.

Conversational agents (CA) occasionally fail to understand the user’s intention or respond inappropriately due to natural language complexity. These conversational breakdowns can happen because of low intent and entity prediction confidence scores. A promising repair strategy in such cases is that the CA proposes to users likely alternatives to proceed. If one of these options matches the user’s intention, the breakdown is repaired successfully. We propose that successful repairs should be followed by a learning mechanism to minimize future breakdowns. After a successful repair, the CA, user, or both can learn each other’s specific phrasing. This prevents similar phrasings from causing reoccurring breakdowns. We compared user preferences for these learning mechanisms in a scenario-based study with manufacturing workers (). Our result showed that users first prefer to share the learning burden with the CA (61.3%), followed by entirely outsourcing the learning burden to the CA (60.7%) as opposed to themselves.