Figure 1: Our system supports nonlinear AI-assisted hypothesis exploration that balances breadth and depth of exploration.Using the node-link diagram shared representation and integrated information hint panels that display preliminary results and related work, a participant in our user study deeply explored a branch of hypotheses around gender income gaps, including a nuanced hypothesis about field-specific gender disparities in income (A); but also simultaneously kept track of the overall hypothesis space, and backtracked to explore other hypothesis branches around variations in income by marital status (B), such as post-divorce impacts on income.A more detailed version Figure 10 can be found in Appendix B.

Large Language Models (LLMs) have introduced a paradigm shift in interaction with AI technology, enabling knowledge workers to complete tasks by specifying their desired outcome in natural language. LLMs have the potential to increase productivity and reduce tedious tasks in an unprecedented way. A systematic study of LLM adoption for work can provide insight into how LLMs can best support these workers. To explore knowledge workers' current and desired usage of LLMs, we ran a survey (n=216). Workers described tasks they already used LLMs for, like generating code or improving text, but imagined a future with LLMs integrated into their workflows and data. We ran a second survey (n=107) a year later that validated our initial findings and provides insight into up-to-date LLM use by knowledge workers. We discuss implications for adoption and design of generative AI technologies for knowledge work.

Selecting an effective utterance among countless possibilities that match a user's intention poses a challenge when using natural language interfaces. To address the challenge, we leveraged the principle of least collaborative effort in communication grounding theory and designed three grounded conversational interactions: 1) a grounding interface allows users to start with a provisional input and then invite a conversational agent to complete their input, 2) a multiple grounding interface presents multiple inputs for the user to select from, and 3) a structured grounding interface guides users to write inputs in a structure best understood by the system. We compared our three grounding interfaces to an ungrounded control interface in a crowdsourced study (N=80) using a natural language system that generates small programs. We found that the grounding interfaces reduced cognitive load and improved task performance. The structured grounding interface further reduced speaker change costs and improved technology acceptance, without sacrificing the perception of control. We discuss the implications of designing grounded conversational interactions in natural language systems.

Forming accurate mental models that align with the actual behavior of an AI system is critical for successful user experience and interactions. One way to develop mental models is through information shared by other users. However, this social information can be inaccurate and there is a lack of research examining whether inaccurate social information influences the development of accurate mental models. To address this gap, our study investigates the impact of social information accuracy on mental models, as well as whether prompting users to validate the social information can mitigate the impact. We conducted a between-subject experiment with 39 crowdworkers where each participant interacted with our AI system that automates a workflow given a natural language sentence. We compared participants' mental models between those exposed to social information of how the AI system worked, both correct and incorrect, versus those who formed mental models through their own usage of the system. Specifically, we designed three experimental conditions: 1) validation condition that presented the social information followed by an opportunity to validate its accuracy through testing example utterances, 2) social information condition that presented the social information only, without the validation opportunity, and 3) control condition that allowed users to interact with the system without any social information. Our results revealed that the inclusion of the validation process had a positive impact on the development of accurate mental models, especially around the knowledge distribution aspect of mental models. Furthermore, participants were more willing to share comments with others when they had the chance to validate the social information. The impact of inaccurate social information on altering user mental models was found to be non-significant, while 69.23% of participants incorrectly judged the social information accuracy at least once. We discuss the implications of these findings for designing tools that support the validation of social information and thereby improve human-AI interactions.

Subject matter experts play an important role in customer support communities by responding to user queries. Some communities have adopted chatbots in addition to SMEs to address commonly asked questions. Yet, SME-bot interactions, particularly teaching paradigms between SMEs and bots remain understudied. We investigate human-AI machine teaching interactions in a scenario-based study (n=48). Participants selected their preferred teaching method in simulated community interactions with a consumer, an SME, and an AI Bot. We investigated preferences across three interactions: demonstration (Showing), preference elicitation (Sorting), and labeling (Categorization). Participants preferred the Showing interaction, followed by Sorting and Categorizing. Participants changed their preferences from lower-effort interactions when considering downstream outcomes. Users considered the community’s perception of interactions between the bot and the SME, specifically transparency of learning outcome, orientation of the feedback, querying the bot and disruptiveness of the interaction. We discuss implications for our findings for teaching interactions in human-AI communities.

While natural language systems continue improving, they are still imperfect. If a user has a better understanding of how a system works, they may be able to better accomplish their goals even in imperfect systems. We explored whether explanations can support effective authoring of natural language utterances and how those explanations impact users’ mental models in the context of a natural language system that generates small programs. Through an online study (n=252), we compared two main types of explanations: 1) system-focused, which provide information about how the system processes utterances and matches terms to a knowledge base, and 2) social, which provide information about how other users have successfully interacted with the system. Our results indicate that providing social suggestions of terms to add to an utterance helped users to repair and generate correct flows more than system-focused explanations or social recommendations of words to modify. We also found that participants commonly understood some mechanisms of the natural language system, such as the matching of terms to a knowledge base, but they often lacked other critical knowledge, such as how the system handled structuring and ordering. Based on these findings, we make design recommendations for supporting interactions with and understanding of natural language systems.

Web applications and services are increasingly important in a distributed internet filled with diverse cloud services and applications, each of which enable the completion of narrowly defined tasks. Given the explosion in the scale and diversity of such services, their composition and integration for achieving complex user goals remains a challenging task for end-users and requires a lot of development effort when specified by hand. We present a demonstration of the Goal Oriented Flow Assistant (GOFA) system, which provides a natural language solution to generate workflows for application integration. Our tool is built on a three-step pipeline: it first uses Abstract Meaning Representation (AMR) to parse utterances; it then uses a knowledge graph to validate candidates; and finally uses an AI planner to compose the candidate flow. We provide a video demonstration of the deployed system as part of our submission.

We demonstrate an AI assisted data labeling system which applies unsupervised and semi-supervised machine learning to facilitate accurate and efficient labeling of large data sets. Our system (1) applies representative data sampling and active learning in order to seed and maintain a semi-supervised learner that assists the human labeler (2) provides visual labeling assistance and optimizes labeling mechanics using predicted labels (3) seamlessly updates and learns from ongoing human labeling activity (4) captures and presents metrics that indicate the quality of labeling assistance, and (5) provides an interactive auto labeling interface to group, review and apply predicted labels in a scalable manner.

Human data labeling with multiple labelers and the resulting conflict resolution remains the norm for many enterprise machine learning pipelines. Conflict resolution can be a time-intensive and costly process. Our goal was to study how human-AI collaboration can improve conflict resolution, by enabling users to automate groups of conflict resolution tasks. However, little is known about whether and how people will rely on automation during conflict resolution. Currently, automation commonly uses labelers' majority vote labels for conflict resolution, as the top chosen label by most labelers is often correct. We envisioned a system where an AI would assist in finding cases where the labeler majority vote was wrong and where automation is supported for batches or groups of conflicts. In order to understand whether humans could use labeler and AI information effectively, we investigated how and when users rely on labeler and AI information and on automated group conflict resolution. We ran a study with 144 Mechanical Turk workers. We found that automation increased users' accuracy/time, use of automated conflict resolution was relatively similar regardless of whether the automation was based on labeler or AI selected labels, and providing labeler and AI selected labels may reduce inappropriate reliance on automation.

Labeling data is an important step in the supervised machine learning lifecycle. It is a laborious human activity comprised of repeated decision making: the human labeler decides which of several potential labels to apply to each example. Prior work has shown that providing AI assistance can improve the accuracy of binary decision tasks. However, the role of AI assistance in more complex data-labeling scenarios with a larger set of labels has not yet been explored. We designed an AI labeling assistant that uses a semi-supervised learning algorithm to predict the most probable labels for each example. We leverage these predictions to provide assistance in two ways: (i) providing a label recommendation and (ii) reducing the labeler’s decision space by focusing their attention on only the most probable labels. We conducted a user study (n=54) to evaluate an AI-assisted interface for data labeling in this context. Our results highlight that the AI assistance improves both labeler accuracy and speed, especially when the labeler finds the correct label in the reduced label space. We discuss findings related to the presentation of AI assistance and design implications for intelligent labeling interfaces.

Human labeling of training data is often a time-consuming, expensive part of machine learning. In this paper, we study "batch labeling", an AI-assisted UX paradigm, that aids data labelers by allowing a single labeling action to apply to multiple records. We ran a large scale study on Mechanical Turk with 156 participants to investigate labeler-AI-batching system interaction. We investigate the efficacy of the system when compared to a single-item labeling interface (i.e., labeling one record at-a-time), and evaluate the impact of batch labeling on accuracy and time. We further investigate the impact of AI algorithm quality and its effects on the labelers' overreliance, as well as potential mechanisms for mitigating it. Our work offers implications for the design of batch labeling systems and for work practices focusing on labeler-AI-batching system interaction.