Despite recognizing that Large Language Models (LLMs) can generate inaccurate or unacceptable responses, universities are increasingly making such models available to their students.Existing university policies defer the responsibility of checking for correctness and appropriateness of LLM responses to students and assume that they will have the required knowledge and skills to do so on their own.In this work, we conducted a series of user studies with students (N=47) from a large North American public research university to understand if and how they critically engage with LLMs.Our participants evaluated an LLM provided by the university in a quasi-experimental setup; first by themselves, and then with a scaffolded design probe that guided them through an end-user auditing exercise.Qualitative analysis of participant think-aloud and LLM interaction data showed that students without basic AI literacy skills struggle to conceptualize and evaluate LLM biases on their own.However, they transition to focused thinking and purposeful interactions when provided with structured guidance.We highlight areas where current university policies may fall short and offer policy and design recommendations to better support students.

YouTube has become an important part of the educational ecosystem, with millions of viewers seeking informative videos and help with coursework. Educational YouTubers create this content, often balancing pedagogical rigor and entertainment value. However, creators need not only to promote their content to find viewers, but also to monetize. In this study, we explore the tensions educational YouTubers face when making monetized educational content. We conduct a qualitative interview study with 12 popular educational YouTubers about their monetization strategies, perceptions of YouTube's algorithmic promotion of their content, and conception of their audience. We find that educational YouTubers are largely driven by a desire to share free and high-quality educational content, and that common monetization strategies like sponsorships and clickbait sometimes interfere with this mission. We describe the careful strategies our participants use to maintain educational integrity while making a living on an algorithmically-driven platform. We then use these findings to draw parallels between YouTubers' challenges with monetizing educational content and the history of educational public broadcast in the United States, which has followed a similar trajectory. In closing, we offer several recommendations for supporting educational YouTubers in creating the high-quality, publicly accessible educational content that is appreciated by a worldwide audience.

Improving end-users' understanding of decisions made by autonomous vehicles (AVs) driven by artificial intelligence (AI) can improve utilization and acceptance of AVs. However, current explanation mechanisms primarily help AI researchers and engineers in debugging and monitoring their AI systems, and may not address the specific questions of end-users, such as passengers, about AVs in various scenarios. In this paper, we conducted two user studies to investigate questions that potential AV passengers might pose while riding in an AV and evaluate how well answers to those questions improve their understanding of AI-driven AV decisions. Our initial formative study identified a range of questions about AI in autonomous driving that existing explanation mechanisms do not readily address. Our second study demonstrated that interactive text-based explanations effectively improved participants' comprehension of AV decisions compared to simply observing AV decisions. These findings inform the design of interactions that motivate end-users to engage with and inquire about the reasoning behind AI-driven AV decisions.

Personalization improves user experience by tailoring interactions relevant to each user’s background and preferences. However, personalization requires information about users that platforms often collect without their awareness or their enthusiastic consent. Here, we study how the transparency of AI inferences on users’ personal data affects their privacy decisions and sentiments when sharing data for personalization. We conducted two experiments where participants (N=877) answered questions about themselves for personalized public arts recommendations. Participants indicated their consent to let the system use their inferred data and explicitly provided data after awareness of inferences. Our results show that participants chose restrictive consent decisions for sensitive and incorrect inferences about them and for their answers that led to such inferences. Our findings expand existing privacy discourse to inferences and inform future directions for shaping existing consent mechanisms in light of increasingly pervasive AI inferences.

Ensuring that Machine Learning (ML) models make correct and meaningful inferences is necessary for the broader adoption of such models into high-stakes decision-making scenarios. Thus, ML model engineers increasingly use eXplainable AI (XAI) tools to investigate the capabilities and limitations of their ML models before deployment. However, explaining sequential ML models, which make a series of decisions at each timestep, remains challenging. We present Visual Interactive Model Explorer (VIME), an XAI toolbox that enables ML model engineers to explain decisions of sequential models in different “what-if” scenarios. Our evaluation with 14 ML experts, who investigated two existing sequential ML models using VIME and a baseline XAI toolbox to explore “what-if” scenarios, showed that VIME made it easier to identify and explain instances when the models made wrong decisions compared to the baseline. Our work informs the design of future interactive XAI mechanisms for evaluating sequential ML-based decision support systems.

Proponents of legal automation believe that translating the law into code can improve the legal system. However, research and reporting suggest that legal software systems often contain flawed translations of the law, resulting in serious harms such as terminating children's healthcare and charging innocent people with fraud. Efforts to identify and contest these mistranslations after they arise treat the symptoms of the problem, but fail to prevent them from emerging. Meanwhile, existing recommendations to improve the development of legal software remain untested, as there is little empirical evidence about the translation process itself. In this paper, we investigate the behavior of fifteen teams---nine composed of only computer scientists and six of computer scientists and legal experts---as they attempt to translate a bankruptcy statute into software. Through an interpretative qualitative analysis, we characterize a significant epistemic divide between computer science and law and demonstrate that this divide contributes to errors, misunderstandings, and policy distortions in the development of legal software. Even when development teams included legal experts, communication breakdowns meant that the resulting tools predominantly presented incorrect legal advice and adopted inappropriately harsh legal standards. Study participants did not recognize the errors in the tools they created. We encourage policymakers and researchers to approach legal software with greater skepticism, as the disciplinary divide between computer science and law creates an endemic source of error and mistranslation in the production of legal software.

On-demand work platforms are attractive alternatives to traditional employment arrangements. However, several questions around employment classification, compensation, data privacy, and equitable outcomes remain open. The abilities of algorithmic management to structure different forms of platform-worker relationships compounds fraught regulatory debates. Understanding the conditions of algorithmic management that result in these variations could point us towards better worker futures. In this work, we studied the platform-worker relationships in Instacart work through the accounts of its workers. From a qualitative analysis of 400 Reddit posts by Instacart’s workers, we identified sources and types of ambiguity that gave rise to open-ended experiences for workers. Ambiguities supplemented gamification mechanisms to regulate worker behaviors. Yet, they also generated affective experiences for workers that enabled their playful participation in the Reddit community. We propose the frame of ludification to explain these seemingly contradicting findings and conclude with implications for accountability in on-demand work platforms.

Tech companies that rely on ads for business argue that users have control over their data via ad privacy settings. However, these ad settings are often hidden. This work aims to inform the design of findable ad controls and study their impact on users’ behavior and sentiment. We iteratively designed ad control interfaces that varied in the setting’s (1) entry point (within ads, at the feed’s top) and (2) level of actionability, with high actionability directly surfacing links to specific advertisement settings, and low actionability pointing to general settings pages (which is reminiscent of companies’ current approach to ad controls). We built a Chrome extension that augments Facebook with our experimental ad control interfaces and conducted a between-subjects online experiment with 110 participants. Results showed that entry points within ads or at the feed’s top, and high actionability interfaces, both increased Facebook ad settings’ findability and discoverability, as well as participants’ perceived usability of them. High actionability also reduced users’ effort in finding ad settings. Participants perceived high and low actionability as equally usable, which shows it is possible to design more actionable ad controls without overwhelming users. We conclude by emphasizing the importance of regulation to provide specific and research-informed requirements to companies on how to design usable ad controls.

Decision Support Systems (DSS) based on Machine Learning (ML) often aim to assist lay decision-makers, who are not math-savvy, in making high-stakes decisions. However, existing ML-based DSS are not always transparent about the probabilistic nature of ML predictions and how uncertain each prediction is. This lack of transparency could give lay decision-makers a false sense of reliability. Growing calls for AI transparency have led to increasing efforts to quantify and communicate model uncertainty. However, there are still gaps in knowledge regarding how and why the decision-makers utilize ML uncertainty information in their decision process. Here, we conducted a qualitative, think-aloud user study with 17 lay decision-makers who interacted with three different DSS: 1) interactive visualization, 2) DSS based on an ML model that provides predictions without uncertainty information, and 3) the same DSS with uncertainty information. Our qualitative analysis found that communicating uncertainty about ML predictions forced participants to slow down and think analytically about their decisions. This in turn made participants more vigilant, resulting in reduction in over-reliance on ML-based DSS. Our work contributes empirical knowledge on how lay decision-makers perceive, interpret, and make use of uncertainty information when interacting with DSS. Such foundational knowledge informs the design of future ML-based DSS that embrace transparent uncertainty communication.

Trustworthy Artificial Intelligence (AI) is characterized, among other things, by: 1) competence, 2) transparency, and 3) fairness. However, end-users may fail to recognize incompetent AI, allowing untrustworthy AI to exaggerate its competence under the guise of transparency to gain unfair advantage over other trustworthy AI. Here, we conducted an experiment with 120 participants to test if untrustworthy AI can deceive end-users to gain their trust. Participants interacted with two AI-based chess engines, trustworthy (competent, fair) and untrustworthy (incompetent, unfair), that coached participants by suggesting chess moves in three games against another engine opponent. We varied coaches' transparency about their competence (with the untrustworthy one always exaggerating its competence). We quantified and objectively measured participants' trust based on how often participants relied on coaches' move recommendations. Participants showed inability to assess AI competence by misplacing their trust with the untrustworthy AI, confirming its ability to deceive. Our work calls for design of interactions to help end-users assess AI trustworthiness.

Computational models that formalize complex human behaviors enable study and understanding of such behaviors. However, collecting behavior data required to estimate the parameters of such models is often tedious and resource intensive. Thus, estimating dataset size as part of data collection planning (also known as Sample Size Determination) is important to reduce the time and effort of behavior data collection while maintaining an accurate estimate of model parameters. In this article, we present a sample size determination method based on Uncertainty Quantification (UQ) for a specific Inverse Reinforcement Learning (IRL) model of human behavior, in two cases: (1)pre-hocexperiment design—conducted in the planning stage before any data is collected, to guide the estimation of how many samples to collect; and (2)post-hocdataset analysis—performed after data is collected, to decide if the existing dataset has sufficient samples and whether more data is needed. We validate our approach in experiments with a realistic model of behaviors of people with Multiple Sclerosis (MS) and illustrate how to pick a reasonable sample size target. Our work enables model designers to perform a deeper, principled investigation of the effects of dataset size on IRL model parameters.

Generative Adversarial Networks (GANs) can automatically generate quality images from learned model parameters. However, it remains challenging to explore and objectively assess the quality of all possible images generated using a GAN. Currently, model creators evaluate their GANs via tedious visual examination of generated images sampled from narrow prior probability distributions on model parameters. Here, we introduce an interactive method to explore and sample quality images from GANs. Our first two user studies showed that participants can use the tool to explore a GAN and select quality images. Our third user study showed that images sampled from a posterior probability distribution using a Markov Chain Monte Carlo (MCMC) method on parameters of images collected in our first study resulted in on average higher quality and more diverse images than existing baselines. Our work enables principled qualitative GAN exploration and evaluation.

Wikipedia articles aim to be definitive sources of encyclopedic content. Yet, only 0.6% of Wikipedia articles have high quality according to its quality scale due to insufficient number of Wikipedia editors and enormous number of articles. Supervised Machine Learning (ML) quality improvement approaches that can automatically identify and fix content issues rely on manual labels of individual Wikipedia sentence quality. However, current labeling approaches are tedious and produce noisy labels. Here, we propose an automated labeling approach that identifies the semantic category (e.g., adding citations, clarifications) of historic Wikipedia edits and uses the modified sentences prior to the edit as examples that require that semantic improvement. Highest-rated article sentences are examples that no longer need semantic improvements. We show that training existing sentence quality classification algorithms on our labels improves their performance compared to training them on existing labels. Our work shows that editing behaviors of Wikipedia editors provide better labels than labels generated by crowdworkers who lack the context to make judgments that the editors would agree with.

Public benefits programs help people afford necessities like food, housing, and healthcare. In the US, such programs are means-tested: applicants must complete long forms to prove financial distress before receiving aid. Online benefits screening tools provide a gloss of such forms, advising households about their eligibility prior to completing full applications. If incorrectly implemented, screening tools may discourage qualified households from applying for benefits. Unfortunately, errors in screening tools are difficult to detect because they surface one at a time and difficult to contest because unofficial determinations do not generate a paper trail. We introduce a method for auditing such tools in four steps: 1) generate test households, 2) automatically populate screening questions with household information and retrieve determinations, 3) translate eligibility guidelines into computer code to generate ground truth determinations, and 4) identify conflicting determinations to detect errors. We illustrated our method on a real screening tool with households modeled from census data. Our method exposed major errors with corresponding examples to reproduce them. Our work provides a necessary corrective to an already arduous benefits application process.

Improving mobile keyboard typing speed increases in value as more tasks move to a mobile setting. Autocorrect reduces the time it takes to manually fix typing errors, which results in typing speed increase. However, recent user studies uncovered an unexplored side-effect: participants' aversion to typing errors despite autocorrect. We present a computational model of typing on keyboards with autocorrect, which enables precise study of expert typists' aversion to typing errors on such keyboards. Unlike empirical typing studies that last days, our model evaluates this phenomenon for any autocorrect accuracy in seconds. We show that typists' aversion to typing errors imposes a limit on upper bound typing speeds, even for highly accurate autocorrect. Our findings motivate future keyboard designs that reduce typists' aversion to typing errors to increase typing speeds.

HCI has become particularly interested in using machine learning (ML) to improve user experience (UX). However, some design researchers claim that there is a lack of design innovation in envisioning how ML might improve UX. We investigate this claim by analyzing 2,494 related HCI research publications. Our review confirmed a lack of research integrating UX and ML. To help span this gap, we mined our corpus to generate a topic landscape, mapping out 7 clusters of ML technical capabilities within HCI. Among them, we identified 3 under-explored clusters that design researchers can dig in and create sensitizing concepts for. To help operationalize these technical design materials, our analysis then identified value channels through which the technical capabilities can provide value for users: self, context, optimal, and utility-capability. The clusters and the value channels collectively mark starting places for envisioning new ways for ML technology to improve people's lives.

Glanceability and low access time are arguably the key assets of a smartwatch. Smartwatches are designed for, and excel at micro-interactions- simple tasks that only take seconds to complete. However, if a user desires to transition to a task requiring sustained usage, we show that there are additional factors that prevent possible longer usage of the smartwatch. In this paper, we conduct a study with 18 participants to empirically demonstrate that interacting with the smartwatch on the wrist leads to fatigue after only a few minutes. In our study, users performed three tasks in two different poses while using a smartwatch. We demonstrate that only after three minutes of use, the change in perceived exertion of the user was anchored as "somewhat strong" on the Borg CR10 survey scale. These results place an upper bound for smartwatch usage that needs to be considered in application and interaction design.

Text entry is an increasingly important activity for mobile device users. As a result, increasing text entry speed of expert typists is an important design goal for physical and soft keyboards. Mathematical models that predict text entry speed can help with keyboard design and optimization. Making typing errors when entering text is inevitable. However, current models do not consider how typists themselves reduce the risk of making typing errors (and lower error frequency) by typing more slowly. We demonstrate that users respond to costly typing errors by reducing their typing speed to minimize typing errors. We present a model that estimates the effects of risk aversion to errors on typing speed. We estimate the magnitude of this speed change, and show that disregarding the adjustments to typing speed that expert typists use to reduce typing errors leads to overly optimistic estimates of maximum errorless expert typing speeds.

An ability to detect behaviors that negatively impact people's wellbeing and show people how they can correct those behaviors could enable technology that improves people's lives. Existing supervised machine learning approaches to detect and generate such behaviors require lengthy and expensive data labeling by domain experts. In this work, we focus on the domain of routine behaviors, where we model routines as a series of frequent actions that people perform in specific situations. We present an approach that bypasses labeling each behavior instance that a person exhibits. Instead, we weakly label instances using people's demonstrated routine. We classify and generate new instances based on the probability that they belong to the routine model. We illustrate our approach on an example system that helps drivers become aware of and understand their aggressive driving behaviors. Our work enables technology that can trigger interventions and help people reflect on their behaviors when those behaviors are likely to negatively impact them.

Human routines are blueprints of behavior, which allow people to accomplish purposeful repetitive tasks at many levels, ranging from the structure of their day to how they drive through an intersection. People express their routines through actions that they perform in the particular situations that triggered those actions. An ability to model routines and understand the situations in which they are likely to occur could allow technology to help people improve their bad habits, inexpert behavior, and other suboptimal routines. However, existing routine models do not capture the causal relationships between situations and actions that describe routines. Our main contribution is the insight that byproducts of an existing activity prediction algorithm can be used to model those causal relationships in routines. We apply this algorithm on two example datasets, and show that the modeled routines are meaningful-that they are predictive of people's actions and that the modeled causal relationships provide insights about the routines that match findings from previous research. Our approach offers a generalizable solution to model and reason about routines.

Driven by curiosity and our own three diverse smartphone application usage datasets, we sought to unpack the nuances of mobile device use by revisiting two recent Mobile HCI studies [1, 17]. Our goal was to add to our broader understanding of smartphone usage by investigating if differences in mobile device usage occurred not only across our three datasets, but also in relation to prior work. We found differences in the top-10 apps in each dataset, in the durations and types of interactions as well as in micro-usage patterns. However, it proved very challenging to attribute such differences to a specific factor or set of factors: was it the time frame in which the studies were executed? The recruitment procedure? The experimental method? Using our somewhat troubled analysis, we discuss the challenges and issues of conducting mobile research of this nature and reflect on caveats related to the replicability and generalizability of such work.

Location prediction enables us to use a person's mobility history to realize various applications such as efficient temperature control, opportunistic meeting support, and automated receptionists. Indoor location prediction is a challenging problem, particularly due to a high density of possible locations and short transition distances between these locations. In this paper we present Indoor-ALPS, an Adaptive Indoor Location Prediction System that uses temporal-spatial features to create individual daily models for the prediction of when a user will leave their current location (transition time) and the next location she will transition to. We tested Indoor-ALPS on the Augsburg Indoor Location Tracking Benchmark and compared our approach to the best performing temporal-spatial mobility prediction algorithm, Prediction by Partial Match (PPM). Our results show that Indoor-ALPS improves the temporal-spatial prediction accuracy over PPM for look-aheads up to 90 minutes by 6.2%, and for up to 30 minute look-aheads by 10.7%. These results demonstrate that Indoor-ALPS can be used to support a wide variety of indoor mobility prediction-based applications.

Mobile devices have become powerful ultra-portable personal computers supporting not only communication but also running a variety of complex, interactive applications. Because of the unique characteristics of mobile interaction, a better understanding of the time duration and context of mobile device uses could help to improve and streamline the user experience. In this paper, we first explore the anatomy of mobile device use and propose a classification of use based on duration and interaction type: glance, review, and engage. We then focus our investigation on short review interactions and identify opportunities for streamlining these mobile device uses through proactively suggesting short tasks to the user that go beyond simple application notifications. We evaluate the concept through a user evaluation of an interactive lock screen prototype, called ProactiveTasks. We use the findings from our study to create and explore the design space for proactively presenting tasks to the users. Our findings underline the need for a more nuanced set of interactions that support short mobile device uses, in particular review sessions.

Sighted individuals often develop significant knowledge about their environment through what they can visually observe. In contrast, individuals who are visually impaired mostly acquire such knowledge about their environment through information that is explicitly related to them. This paper examines the practices that visually impaired individuals use to learn about their environments and the associated challenges. In the first of our two studies, we uncover four types of information needed to master and navigate the environment. We detail how individuals' context impacts their ability to learn this information, and outline requirements for independent spatial learning. In a second study, we explore how individuals learn about places and activities in their environment. Our findings show that users not only learn information to satisfy their immediate needs, but also to enable future opportunities -- something existing technologies do not fully support. From these findings, we discuss future research and design opportunities to assist the visually impaired in independent spatial learning.

One of the fundamental operations in today's user interfaces is pointing to targets, such as menus, buttons, and text. Making an error when selecting those targets in real-life user interfaces often results in some cost to the user. However, the existing target-directed pointing models do not consider the cost of error when predicting task completion time. In this paper, we present a model based on expected value theory that predicts the impact of the error cost on the user's completion time for target-directed pointing tasks. We then present a target-directed pointing user study, which results show that time-based costs of error significantly impact the user's performance. Our results also show that users perform according to an expected completion time utility function and that optimal performance computed using our model gives good prediction of the observed task completion times.

General-purpose software applications are usually not tailored for a specific user with specific tasks, strategies or preferences. In order to achieve optimal performance with such applications, users typically need to transition to an alternative efficient behavior. Often, features of such alternative behaviors are not initially accessible and first need to be customized. However, few research works formally study and empirically measure what drives a user to customize. In this paper, we describe the challenges involved in empirically studying customization behaviors, and propose a methodology for formally measuring the impact of potential customization factors. We then demonstrate this methodology by studying the impact of different customization factors on customization behaviors. Our results show that increasing exposure and awareness of customization features, and adding social influence can significantly affect the user's customization behavior.

Learning an environment can be challenging for people with visual impairments. Braille maps allow their users to understand the spatial relationship between a set of places. However, physical Braille maps are often costly, may not always cover an area of interest with sufficient detail, and might not present up-to-date information. We built a handheld system for representing geographical information called SpaceSense, which includes custom spatial tactile feedback hardware-multiple vibration motors attached to different locations on a mobile touch-screen device. It offers high-level information about the distance and direction towards a destination and bookmarked places through vibrotactile feedback to help the user maintain the spatial relationships between these points. SpaceSense also adapts a summarization technique for online user reviews of public and commercial venues. Our user study shows that participants could build and maintain the spatial relationships between places on a map more accurately with SpaceSense compared to a system without spatial tactile feedback. They pointed specifically to having spatial tactile feedback as the contributing factor in successfully building and maintaining their mental map.

We present Waken, an application-independent system that recognizes UI components and activities from screen captured videos, without any prior knowledge of that application. Waken can identify the cursors, icons, menus, and tooltips that an application contains, and when those items are used. Waken uses frame differencing to identify occurrences of behaviors that are common across graphical user interfaces. Candidate templates are built, and then other occurrences of those templates are identified using a multi-phase algorithm. An evaluation demonstrates that the system can successfully reconstruct many aspects of a UI without any prior application-dependant knowledge. To showcase the design opportunities that are introduced by having this additional meta-data, we present the Waken Video Player, which allows users to directly interact with UI components that are displayed in the video.