Human speech often contains ambiguity stemming from the use of demonstrative pronouns (DPs), such as “this” and “these.” While we can typically decipher which objects of interest DPs are referring to based on context, modern day voice assistants (VAs – such as Google Assistant and Siri) are yet unable to process queries containing such ambiguity. For instance, to humans, a question such as “how much is this?” can be clarified through visual reference (e.g., a buyer gestures to the seller the object they would like to purchase). To bridge this gap between human and machine cognition, we built and examined a touch + voice multimodal VA prototype that enables users to select key spatial information to embed as context and query the VA. The prototype converts results of mobile, real-time object recognition and optical character recognition models into augmented reality buttons that represent features. Users can interact with and modify the selected features through a word grid. We conducted a study to investigate: 1) how touch performs as an additional modality to resolve ambiguity in queries, 2) how users use DPs when interacting with VAs, and 3) how users perceive a VA that can understand DPs. From this procedure we found that as the query becomes more complex, users prefer the multimodal VA over the standard VA without experiencing elevated cognitive load. Additionally, even though it took some time getting used to, many participants eventually became comfortable with using DPs to interact with the multimodal VA and appreciated the improved human-likeness of human-VA conversations.

With computational algorithms making an increasing number of deeply consequential, and often problematic judgments on our behalf, there is a growing interest in slowing down technology to encourage users to reflect on judgments made by algorithms. Prior work in slow technology has established slowness as an agent of reflection and serendipity; however, it has been unclear whether this waiting time actually helps users gain useful insight or any other benefits as they make judgments using an algorithm. To this end, we conducted a series of online and in-person between-subject user studies in which we isolate the impact of an algorithm's speed on how users incorporate the algorithm's advice when making judgments in the context of simple visual recognition tasks. We find that our participants followed good quality algorithms more and bad quality algorithms somewhat less if the response time of the algorithm is slower. Furthermore, qualitative analysis of the in-person study interviews reveals that the waiting was not time wasted, but was often used to reflect on the task and the estimation process of themselves and the algorithm, and to compare and reevaluate the two processes. Based on these findings, we outline design implications of future algorithmic systems.

Many students underperform on exams due to experiencing high test anxiety. We report on a study comparing a novel intervention of seeking support from one's social network to the more common approaches of expressive writing and studying task-relevant materials for simulated open-ended test questions. We measured in-the-moment (state) anxiety before and after each intervention, and correctness of the solutions. We also surveyed students to learn about their perceptions of the interventions. Our results showed that social support decreased the anxiety of high test-anxious students by 21% with the reduction in anxiety correlating with the number of messages received. Social support also allowed high test-anxious students to score at the level of low test-anxious students. Expressive writing showed a similar effect, but increased the anxiety of low test-anxious students by 61%. Studying task materials had no effect on anxiety and high test-anxious students performed worse than low test-anxious students. Despite benefiting from social support, we found that students were uncomfortable soliciting support from their online social network. Realizing the benefits of this approach may therefore require different formulations of social support in practice.

Many online platforms use curation algorithms that are opaque to the user. Recent work suggests that discovering a filtering algorithm's existence in a curated feed influences user experience, but it remains unclear how users reason about the operation of these algorithms. In this qualitative laboratory study, researchers interviewed a diverse, non-probability sample of 40 Facebook users before, during, and after being presented alternative displays of Facebook's News Feed curation algorithm's output. Interviews revealed 10 "folk theories' of automated curation, some quite unexpected. Users who were given a probe into the algorithm's operation via an interface that incorporated "seams,' visible hints disclosing aspects of automation operations, could quickly develop theories. Users made plans that depended on their theories. We conclude that foregrounding these automated processes may increase interface design complexity, but it may also add usability benefits.

The modality students use to complete programming-centric exams is shifting from paper to computer. We developed and distributed a survey to investigate how exam modality affects student perceptions of test anxiety and performance during programming-centric exams. We examine the distribution of modality preference among students and understand how students cope with test anxiety. We found a majority of students report at least moderate perceived anxiety (rating -- 4 on 7-pt scale) on paper- (69%) and computer-based (64%) exams. Though moderate anxiety was experienced by a majority of the students, we found 1 in 5 students had no strategy to cope with anxiety. Additionally, we examine how students perceived anxiety during testing affects their thought process and behavior. We found computer-based exams were perceived as more beneficial to the quality and speed of student's solutions and a majority of students preferred computer-based testing (67%). Lastly, we introduce possible technological solutions to benefit students experiencing test anxiety while not impeding less anxious students.

Expanding and improving design knowledge is a vital part of higher education due to the growing demand for employees who can think both critically and creatively. However, developing effective methods for assessing what students have learned in design courses is one of the most elusive challenges of design education due to the subjective nature of design. For example, evaluating design outcomes is problematic due to the common pattern of increasing enrollments and reduced resources for design instruction. In this article, we propose and evaluate a new assessment method that uses a novel application of Bayesian Truth Serum (BTS), a scoring algorithm, in order to provide a scalable and reliable measure of design knowledge. This method requires no subjective input from the design instructor, nor does it require answers to questions that have distinct right or wrong answers. We tested this method over a 4-week period with 71 design students in an upper-level design course. For the study, participants were asked to provide responses to multiple-choice BTS survey questions, generate ideas for a design problem, and provide feedback on other participants' ideas. The survey data were used to calculate BTS indices of expertise and statistical tests were performed to determine how the indices correlated with participant ideation and critique proficiency. The results from this study show a modest correlation between the BTS indices of expertise and later performance on generative design tasks and a correlation between the students' ability to critique designs and their BTS scores. These findings suggest that the BTS assessment method can be used to supplement existing evaluation practices for individual design assessment, particularly in courses where group projects are used as the primary means of evaluation. In addition, the results show promise for using the BTS method in classes where design projects or design critiques are not feasible due to time constraints or large class sizes.

A common framework for studying perception and performance in both human-technology interaction and music is presented. The framework represents the cognitive challenges faced by both musicians and human operators in technological systems. In the perceptual realm, both must infer distal or covert states from proximally available information (e.g., inferring the emotional meaning of a musical composition; inferring the functional meaning of interface displays). In the realm of action, both must select proximally available actions, or means, to achieve distal ends or goals (e.g., a conductor using hand movements to direct an orchestra, an operator using interface controls to tune an industrial process or respond to a fault). The framework represents these proximal-distal relations and enables quantitative measurement of the degree to which performers adapt to them. The framework is illustrated with a review of music and human-technology interaction research and our own study of the coordination between a professional conductor's hand movements and a concertmaster's bowing actions in the opening of Beethoven's Fifth Symphony. In providing a common theoretical framework for both music and engineering, we hope to enhance prospects for research on group musical performance to inspire novel, robust design models for modern sociotechnical systems.

Performers in time-stressed, information-rich tasks develop rule-based, simplification strategies to cope with the severe cognitive demands imposed by judgment and decision making. Linear regression modeling, proven useful for describing judgment in a wide range of static tasks, may provide misleading accounts of these heuristics. That approach assumes cue-weighting and cue-integration are well described by compensatory strategies. In contrast, evidence suggests that heuristic strategies in dynamic tasks may instead reflect rule-based, noncompensatory cue usage. We therefore, present a technique called genetics-based policy capturing (GBPC) for inferring noncompensatory rule-based heuristics from judgment data as an alternative to regression. In GBPC, rule-base representation and search uses a genetic algorithm, and fitting the model to data using multiobjective optimization to maximize fit on three dimensions: completeness (all human judgments are represented); specificity (maximal concreteness); and parsimony (no unnecessary rules are used). GBPC is illustrated using data from the highest and lowest scoring participants in a simulated dynamic, combat information center (CIC) task. GBPC inferred rule-bases for these two performers that shed light on both skill and error. We compare the GBPC results with regression-based lens modeling of the same data set, and discuss how the GBPC results allowed us to interpret the high scoring performer's highly significant use of unmodeled knowledge (C=1) revealed by lens model analysis. The GBPC findings also allow us to now interpret a similarly high use of unmodeled knowledge (C=1)in a previously published lens model analysis of a different data set collected in the same experimental task. We conclude by discussing training implications, and also prospects for the development of integrated GBPC models of both human judgment and the task environment, thus providing a noncompensatory formulation of the lens model (a genetics-based lens model, or GBLM) of the integrated human-environment system.

The paper proposes a methodology to understand human performance in complex supervisory control domains through the use of inductive inference methods, and E. Brunswik's (1955) lens framework. Operators in today's highly automated control systems are confronted with vast arrays of information. In order to design effective workspaces in these domains, therefore, researchers must understand information demand and utilization from a systems perspective. That is, to understand system information requirements, one must attempt to understand the diagnosticity of information in the task domain, as well as the utility of the information by human operators. The paper outlines a framework, in the spirit of Brunswik's lens model, to represent information utility and diagnosticity. Within this framework, a computational inductive inference model is described which seeks to capture human decision policies, as well as to provide a method of comparing these policies to the information environment.

We investigated performance in a complex, dynamic decision-making task. Individual judgment performance in a command and control environment was modeled as linear combinations of environmental cue values using a lens model approach. Examination of the judgment models indicated that participants had similar judgment policies, while correlational and error analyzes indicated that performance differences were due to participants' abilities to execute judgment strategies rather than their knowledge of the task environment. The research demonstrated how the lens model approach can be extended to account for dynamic aspects of decision making in complex environments, through the use of individual, time dependent environmental models for each participant. Additionally, the research showed that a lens model approach is useful for characterizing factors in individual performance in complex judgment tasks. Empirically, this research suggests that training on task knowledge should be supplemented by training which focuses on the consistent execution of judgment strategies.

An understanding of how both psychological and environmental factors mutually constrain skilled behavior is required to effectively support human activity. As a step toward meeting this need, a process model of skilled human interaction with a dynamic and uncertain environment is presented. The model was able to mimic human behavior in a laboratory task requiring one- and two-person crews to direct the activities of a fleet of agents to locate and process valued objects in a simulated world. The process model is a pair of highly interactive components that together mimic the behavior of the human-environment system. One component is a representation of the external environment as a dynamically changing set of opportunities for action. The second component is a dynamic representation of skilled human decision making and planning behavior within the environment so described. The process model is an expression of a general theory of skilled interaction assuming that perception and action mechanisms sensitive to environmental constraints are responsible for generating much of behavior, and where the need for additional cognitive processing of internal representations may result from environmental designs that do not adequately support the perceptual guidance of activity.>

A research program using a microworld methodology for investigating human performance in a dynamic uncertain environment is described. One- and two-person crews performed a supervisory control task that involved piloting a low-fidelity scout helicopter and supervising four additional helicopters. The simulated environment contained cargo and enemy craft that had to be discovered by searching a 100 mi/sup 2/ forested region. The task required planning, intervention, and several types of resource management. An expert one-person crew performed comparably to novice two-person crews, and both of these were superior to novice one-person crews. More detailed analyses of performance revealed several types of limited adaptivity in the behavior of the novice crews.>

A model of skilled human performance in a complex laboratory task is presented. The task required human crews to perform manual and supervisory control of multiple craft to discover and process objects in a simulated world. The model could perform the task as well as the most highly skilled human crews and could be parameterized to mimic the behavior of crews that differed in both crew size and experience level. The model's 'perceptual system' consists of a set of cue-weighted functions that map displayed information directly into values indicating the appropriateness of various actions. The perceptual system processes an 'image' prediction of the environment to plan future actions. A central mechanism detects and resolves potential conflicts between the simulated craft's actions, and a hierarchically structured set of action mechanisms simulates the manipulation of interface controls. Relationships with other models of skilled performance are discussed.>