Even as virtual reality devices and applications become more pervasive, our understanding of user behavior remains in its infancy. Typical studies involve some combination of post hoc summative evaluation (i.e., questionnaires after an experience is finished) and interaction log analysis (i.e., head position and clicks). Such methods do not illuminate the reasons, intentions, and causality underlying the user's actions. Our research aims to bridge this gap in the context of one specific application area: 360° video viewing. The fundamental question we ask is why users pan when they do. We present a study of users (n = 24) viewing 360° video and reporting their intentions each time they move their head to look at something new. By categorizing viewer behavior with detailed annotations, we established a structured framework for understanding viewer engagement. Additionally, we introduced a compiled dataset that merged pan-action data with corresponding user-reported motivations, presenting a valuable asset for subsequent research on VR viewer behavior. Through our approach, we facilitate the comprehension of human interaction within VR contexts, providing a foundational tool for future studies to incorporate viewer intentions with their movements.

Clear instructions are a necessity for obtaining accurate results from crowd workers. Even small ambiguities can force workers to choose an interpretation arbitrarily, resulting in errors and inconsistency. Crisp instructions require significant time to design, test, and iterate. Recent approaches have engaged workers to detect and correct ambiguities. However, this process increases the time and money required to obtain accurate, consistent results. We present TaskLint, a system to automatically detect problems with task instructions. Leveraging a diverse set of existing NLP tools, TaskLint identifies words and sentences that might foretell worker confusion. This is analogous to static analysis tools for code ("linters"), which detect possible features in code that might indicate the presence of bugs. Our evaluation of TaskLint using task instructions created by novices confirms the potential for static tools to improve task clarity and the accuracy of results, while also highlighting several challenges.

Modern manufacturing processes are in a state of flux, as they adapt to increasing demand for flexible and self-configuring production. This poses challenges for training workers to rapidly master new machine operations and processes, i.e. machine tasks. Conventional in-person training is effective but requires time and effort of experts for each worker trained and not scalable. Recorded tutorials, such as video-based or augmented reality (AR), permit more efficient scaling. However, unlike in-person tutoring, existing recorded tutorials lack the ability to adapt to workers’ diverse experiences and learning behaviors. We present AdapTutAR, an adaptive task tutoring system that enables experts to record machine task tutorials via embodied demonstration and train learners with different AR tutoring contents adapting to each user’s characteristics. The adaptation is achieved by continually monitoring learners’ tutorial-following status and adjusting the tutoring content on-the-fly and in-situ. The results of our user study evaluation have demonstrated that our adaptive system is more effective and preferable than the non-adaptive one.

Redacting faces in images is trivial when the number of faces is small and the annotator is trusted. For large batches, automated face detection has been the only currently viable solution, yet even the best ML-based solutions have error rates that would be unacceptable for sensitive applications. Crowd-based face detection/redaction systems exist, yet the process and the cost make them not feasible. We present Pterodactyl, a system for detecting (and redacting) faces at scale. It uses the AdaptiveFocus filter, which splits the image into smaller regions and uses machine learning to select a median filter for each region to hide the facial identities in the image while simultaneously allowing those faces to be detectable by crowd workers. The filter uses a convolutional neural network trained on images associated with the median filter level that allows detection and prevents identification. This filter allows Pterodactyl to achieve human-level detection with just 14% crowd labor as another recent crowd-based face detection/redaction system (IntoFocus). Our evaluation found that the redaction accuracy was higher than a commercial machine-based application and on par with IntoFocus while requiring 86% less crowd work (number of comparable tasks).

Mobile robots and IoT (Internet of Things) devices can increase productivity, but only if they can be programmed by workers who understand the domain. This is especially true in manufacturing. Visual programming in the spatial context of the operating environment can enable mental models at a familiar level of abstraction. However, spatial-visual programming is still in its infancy; existing systems lack IoT integration and fundamental constructs, such as functions, that are essential for code reuse, encapsulation, or recursive algorithms. We present Vipo, a spatial-visual programming system for robot-IoT workflows. Vipo was designed with input from managers at six factories using mobile robots. Our user study (n=22) evaluated efficiency, correctness, comprehensibility of spatial-visual programming with functions.

Redaction of private information from images is the kind of tedious, yet context-independent, task for which crowdsourcing is especially well suited. Despite tremendous progress, machine learning is not keeping pace with the needs of sensitive applications in which inadvertent disclosure could have real-world consequences. Human workers can detect faces that machines cannot; however, an open call to crowds would entail disclosure. We present IntoFocus, a method for engaging crowd workers to redact faces from images without disclosing the facial identities of people depicted. The method works iteratively, starting with a heavily filtered form of the image, and gradually reducing the strength of the filter, with a different set of workers reviewing the image at each step. IntoFocus exploits the gap between the filter level at which a face becomes unidentifiable and the level at which it becomes undetectable. To calibrate the algorithm, we performed a perceptual study of detection and identification of faces in images filtered with the median filter. We present the system design, the results of the perception study, and the results of a summative evaluation of the system

Crowdsourcing has the inherent potential to shift tedious work from requesters with limited time (or patience) to an on-demand workforce. However, the time saved by requesters is offset by the time they must spend preparing instructions and refining them to address the ambiguities that typically arise. Hastily written instructions should be welcomed and supported. This paper presents a set of methods that enable workers to cope with unclear or ambiguous instructions and produce high quality results with minimal reliance on the requester. Workers’ intuition about the requester’s needs is leveraged to move onus for answering questions and revising instructions to workers. Our system, WingIt, implements these methods and demonstrates the relative tradeoffs between resolving by questions versus editing instructions directly, and between waiting for immediate response from the requester versus allowing workers to proceed based on their best guess of the requester’s intent.

Design contests and group creativity support systems have demonstrated the value of crowds for producing a solution to a design or engineering problem. However, when the goal is not one idea, but many ideas, an uncoordinated crowd effort would likely lead to a set of ideas clustered around those that are easiest to think of. We present BlueSky, a crowd-powered system that coordinates microtask workers to enumerate a uniform sample of textual ideas on a given topic. Through the process of soliciting ideas, an ontology is developed, which is used to categorize the ideas in the list. That categorization then reveals combinations of attributes that have not yet been covered. Our evaluation with four list topics compared BlueSky to freeform solicitation of ideas with respect to comprehensiveness (coverage over the entire idea space) and uniformity (mitigating the tendency to emphasize ideas that are easy to think of).

Task instruction quality is widely presumed to affect outcomes, such as accuracy, throughput, trust, and worker satisfaction. Best practices guides written by experienced requesters share their advice about how to craft task interfaces. However, there is little evidence of how specific task design attributes affect actual outcomes. This paper presents a set of studies that expose the relationship between three sets of measures: (a) workers’ perceptions of task quality, (b) adherence to popular best practices, and (c) actual outcomes when tasks are posted (including accuracy, throughput, trust, and worker satisfaction). These were investigated using collected task interfaces, along with a model task that we systematically mutated to test the effects of specific task design guidelines.

The wealth of resources online has empowered individuals and businesses with an unprecedented volume of information to aid in decision making processes. However, finding the many details needed for a non-trivial decision can be very labor-intensive. We present AskSheet, a general system that leverages human computation to acquire the inputs to an arbitrary decision spreadsheet provided by the user. The key innovation is the ability to prioritize the inputs by analyzing the user's spreadsheet formulas to calculate value of information for each of the blanks. By directing workers to find the details that impact the end result most, it results in a conclusive decision without gathering all of the inputs.

Targeted paraphrasing is a new approach to the problem of obtaining cost-effective, reasonable quality translation, which makes use of simple and inexpensive human computations by monolingual speakers in combination with machine translation. The key insight behind the process is that it is possible to spot likely translation errors with only monolingual knowledge of the target language, and it is possible to generate alternative ways to say the same thing (i.e., paraphrases) with only monolingual knowledge of the source language. Formal evaluation demonstrates that this approach can yield substantial improvements in translation quality, and the idea has been integrated into a broader framework for monolingual collaborative translation that produces fully accurate, fully fluent translations for a majority of sentences in a real-world translation task, with no involvement of human bilingual speakers.

Today’s mobile devices are equipped with a variety of tools that enable users to capture and share their daily experiences. However, designing authoring tools that effectively integrate the discrete media-capture components of mobile devices to enable rich expression---especially by children---remains a challenge. Evaluating such tools authentically, as they are being used in-situ, can be even more challenging. We detail a long-term, multimethod study on the use of StoryKit, a mobile storytelling application. By taking advantage of a public distribution channel, we were able to evaluate StoryKit’s use on a scale beyond that usually found in lab settings or limited field trials. Our results show that StoryKit’s simple but well-integrated interface attracted a high number of dedicated users in education contexts at all levels, including children with special learning needs. We include a discussion of the challenges and opportunities that similar “in the wild” studies hold for HCI research.

The rapid growth of human computation within research and industry has produced many novel ideas aimed at organizing web users to do great things. However, the growth is not adequately supported by a framework with which to understand each new system in the context of the old. We classify human computation systems to help identify parallels between different systems and reveal "holes" in the existing work as opportunities for new research. Since human computation is often confused with "crowdsourcing" and other terms, we explore the position of human computation with respect to these related topics.

Informal educational experiences with grandparents and other older adults can be an important component of children's education, especially in circumstances where high quality educational services and facilities are not readily available. Mobile devices offer unique capabilities to support such interactions. We report on an ongoing participatory design project with an intergenerational design group to create mobile applications for reading and editing books, or even creating all new stories on an Apple iPhone.

Displaying scanned book pages in a web browser is difficult, due to an array of characteristics of the common user's configuration that compound to yield text that is degraded and illegibly small. For books which contain only text, this can often be solved by using OCR or manual transcription to extract and present the text alone, or by magnifying the page and presenting it in a scrolling panel. Books with rich illustrations, especially children's picture books, present a greater challenge because their enjoyment is dependent on reading the text in the context of the full page with its illustrations. We have created two novel prototypes for solving this problem by magnifying just the text, without magnifying the entire page. We present the results of a user study of these techniques. Users found our prototypes to be more effective than the dominant interface type for reading this kind of material and, in some cases, even preferable to the physical book itself.

Electronic Health Records (EHRs) and other temporal databases contain hidden patterns that reveal important cause-and-effect phenomena. Finding these patterns is a challenge when using traditional query languages and tabular displays. We present an interactive visual tool that complements query formulation by providing operations to align, rank and filter the results, and to visualize estimates of the intervals of validity of the data. Display of patient histories aligned on sentinel events (such as a first heart attack) enables users to spot precursor, co-occurring, and aftereffect events. A controlled study demonstrates the benefits of providing alignment (with a 61% speed improvement for complex tasks). A qualitative study and interviews with medical professionals demonstrates that the interface can be learned quickly and seems to address their needs.