Marginalized communities often face interconnected barriers that undermine well-being, yet design methods rarely explicitly account for how disadvantages compound or how strengths can reinforce each other. Building on Capability Sensitive Design (CSD), this research extends the framework to address corrosive disadvantages—barriers that undermine multiple capabilities—and fertile functionings—capabilities that positively reinforce others. We applied this extended framework in a participatory study with newcomers to Canada. Using capability hierarchy mapping, co-design workshop, and field study, we identified key capability gaps and their interconnectedness, surfaced community knowledge, and translated values into actionable design requirements. Our findings show that explicitly mapping advantages and disadvantages enables more targeted, contextually grounded interventions. We conclude with methodological guidance for applying this approach to other marginalized contexts in HCI, where designing for equity requires accounting for how capabilities interact.

Agentic AI is being heralded as the next step in the development of AI systems. Agentics, complex ensembles of different machine learning, data processing, and generative AI models, can provide new autonomous and proactive decision-making capabilities to organizations, participate in complex workflows, and, when needed, seek guidance from and provide insights to human users in natural languages. Collectively, we wish to explore how and why to design agentic systems to be situated within specific social and organizational contexts, the value of social theory and perspectives to this work, and the potential of this move to address critical issues with AI. Given the focus on social and organization context as essential to agentic design, we see this work as directly related to the ACM COMPASS 2025 theme “computing in place”. We seek to bring together scholars from the diversity of disciplines within ACM to develop research agendas, projects, and joint teaching initiatives that support the development of social agentic design and analysis.

AI agents are commonly aligned with "human values" through reinforcement learning from human feedback (RLHF), where a single reward model is learned from aggregated human feedback and used to align an agent's behavior. However, human values are not homogeneous--different people hold distinct and sometimes conflicting values. Aggregating feedback into a single reward model risks disproportionately suppressing minority preferences. To address this, we present a novel reward modeling approach for learning individualized reward models. Our approach uses a language model to guide users through reflective dialogues where they critique agent behavior and construct their preferences. This personalized dialogue history, containing the user's reflections and critiqued examples, is then used as context for another language model that serves as an individualized reward function (what we call a "verbal reward model") for evaluating new trajectories. In studies with 30 participants, our method achieved a 9-12% improvement in accuracy over non-reflective verbal reward models while being more sample efficient than traditional supervised learning methods.

We investigate two approaches to designing neurotic behavior – the inability to deal with a stressful situation – and if such behavior can encourage more care shown towards a domestic robot. Robots are generally designed to be polite and selfless when working in daily situations, striving to be prosocial whenever possible. While negative behaviors such as neuroticism may initially seem undesirable, they serve a purpose in human-human interaction and may still be usable by robots in a positive way, such as communicating frustration, doubt, or anger. In our experiment, participants do an unrelated task as a Roomba cleans the room. In a 2x2 (valence and escalation) experiment on neurotic behavior design, the robot collides with objects while the participant is doing their task and displays a positive or negative valanced sound, and escalates (getting quicker or longer) or does not escalate that behavior. Our results showcase that designing neurotic behaviors is not simple, with escalation being a more effective control of perceived neuroticism, and that robot personality can indeed influence people, unprompted, to check on and interact with the robot. There was also no qualitative indication that this was perceived negatively by any of our participants. Our results demonstrate how neurotic behavior, stereotypically undesirable, can be useful in promoting positive interaction and prompts further investigation into non prosocial robot behaviors.

This research explores how to ease life transition of newcomers through the technology-mediated process of capability-based journaling, using a tool we created called the Life Transition Journal (LTJ). As part of an 8-week study with recently arrived international students, LTJ guided users by integrating structured planning and reflective practices to support physical and emotional well-being. Through qualitative analysis, we observed four types of behavioural changes in the form of increased exploration, organization, self-understanding and initiatives, as well as an awareness of mental well-being. Drawing on these findings, we share the design implications of leveraging capability-based plans and actions in digital journaling.

Curiosity-driven learning has shown significant positive effects on students’ learning experiences and outcomes. But despite this importance, reports show that children lack this skill, especially in formal educational settings.

Much HCI research on prompting prosocial behaviors focuses on methods for increasing empathy. However, increased empathy may have unintended negative consequences. Our work offers an alternative solution that encourages critical reflection for nurturing compassion, which involves motivation and action to help others. In a between-subject experiment, participants (N=60) viewed a climate change documentary while receiving no prompts (CON), reflective prompts to focus on their emotions (RE) or surprises (RS). State compassion, critical reflection, and motivation to act or learn were measured at the end of the session (post-video) and two weeks later (follow-up). Despite participants’ condition not affecting compassion, critical reflection was positively correlated with post-video state compassion. RE and RS participants demonstrated deeper reflection and reported higher motivation to learn post-video, and more prosocial behavioral changes during follow-up. RS participants reported better follow-up recall than RE participants. We conclude by discussing implications on designing technology to support compassion and longer-term critical reflection.

Teachable robots can offer benefits to students through the use of social behaviours, such as speech, gaze, and gestures, to promote engagement and learning. Adapting these behaviours can deliver personalised interactions to better suit each individual. There is a growing body of research utilising reinforcement learning in social robotics, however there is limited research in the use of adaptive dialog behaviours for social robots. We propose an adaptive response-selection algorithm for a teachable robot which aims to improve user engagement in the teaching task. The proposed approach uses Q-learning to learn an individualised policy. The algorithm is rewarded according to the time taken per teaching input, and the amount paraphrasing in the user’s response. A user study has been conducted to evaluate the algorithm, compared to a method of random response-selection. The results indicate that an adaptive approach learns to select more rewarding actions over time, and personalise to the individual user.

Benefits of learning by teaching (LbT) have been highlighted by previous studies from a pedagogical lens, as well as through computer-supported systems. However, the challenges that university students face in technology-mediated LbT---whether it be teaching oneself, teaching a peer, or teaching an agent---are not well understood. Furthermore, there is a gap in knowledge on the challenges that students encounter throughout the process of teaching (content selection, preparation, teaching, receiving and giving feedback, and reflection) despite its importance to the design of LbT platforms. Thus, we conducted a study with 24 university students where they taught content they had not fully grasped, without guidance, and participated in a semi-structured interview. Results demonstrate that participants encountered the following challenges: psychological barriers relating to self and others, and lack of know-how. Furthermore, we illuminate design implications required to overcome these challenges and benefit from LbT without requiring prior training in pedagogy.

With privacy-related concepts often abstract and difficult to define, comics can be an effective visual storytelling medium for explaining and raising awareness about privacy. However, existing privacy and security educational comics do not support content creation. To address this, we contribute PrivacyToon, a comic-based authoring tool that leverages concept-driven storytelling and ideation cards to help users create customizable privacy-related visual content. Our exploratory user study with 23 students and teachers shows PrivacyToon’s potential as a creative tool for communicating privacy concepts and stories. Our results show that a wide range of creativity preferences and contexts must be considered when designing systems that integrate ideation card-based design processes.

In this paper, we explore how expressive auditory gestures added to the speech of a pedagogical agent influence the human-agent relationship and learning outcomes. In a between-subjects experiment, 41 participants assumed the role of a tutor to teach a voice-based agent. The agent used either: expressive interjections (e.g.,“yay”, “hmm”, “oh”), brief expressive musical executions, or no auditory gestures at all (control condition), throughout the interaction. Overall, the results indicate that both gestures can positively affect the interaction, but in particular, interjections can significantly increase feelings of emotional rapport with the agent and enhance motivation in learners. The implications of our findings are discussed as our work adds to the understanding of conversational agent design and can be useful for education as well as other domains in which dialogue systems are used.

Recent work demonstrated how we can design and use coding strips, a form of comic strips with corresponding code, to enhance teaching and learning in programming. However, creating coding strips is a creative, time-consuming process. Creators have to generate stories from code (code↦story) and design comics from stories (story↦comic). We contribute CodeToon, a comic authoring tool that facilitates this code-driven storytelling process with two mechanisms: (1) story ideation from code using metaphor and (2) automatic comic generation from the story. We conducted a two-part user study that evaluates the tool and the comics generated by participants to test whether CodeToon facilitates the authoring process and helps generate quality comics. Our results show that CodeToon helps users create accurate, informative, and useful coding strips in a significantly shorter time. Overall, this work contributes methods and design guidelines for code-driven storytelling and opens up opportunities for using art to support computer science education.

Traditional crowdsourcing has mostly been viewed as requester-worker interaction where requesters publish tasks to solicit input from human crowdworkers. While most of this research area is catered towards the interest of requesters, we view this workflow as a teacher-learner interaction scenario where one or more human-teachers solve Human Intelligence Tasks to train machine learners. In this work, we explore how teachable machine learners can impact their human-teachers, and whether they form a trustable relation that can be relied upon for task delegation in the context of crowdsourcing. Specifically, we focus our work on teachable agents that learn to classify news articles while also guiding the teaching process through conversational interventions. In a two-part study, where several crowd workers individually teach the agent, we investigate whether this learning by teaching approach benefits human-machine collaboration, and whether it leads to trustworthy AI agents that crowd workers would delegate tasks to. Results demonstrate the benefits of the learning by teaching approach, in terms of perceived usefulness for crowdworkers, and the dynamics of trust built through the teacher-learner interaction.

Previous studies have highlighted the benefits of pedagogical conversational agents using socially-oriented conversation with students. In this work, we examine the effects of a conversational agent’s use of affiliative and self-defeating humour — considered conducive to social well-being and enhancing interpersonal relationships — on learners’ perception of the agent and attitudes towards the task. Using a between-subjects protocol, 58 participants taught a conversational agent about rock classification using a learning-by-teaching platform, the Curiosity Notebook. While all agents were curious and enthusiastic, the style of humour was manipulated such that the agent either expressed an affiliative style, a self-defeating style, or no humour. Results demonstrate that affiliative humour can significantly increase motivation and effort, while self-defeating humour, although enhancing effort, negatively impacts enjoyment. Findings further highlight the importance of understanding learner characteristics when using humour.

In this work, we designed a teachable robot that encourages a pair of students to discuss their thoughts and teaching decisions during the tutoring session. The robot adapts to the students’ talking activity and adjusts the frequency and type of encouragement. We hypothesize that the robot’s encouragement of group discussion can enhance the social engagement of group members, leading to improved learning and enjoyment. We ran a user study (n = 68), where a pair of participants (dyad) worked together to teach a humanoid robot about rocks and minerals. In the adaptive condition, the robot uses reinforcement learning to maximise interaction between the dyad members. Results show that the adaptive robot was successful in creating more dialogue between dyad members and in increasing task engagement, but did not affect learning or enjoyment. Over time, the adaptive robot was also able to encourage both members to contribute more equally to the conversation.

Recent work investigated the potential of comics to support the teaching and learning of programming concepts and suggested several ways coding strips, a form of comic strip with its corresponding code, can be used. Building on this work, we tested the recommended use cases of coding strip in an undergraduate introductory computer science course at a large comprehensive university. At the end of the course, we surveyed students to assess their experience and found they benefited in various ways. Our work contributes a demonstration of the various ways comics can be used in introductory CS courses and an initial understanding of benefits and challenges with using comics in computing education gleaned from an analysis of students' survey responses and code submissions.

While learning by teaching is a popular pedagogical technique, it is a learning phenomenon that is difficult to study due to variability in the tutor-tutee pairings and learning environments. In this paper, we introduce the Curiosity Notebook, a web-based research infrastructure for studying learning by teaching via the use of a teachable agent. We describe and provide rationale for the set of features that are essential for such a research infrastructure, outline how these features have evolved over two design iterations of the Curiosity Notebook and through two studies---a 4-week field study with 12 elementary school students interacting with a NAO robot and an hour-long online observational study with 41 university students interacting with an agent---demonstrate the utility of our platform for making observations of learning-by-teaching phenomena in diverse learning environments. Based on these findings, we conclude the paper by reflecting on our design evolution and envisioning future iterations of the Curiosity Notebook.

Over the years, concreteness fading has been used to design learning materials and educational tools for children. Unfortunately, it remains an underspecified technique without a clear guideline on how to design it, resulting in varying forms of concreteness fading and conflicting results due to the design inconsistencies. To our knowledge, no research has analyzed the existing designs of concreteness fading implemented across different settings, formulated a generic framework, or explained the design dimensions of the technique. This poses several problems for future research, such as lack of a shared vocabulary for reference and comparison, as well as barriers to researchers interested in learning and using this technique. Thus, to inform and support future research, we conducted a systematic literature review and contribute: (1) an overview of the technique, (2) a discussion of various design dimensions and challenges, and (3) a synthesis of key findings about each dimension. We open source our dataset to invite other researchers to contribute to the corpus, supporting future research and discussion on concreteness fading.

Software development is a complex endeavor that depends on a wide variety of contextual factors involving a large amount of distributed information such as technology-related tasks, software operating environments and stakeholder requirements. Most of this context is implicit and captured in the developers' minds (tacit) or distributed through volumes of documentation. Developers have to maintain mental models of this variety of tasks and information as they produce the software. As a result, context can be easily lost or forgotten and developers often use adhoc approaches while finishing the project. We present in this paper the preliminary results of a study that aims at analyzing qualitatively whether supporting software developers with a chatbot during task execution can improve the overall development experience. The chatbot can assist the developers in executing different tasks based on implicit contextual information. We propose an implementation to explore the viability of using textual chatbots to assist developers automatically and proactively with software development project activities that recur. We believe that understanding the interaction of developers with the systems supported by chatbots is key to improving the developer experience and advancing software engineering practices by providing needed timely support for developers.

Question asking is an important tool for constructing academic knowledge, and a self-reinforcing driver of curiosity. However, research has found that question asking is infrequent in the classroom and children's questions are often superficial, lacking deep reasoning. In this work, we developed a pedagogical agent that encourages children to ask divergent-thinking questions, a more complex form of questions that is associated with curiosity. We conducted a study with 95 fifth grade students, who interacted with an agent that encourages either convergent-thinking or divergent-thinking questions. Results showed that both interventions increased the number of divergent-thinking questions and the fluency of question asking, while they did not significantly alter children's perception of curiosity despite their high intrinsic motivation scores. In addition, children's curiosity trait has a mediating effect on question asking under the divergent-thinking agent, suggesting that question-asking interventions must be personalized to each student based on their tendency to be curious.

Autonomous vehicles have been rapidly progressing towards full autonomy using fixed driving styles, which may differ from individual passenger preferences. Violating these preferences may lead to passenger discomfort or anxiety. We studied passenger responses to different driving style parameters in a physical autonomous vehicle. We collected galvanic skin response, heart rate, and eye-movement patterns from 20 participants, along with self-reported comfort and anxiety scores. Our results show that the presence and proximity of a lead vehicle not only raised the level of all measured physiological responses, but also exaggerated the existing effect of the longitudinal acceleration and jerk parameters. Skin response was also found to be a significant predictor of passenger comfort and anxiety. By using multiple independent events to isolate different driving style parameters, we demonstrate a method to control and analyze such parameters in future studies.

Artificial intelligence (AI) assistants for clinical decision making show increasing promise in medicine. However, medical assessments can be contentious, leading to expert disagreement. This raises the question of how AI assistants should be designed to handle the classification of ambiguous cases. Our study compared two AI assistants that provide classification labels for medical time series data along with quantitative uncertainty estimates: conventional vs. ambiguity-aware. We simulated our ambiguity-aware AI based on real-world expert discussions to highlight cases likely to lead to expert disagreement, and to present arguments for conflicting classification choices. Our results demonstrate that ambiguity-aware AI can alter expert workflows by significantly increasing the proportion of contentious cases reviewed. We also found that the relevance of AI-provided arguments (selected from guidelines either randomly or by experts) affected experts' accuracy at revising AI-suggested labels. Our work contributes a novel perspective on the design of AI for contentious clinical assessments.

Autonomous Vehicles (AV) will transform transportation, but also the interaction between vehicles and pedestrians. In the absence of a driver, it is not clear how an AV can communicate its intention to pedestrians. One option is to use visual signals. To advance their design, we conduct four human-participant experiments and evaluate six representative AV visual signals for visibility, intuitiveness, persuasiveness, and usability at pedestrian crossings. Based on the results, we distill twelve practical design recommendations for AV visual signals, with focus on signal pattern design and placement. Moreover, the paper advances the methodology for experimental evaluation of visual signals, including lab, closed-course, and public road tests using an autonomous vehicle. In addition, the paper also reports insights on pedestrian crosswalk behaviours and the impacts of pedestrian trust towards AVs on the behaviors. We hope that this work will constitute valuable input to the ongoing development of international standards for AV lamps, and thus help mature automated driving in general.

Understanding the neurophysiological mechanisms underlying curiosity and therefore being able to identify the curiosity level of a person, would provide useful information for researchers and designers in numerous fields such as neuroscience, psychology, and computer science. A first step to uncovering the neural correlates of curiosity is to collect neurophysiological signals during states of curiosity, in order to develop signal processing and machine learning (ML) tools to recognize the curious states from the non-curious ones. Thus, we ran an experiment in which we used electroencephalography (EEG) to measure the brain activity of participants as they were induced into states of curiosity, using trivia question and answer chains. We used two ML algorithms, i.e. Filter Bank Common Spatial Pattern (FBCSP) coupled with a Linear Discriminant Algorithm (LDA), as well as a Filter Bank Tangent Space Classifier (FBTSC), to classify the curious EEG signals from the non-curious ones. Global results indicate that both algorithms obtained better performances in the 3-to-5s time windows, suggesting an optimal time window length of 4 seconds (63.09% classification accuracy for the FBTSC, 60.93% classification accuracy for the FBCSP+LDA) to go towards curiosity states estimation based on EEG signals.

The abstract nature of programming makes learning to code a daunting undertaking for many novice learners. In this work, we advocate the use of comics-a medium capable of presenting abstract ideas in a concrete, familiar way-for introducing programming concepts. Particularly, we propose a design process and related tools to help students and teachers create coding strips, a form of comic strips that are associated with a piece of code. We conducted two design workshops with students and high school computer science teachers to evaluate our design process and tools. We find that our design process and tools are effective at supporting the design of coding strips and that both students and teachers are excited about using coding strip as a tool for learning and teaching programming concepts.

Curiosity-the intrinsic desire for new information-can enhance learning, memory, and exploration. Therefore, understanding how to elicit curiosity can inform the design of educational technologies. In this work, we investigate how a social peer robot's verbal expression of curiosity is perceived, whether it can affect the emotional feeling and behavioural expression of curiosity in students, and how it impacts learning. In a between-subjects experiment, 30 participants played the game LinkIt!, a game we designed for teaching rock classification, with a robot verbally expressing: curiosity, curiosity plus rationale, or no curiosity. Results indicate that participants could recognize the robot's curiosity and that curious robots produced both emotional and behavioural curiosity contagion effects in participants.

Expert disagreement is pervasive in clinical decision making and collective adjudication is a useful approach for resolving divergent assessments. Prior work shows that expert disagreement can arise due to diverse factors including expert background, the quality and presentation of data, and guideline clarity. In this work, we study how these factors predict initial discrepancies in the context of medical time series analysis, examining why certain disagreements persist after adjudication, and how adjudication impacts clinical decisions. Results from a case study with 36 experts and 4,543 adjudicated cases in a sleep stage classification task show that these factors contribute to both initial disagreement and resolvability, each in their own unique way. We provide evidence suggesting that structured adjudication can lead to significant revisions in treatment-relevant clinical parameters. Our work demonstrates how structured adjudication can support consensus and facilitate a deep understanding of expert disagreement in medical data analysis.

Crowdworkers regularly support their work with scripts, extensions, and software to enhance their productivity. Despite their evident significance, little is understood regarding how these tools affect crowdworkers' quality of life and work. In this study, we report findings from an interview study (N=21) aimed at exploring the tooling practices used by full-time crowdworkers on Amazon Mechanical Turk. Our interview data suggests that the tooling utilized by crowdworkers (1) strongly contributes to the fragmentation of microwork by enabling task switching and multitasking behavior; (2) promotes the fragmentation of crowdworkers' work-life boundaries by relying on tooling that encourages a 'work-anywhere' attitude; and (3) aids the fragmentation of social ties within worker communities through limited tooling access. Our findings have implications for building systems that unify crowdworkers' work practice in support of their productivity and well-being.

Collaborative crowdsourcing tasks allow crowd workers to solve problems that they could not handle alone, but worker motivation in these tasks is not well understood. In this paper, we study how to motivate groups of workers by paying them equitably. To this end, we characterize existing collaborative tasks based on the types of information available to crowd workers. Then, we apply concepts from equity theory to show how fair payments relate to worker motivation, and we propose two theoretically grounded classes of fair payments. Finally, we run two experiments using an audio transcription task on Amazon Mechanical Turk to understand how workers perceive these payments. Our results show that workers recognize fair and unfair payment divisions, but are biased toward payments that reward them more. Additionally, our data suggests that fair payments could lead to a small increase in worker effort. These results inform the design of future collaborative crowdsourcing tasks.

Listening to heart sounds is an important first step in evaluating the cardiovascular system and is important in the early detection of cardiovascular disease. We present and evaluate a framework for combining machine learning algorithms, crowd workers, and experts in the classification of heart sound recordings. The development of a hybrid human-machine framework is motivated by the past success in utilizing human computation to solve problems in medicine and the use of human-machine frameworks in other domains. We describe the methods that decide when and how to escalate the analysis of heart sounds to different resources and incorporate their decision into a final classification. Our framework was tested with a combination of machine classifiers and crowd workers from Amazon's Mechanical Turk. The results indicate a hybrid approach achieves greater performance than a baseline classifier alone, utilizing less expert resources while achieving similar performance, compared to a framework without the crowd.

Crowdsourced classification of data typically assumes that objects can be unambiguously classified into categories. In practice, many classification tasks are ambiguous due to various forms of disagreement. Prior work shows that exchanging verbal justifications can significantly improve answer accuracy over aggregation techniques. In this work, we study how worker deliberation affects resolvability and accuracy using case studies with both an objective and a subjective task. Results show that case resolvability depends on various factors, including the level and reasons for the initial disagreement, as well as the amount and quality of deliberation activities. Our work reinforces the finding that deliberation can increase answer accuracy and the importance of verbal discussion in this process. We contribute a new public data set on worker deliberation for text classification tasks, and discuss considerations for the design of deliberation workflows for classification.

Numerous crowdsourcing platforms are now available to support research as well as commercial goals. However, crowdsourcing is not yet widely adopted by researchers for generating, processing or analyzing research data. This study develops a deeper understanding of the circumstances under which crowdsourcing is a useful, feasible or desirable tool for research, as well as the factors that may influence researchers' decisions around adopting crowdsourcing technology. We conducted semi-structured interviews with 18 researchers in diverse disciplines, spanning the humanities and sciences, to illuminate how research norms and practitioners' dispositions were related to uncertainties around research processes, data, knowledge, delegation and quality. The paper concludes with a discussion of the design implications for future crowdsourcing systems to support research.

Consistency is a practical metric that evaluates an instrument's reliability based on its ability to yield the same output when repeatedly given a particular input. Despite its broad usage, little is understood about the feasibility of using consistency as a measure of worker reliability in crowdwork. In this paper, we explore the viability of measuring a worker's reliability by their ability to conform to themselves. We introduce and describe Deja Vu, a mechanism for dynamically generating task queues with consistency probes to measure the consistency of workers who repeat the same task twice. We present a study that utilizes Deja Vu to examine how generic characteristics of the duplicate task - such as placement, difficulty, and transformation - affect a worker’s task consistency in the context of two unique object detection tasks. Our findings provide insight into the design and use of consistency-based reliability metrics.

Service robots are becoming a widespread tool for assisting humans in scientific, industrial and even domestic settings. Yet, our understanding of how to motivate and sustain interactions between human users and robots remains limited. In this work, we conducted a study to investigate how surprising robot behaviour evokes curiosity and influences trust and engagement in the context of participants interacting with Recyclo, a service robot for providing recycling recommendations. In a Wizard-of-Oz experiment, 36 participants were asked to interact with Recyclo to recognize and sort a variety of objects, and were given object recognition responses that were either unsurprising or surprising. Results show that surprise gave rise to information seeking behavior indicative of curiosity, while having a positive influence on engagement and negative influence on trust.

Audio annotation is key to developing machine-listening systems; yet, effective ways to accurately and rapidly obtain crowdsourced audio annotations is understudied. In this work, we seek to quantify the reliability/redundancy trade-off in crowdsourced soundscape annotation, investigate how visualizations affect accuracy and efficiency, and characterize how performance varies as a function of audio characteristics. Using a controlled experiment, we varied sound visualizations and the complexity of soundscapes presented to human annotators. Results show that more complex audio scenes result in lower annotator agreement, and spectrogram visualizations are superior in producing higher quality annotations at lower cost of time and human labor. We also found recall is more affected than precision by soundscape complexity, and mistakes can be often attributed to certain sound event characteristics. These findings have implications not only for how we should design annotation tasks and interfaces for audio data, but also how we train and evaluate machine-listening systems.

Crowdsourcing systems are designed to elicit help from humans to accomplish tasks that are still difficult for computers. How to motivate workers to stay longer and/or perform better in crowdsourcing systems is a critical question for designers. Previous work have explored different motivational frameworks, both extrinsic and intrinsic. In this work, we examine the potential for curiosity as a new type of intrinsic motivational driver to incentivize crowd workers. We design crowdsourcing task interfaces that explicitly incorporate mechanisms to induce curiosity and conduct a set of experiments on Amazon's Mechanical Turk. Our experiment results show that curiosity interventions improve worker retention without degrading performance, and the magnitude of the effects are influenced by both personal characteristics of the worker and the nature of the task.

An important class of tasks that are underexplored in current human computation systems are complex tasks with global constraints. One example of such a task is itinerary planning, where solutions consist of a sequence of activities that meet requirements specified by the requester. In this paper, we focus on the crowdsourcing of such plans as a case study of constraint-based human computation tasks and introduce a collaborative planning system called Mobi that illustrates a novel crowdware paradigm. Mobi presents a single interface that enables crowd participants to view the current solution context and make appropriate contributions based on current needs. We conduct experiments that explain how Mobi enables a crowd to effectively and collaboratively resolve global constraints, and discuss how the design principles behind Mobi can more generally facilitate a crowd to tackle problems involving global constraints.

Behind every search query is a high-level mission that the user wants to accomplish. While current search engines can often provide relevant information in response to well-specified queries, they place the heavy burden of making a plan for achieving a mission on the user. We take the alternative approach of tackling users' high-level missions directly by introducing a human computation system that generates simple plans, by decomposing a mission into goals and retrieving search results tailored to each goal. Results show that our system is able to provide users with diverse, actionable search results and useful roadmaps for accomplishing their missions.

The emergence of human computation systems, including Mechanical Turk and games with a purpose, has made it feasible to distribute relevance judgment tasks to workers over the Web. Most human computation systems assign tasks to individuals randomly, and such assignments may match workers with tasks that they may be unqualified or unmotivated to perform. We compare two groups of workers, those given a choice of queries to judge versus those who are not, in terms of their self-rated competence and their actual performance. Results show that when given a choice of task, workers choose ones for which they have greater expertise, interests, confidence, and understanding.

Since its introduction at CHI 2004, the ESP Game has inspired many similar games that share the goal of gathering data from players. This paper introduces a new mechanism for collecting labeled data using "games with a purpose." In this mechanism, players are provided with either the same or a different object, and asked to describe that object to each other. Based on each other's descriptions, players must decide whether they have the same object or not. We explain why this new mechanism is superior for input data with certain characteristics, introduce an enjoyable new game called "TagATune" that collects tags for music clips via this mechanism, and present findings on the data that is collected by this game.