AI systems are increasingly being positioned to assist people in decision-making. However, recent empirical studies show critical concerns that people over-rely on AI advice without analytically engaging with it. While HCI research explores how people rely on AI advice, we argue that it largely overlooks an important aspect: replicating realistic decision-making scenarios. Human-AI interaction factors influence people’s reliance on AI advice. To understand human-AI interaction factors and their interplay, we conducted an analytical review of recent studies in human-AI reliance literature. We analyzed the decision-making tasks in research and their validity in application-grounded contexts. Our findings show that user engagement is a precious commodity for relying on AI advice; however, it comes at a cost. We also discuss factors contributing to “appropriate reliance”, existing research gaps, and recommendations for intervention design for human-AI reliance. Our work contributes to the critical body of research on building appropriate reliance on AI advice.

Users often over-rely on AI-assisted decisions without analytically engaging with them, even in practical domains.In this work, we explore persuading users to analytically engage with AI assistance to reduce their over-reliance using a complex business case of customer classification.We explore the effect of persuasive cognitive engagement through explanations and communicating system uncertainty to examine the behavior of participants having diverse expertise.We leverage their feedback and objective behavior to understand their perception of the AI performance.Our findings show a contrast in participants' subjective and objective behavior, indicating inappropriate reliance on AI assistance with the perception of system performance.However, we observe the positives of interactive cognitive engagement and identify further directions to get deeper insights into expert domains with personalized AI assistance and behavioral persuasion.

Crowdsourcing in China is a thriving industry. Among its most interesting structures, we find crowdfarms, in which crowdworkers self-organize as small organizations to tackle macrotasks. Little, however, is known as to which practices these crowdfarms use to tackle the macrotasks, and this goes hand in hand with the current practice of the HCI research community to treat all forms of complex crowdsourcing work as practically the same. However, macrotasks differ substantially regarding structure and decomposability. Treating them under one umbrella term - macrotasking - can lead to an imprecise understanding of the workforce involved. We address this gap by examining the work practices of 31 Chinese crowdfarms on the four main macrotask types, namely: modular, interlaced, wicked, and container macrotasks. Our results confirm essential differences in how these nascent crowd organizations address different macrotasks and shed light on what platforms can do to improve the uptake of such work.

Crowd-powered conversational systems (CPCS) are gaining considerable attention for their potential utility in a variety of application domains, for which automated conversational interfaces are still too limited. CPCS currently suffer from long response delays, which hampers their potential as conversational partners. The majority of prior work in this area has focused on demonstrating the feasibility of the approach and improving performance, while evaluation studies have primarily focused on response latency and ways to reduce it. Relatively little is currently known about how response delays in a CPCS can affect user experience. While the importance of reducing response latency is widely recognized in the broader field of human-computer interaction, little attention has been paid to how response quality, response delay, conversational context, and the complexity of the task affect how users experience the conversation, and how they perceive waiting for responses in particular. We conducted a between-subjects experiment (N = 478), to examine the influence of these four factors on the overall waiting experience of users. Results show that users 1) evaluated the waiting experience more negatively when the response delay was longer than 8 seconds, 2) underestimated the elapsed time but experienced more frustration in tasks with high complexity, 3) underestimated the elapsed time and experienced less frustration with high quality bot's utterances, 4) judged response delays to be slightly longer, and experienced more frustration in an emotion-centric CPCS compared to a task-centric CPCS. Our insights can inform the design of future CPCSs with regards to defining performance requirements and anticipating their potential impact on the user experience they can facilitate.

Crowdsourcing is a new value creation business model. Annual revenue of the Chinese market alone is hundreds of millions of dollars, yet few studies have focused on the practices of the Chinese crowdsourcing workforce, and those that do mainly focus on solo crowdworkers. We have extended our study of solo crowdworker practices to include crowdfarms, a relatively new entry to the gig economy: small companies that carry out crowdwork as a key part of their business. We report here on interviews of people who work in 53 crowdfarms. We describe how crowdfarms procure jobs, carry out macrotasks and microtasks, manage their reputation, and employ different management practices to motivate crowdworkers and customers.

Crowd-Powered Conversational Systems (CPCS) are gaining traction due to their potential utility in a range of application fields where automated conversational interfaces are still inadequate. Currently, long response times negatively impact CPCSs, limiting their potential application as conversational partners. Related research has focused on developing algorithms for swiftly hiring workers and synchronous crowd coordination techniques to ensure high-quality work. Evaluation studies typically concern system reaction times and performance measurements, but have so far not examined the effects of extended wait times on users. The goal of this study, based on time perception models, is to explore how effective different time fillers are at reducing the negative impacts of waiting in CPCSs. To this end, we conducted a rigorous simulation-based between-subjects (N = 930) study on the Prolific crowdsourcing platform to assess the influence of different filler types across three levels of delay (8, 16 & 32s) for Information Retrieval (IR) and stress management tasks. Our results show that asking users to perform secondary tasks (e.g., microtasks or breathing exercises) while waiting for longer periods of time helped divert their attention away from timekeeping, increased their engagement, and resulted in shorter perceived waiting times. For shorter delays, conversational fillers generated more intense immersion and contributed to shorten the perception of time.

Recent research highlights the potential of crowdsourcing in China. Yet very few studies explore the workplace context and experiences of Chinese crowdworkers. Those that do, focus mainly on the work experiences of solo crowdworkers but do not deal with issues pertaining to the substantial amount of people working in 'crowdfarms'. This article addresses this gap as one of its primary concerns. Drawing on a study that involves 48 participants, our research explores, compares and contrasts the work experiences of solo crowdworkers to those of crowdfarm workers. Our findings illustrate that the work experiences and context of the solo workers and crowdfarm workers are substantially different, with regards to their motivations, the ways they engage with crowdsourcing, the tasks they work on, and the crowdsourcing platforms they utilize. Overall, our study contributes to furthering the understandings on the work experiences of crowdworkers in China.

On-demand emotional support is an expensive and elusive societal need that is exacerbated in difficult times — as witnessed during the COVID-19 pandemic. Prior work in affective crowdsourcing has examined ways to overcome technical challenges for providing on-demand emotional support to end users. This can be achieved by training crowd workers to provide thoughtful and engaging on-demand emotional support. Inspired by recent advances in conversational user interface research, we investigate the efficacy of a conversational user interface for training workers to deliver psychological support to users in need. To this end, we conducted a between-subjects experimental study on Prolific, wherein a group of workers (N=200) received training on motivational interviewing via either a conversational interface or a conventional web interface. Our results indicate that training workers in a conversational interface yields both better worker performance and improves their user experience in on-demand stress management tasks.

The IoT industry supplies a plethora of Internet connected devices and services supporting smart home automation. However, end-users having little knowledge of the features and possibilities of such technologies, face difficulties in conjuring up useful application scenarios combining such devices and services, thus missing out on potential applications outside those provided by vendors. A remedy for such end-users can potentially be found in crowdsourcing IoT scenario creation. For such an enterprise to be viable it is essential to assess whether crowdsourcing can result in practical and original scenarios. This article reports two studies aiming to establish the practicality and originality of crowdsourced IoT scenarios for smart homes. In the first study, we recruited 102 crowd workers who created 306 scenarios in various categories. We then recruited a second cohort of 620 crowd workers to rate the scenarios’ creativity. In the second study, we evaluated the corpus of IoT scenarios by 20-experienced smart home users recruited through a screening survey. Our results show that the crowd evaluations of originality and creativity are strongly correlated with those of smart home users. Our major IoT-specific findings in relation to creativity are: a) The number of IoT devices and the number of combination of devices impact how creative the scenarios are perceived; b) Workers with self-reported intermediate programming knowledge wrote more creative scenarios when compared to workers having expert knowledge; c) Computational metrics such as text metrics can provide the basis for automated assessment of the scenarios’ creativity. Finally, an inductive thematic analysis of the scenarios revealed interesting themes (e.g., types of rules, automation styles and novel operators) which can serve as a guide for designing more expressive and intuitive end-user development solutions, in the context of IoT.

Despite the growing interest in crowdsourcing, this new labor model has recently received severe criticism. The most important point of this criticism is that crowdworkers are often underpaid and overworked. This severely affects job satisfaction and productivity. Although there is a growing body of evidence exploring the work experiences of crowdworkers in various countries, there have been a very limited number of studies to the best of our knowledge exploring the work experiences of Chinese crowdworkers. In this paper we aim to address this gap. Based on a framework of well-established approaches, namely the Job Demands-Resources model, the Work Design Questionnaire, the Oldenburg Burnout Inventory, the Utrecht Work Engagement Scale, and the Organizational Commitment Questionnaire, we systematically study the work experiences of 289 crowdworkers who work for ZBJ.com - the most popular Chinese crowdsourcing platform. Our study examines these crowdworker experiences along four dimensions: (1) crowdsourcing job demands, (2) job resources available to the workers, (3) crowdwork experiences, and (4) platform commitment. Our results indicate significant differences across the four dimensions based on crowdworkers' gender, education, income, job nature, and health condition. Further, they illustrate that different crowdworkers have different needs and threshold of demands and resources and that this plays a significant role in terms of moderating the crowdwork experience and platform commitment. Overall, our study sheds light to the work experiences of the Chinese crowdworkers and at the same time contributes to furthering understandings related to the work experiences of crowdworkers.

Expressions of territoriality have been positioned as one of the main reasons users alter their behaviors and perceptions of spatiality and sociality while engaging with location-based social networks (LBSN). Despite the potential for this interplay to further our understanding of LBSN usage in the context of identity, very little work has actually been done toward this. Addressing this gap in the literature is one of the chief aims of the article. Drawing on an original 6-week study with 42 participants utilizing a bespoke LBSN entitled “GeoMoments,” our research explores the following: (1) the way that territoriality is linked to self-identity; and (2) how this interplay affects the interactions between users as well as the environments they inhabit. Our findings suggest that participants affirmed their self-identity by selectively posting and claiming ownership of their neighborhood through the LBSN. Here, the locative decisions are made related to risk, hierarchies, and the users’ relationship to the area. This practice then led participants to discover and interact with the digital information overlaying their physical environments in a playful manner. These interactions demonstrate the perceived power structures that are facilitated by identity claims over a virtual area. In the main, our results reaffirm that territoriality is a central concept in understanding LBSN use, while also drawing attention to the temporality involved in user-to-user and user-to-place interactions pertaining to physical place mediated by LBSN.

New online stores and digital distribution methods have led to the development of alternative monetization models for video-games, such as free-to-play games with advertisements. Although there are many games using such models, until now the effect on the player experience from such interruptions has not been studied. In this controlled experiment, we requested that participants (N=236) play one of three different versions of a platformer game with: 1) no interruptions, 2) 30-second video advertisements, and 3) a multiple-choice questionnaire. We then evaluated the effects on the player experience. The study shows differences in their experiences, namely in: competence, immersion, annoyance, affects, and the reliability of the questionnaire answers. The contribution of this work is to identify which player experience variables are affected by interruptions, which can be valuable for selecting the business model and guiding the game design process.

Amongst the variety of (multi-modal) interaction techniques that are being developed and explored, the Motion Matching paradigm provides a novel approach to selection and control. In motion matching, users interact by rhythmically moving their bodies to track the continuous movements of different interface targets. This paper builds upon the current algorithmic and usability focused body of work by exploring the product possibilities and implications of motion matching. Through the development and qualitative study of four novel and different real-world motion matching applications --- with 20 participants --- we elaborate on the suitability of motion matching in different multi-user scenarios, the less pertinent use in home environments and the necessity for multi-modal interaction. Based on these learnings, we developed three novel motion matching based interactive lamps, which report on clear paths for further dissemination of the embodied interaction technique's experience. This paper hereby informs the design of future motion matching interfaces and products.

Trying to understand a player's characteristics with regards to a computer game is a major line of research known as player modeling. The purpose of player modeling is typically the adaptation of the game itself. We present two studies that extend player modeling into player profiling by trying to identify abstract personality traits, such as the need for cognition and self-esteem , through a player's in-game behavior. We present evidence that game mechanics that can be broadly adopted by several game genres, such as hints and a player's self-evaluation at the end of a level, correlate with the aforementioned personality traits. We conclude by presenting future directions for research regarding this topic, discuss the direct applications for the games industry, and explore how games can be developed as profiling tools with applications to other contexts.

Research studies and recruitment processes often rely on psychometric instruments to profile respondents with regards to their ethical orientation. Completing such questionnaires can be tedious and is prone to self-presentation bias. Noting how video games often expose players to complex plots, filled with dilemmas and morally dubious options, the opportunity emerges to evaluate player’s moral orientation by analysing their in-game behaviour. In order to explore the feasibility of such an approach, we examine how users’ moral judgment correlates with choices they make in non-linear narratives, frequently present in video games. An interactive narrative presenting several moral dilemmas was created. An initial user study (N = 80) revealed only weak correlations between the users’ choices and their ethical inclinations in all ethical scales. However, by training a genetic algorithm on this data set to quantify the influence of each branch on recognising moral inclination we found a strong positive correlation between choice behaviour and self-reported ethical inclinations on a second independent group of participants (N = 20). The contribution of this work is to demonstrate how genetic algorithms can be applied in interactive stories to profile users’ ethical stance.

Researchers who perform Ecological Momentary Assessment (EMA) studies tend to rely on informatics experts to set up and administer their data collection protocols with digital media. Contrary to standard surveys and questionnaires that are supported by widely available tools, setting up an EMA protocol is a substantial programming task. Apart from constructing the survey items themselves, researchers also need to design, implement, and test the timing and the contingencies by which these items are presented to respondents. Furthermore, given the wide availability of smartphones, it is becoming increasingly important to execute EMA studies on user-owned devices, which presents a number of software engineering challenges pertaining to connectivity, platform independence, persistent storage, and back-end control. We discuss TEMPEST, a web-based platform that is designed to support non-programmers in specifying and executing EMA studies. We discuss the conceptual model it presents to end-users, through an example of use, and its evaluation by 18 researchers who have put it to real-life use in 13 distinct research studies.

Self-esteem is a personality trait utilized to support the diagnosis of several psychological conditions. With this study we investigate the potential that computer games can have in assessing self-esteem. To that end, we designed and developed a platformer game and analyzed how in-game behavior relates to Rosenberg's Self-Esteem Scale. We examined: i) how a player's self-esteem influences game performance, ii) how a player's self-esteem generally influences in-game behavior iii) the possible game mechanics that assist in inferring a player's self-esteem. The study was conducted in two phases (N=98 and N=85). Results indicate that self-esteem does not have any impact on the player's performance, on the other hand, we found that players' self-evaluation of game performance correlates with their self-esteem.

This paper presents a prototype of a smart home control system operated through motion matching input. In motion matching, targets move continuously in a singular and pre-defined path; users interact with these targets by tracking their movement for a short period of time. Our prototype captures user input through the motion sensors embedded in off-the-shelf smartwatches while users track the moving targets with their arms and hands. The wearable nature of the tracking system makes our prototype ideal for interaction with numerous devices in a smart home.

Player behavior during game play can be used to construct player models that help adapt the game and make it more fun for the player involved. Similarly in-game behavior could help model personality traits that describe people's attitudes in a fashion that can be stable over time and over different domains, e.g., to support health coaching, or other behavior change approaches. This paper demonstrates the feasibility of this approach by relating Need for Cognition (NfC) a personality trait that can predict the effectiveness of different persuasion strategies upon users to a commonly used game mechanic -- hints. An experiment with N=188 participants confirmed our hypothesis that NfC has a negative correlation with the number of hints players follow during the game. Future work should confirm if adherence to hints can be used as a predictor of behavior in different games, and to find other game mechanics than hints, that help predict user traits.

User experience (UX) research on pervasive technologies faces considerable challenges regarding today's mobile context-sensitive applications: evaluative field studies lack control, whereas lab studies miss the interaction with a dynamic context. This dilemma has inspired researchers to use virtual environments (VEs) to acquire control while offering the user a rich contextual experience. Although promising, these studies are mainly concerned with usability and the technical realization of their setup. Furthermore, previous setups leave room for improvement regarding the user's immersive experience. This paper contributes to this line of research by presenting a UX case study on mobile advertising with a novel CAVE-smartphone interface. We conducted two experiments in which we evaluated the intrusiveness of a mobile location-based advertising app in a virtual supermarket. The results confirm our hypothesis that context-congruent ads lessen the experienced intrusiveness thereby demonstrating that our setup is capable of generating preliminary meaningful results with regards to UX. Furthermore, we share insights in conducting these studies.

This paper presents Reconexp, a diary method supported by a distributed application, which partly runs on a mobile device and partly on a website, enabling us to survey user attitudes, experiences and requirements in field studies. Reconexp combines aspects of the Experience Sampling Method and the Day Reconstruction Method aiming to reduce data loss, improve data quality and reduce burden put upon participants. We discuss our first experiences of using this method in the context of a study of communication needs of working parents with young children.

We are interested in systems that support awareness between individuals, by exchanging information that is automatically captured and presenting it to members of their social network. Here we demonstrate a principle for the operation of these systems which we describe as pervasive awareness: awareness information is aggregated opportunistically as mobile devices carrying some information migrate across space and cluster dynamically. We present a minimal demonstration of the principle where qualitative location information is used to select information offered by context capture devices (for the demonstration these are cameras).