Surgery continues to be perceived as an art, where proficiency is primarily achieved through years of experience. Artificial Intelligence research has yielded insight into the performance of expert surgeons and their associations with patient outcomes. Clinician expertise has led to the development of systematic assessments for fundamental skills (e.g., End-to-End Assessment of Suturing Expertise [EASE]) that contribute to positive outcomes. However, evaluating these skills requires manual expert review of endoscopic videos and is prone to inconsistencies between human raters. In this work, we present AutoEASE, the first end-to-end pipeline to automatically assess suturing performance from raw endoscopic video data using EASE rubrics. Our system utilizes a Mixture of Expert models (MoE) ; Multiscale vision transformers and 3D convolutional neural networks trained on Robot-assisted Radical Prostatectomy videos with over 13000 data points. For a given stitch clip, the MoE pipeline first determines each phase (needle handling, driving, withdrawal) of a continuous stitch and predicts a binary score (fail / ideal) for seven sub-skills based on rubrics defined in EASE. AutoEASE achieves 0.98 AUC while detecting each stitch phase. For EASE score prediction, the complete end-to-end pipeline attains ≥ 0.77 AUC in sub-skills associated with needle handling and driving. The promising performance of AutoEASE at the individual stitch level demonstrates the feasibility of developing more sophisticated assessment and reporting tools for complete surgical procedures objectively and at scale.

Online competitive action games have flourished as a space for entertainment and social connections, yet they face challenges from a small percentage of players engaging in disruptive behaviors. This study delves into the under-explored realm of understanding the effects of moderation on player behavior within online competitive action games on an example of a popular title - Call of Duty®:Modern Warfare®II. We employ a quasi-experimental design and causal inference techniques to examine the impact of moderation in a real-world industry-scale moderation system. We further delve into novel aspects around the impact of delayed moderation, as well as the severity of applied punishment. We examine these effects on a set of four disruptive behaviors including cheating, offensive username, chat, and voice. Our findings uncover the dual impact moderation has on reducing disruptive behavior and discouraging disruptive players from participating. We further uncover differences in the effectiveness of quick and delayed moderation and the varying severity of punishment. Our examination of real-world gaming interactions sets a precedent in understanding the effectiveness of moderation and its impact on player behavior. Our insights offer actionable suggestions for the most promising avenues for improving real-world moderation practices, as well as the heterogeneous impact moderation has on different players.

The ever-increasing pace of scientific publication necessitates methods for quickly identifying relevant papers. While neural recommenders trained on user interests can help, they still result in long, monotonous lists of suggested papers. To improve the discovery experience we introduce multiple new methods for augmenting recommendations with textual relevance messages that highlight knowledge-graph connections between recommended papers and a user’s publication and interaction history. We explore associations mediated by author entities and those using citations alone. In a large-scale, real-world study, we show how our approach significantly increases engagement—and future engagement when mediated by authors—without introducing bias towards highly-cited authors. To expand message coverage for users with less publication or interaction history, we develop a novel method that highlights connections with proxy authors of interest to users and evaluate it in a controlled lab study. Finally, we synthesize design implications for future graph-based messages.

Conversational agents (CAs) are systems that interact with humans through natural language user interfaces. They include systems with a range of conversational capabilities and modalities. For example, there are text- or voice-only question-answering interactions such as Apple Siri, Google Assistant, and Amazon Alexa, and there are also multimodal conversational AI agents that can engage users in long-term dialogues. Advances in speech recognition, natural language processing, and computer vision have resulted in a greater acceptance and use of CAs. CAs have already started to play important roles in various healthcare settings, including assisting clinicians during consultations, assisting consumers in changing health behaviours, and helping patients such as the elderly in their living environments. \n \nThere have been several systematic reviews on the use of CAs in health and wellbeing recently. Although the field still appears to be nascent, the emerging evidence has shown user acceptance of CAs in the healthcare domain as well as the early promises in boosting healthcare outcomes in both physical and mental health. Despite the increasing adoption and the benefits of using CAs to support health and wellbeing, the review studies also revealed (i) patient safety was rarely examined, (ii) health outcomes were inadequately measured, and (iii) no standardised evaluation methods were employed. There were also limitations in reporting the technical implementation details of CAs used, making the replicability of prior studies problematic. \n \nIn addition to addressing some of the current challenges and limitations, this special issue features cutting-edge research on designing, developing, and evaluating CAs for health and wellbeing that aim to improve health outcomes and services, and satisfy unique application needs (e.g., safety, trust, and user experience). The seven articles included in this special issue cover many application areas ranging from mental health and social support to information seeking to coaching. Amongst the accepted articles, mental health and social support themes represented the primary research foci. The articles also covered different population groups including older adults, young adults, and homeless people. Based on their foci, we have grouped the articles in this special issue by three areas: mental health, older adult wellbeing, and social support and coaching.

AI technologies have been incorporated into many end-user applications. However, expectations of the capabilities of such systems vary among people. Furthermore, bloated expectations have been identified as negatively affecting perception and acceptance of such systems. Although the intelligibility of ML algorithms has been well studied, there has been little work on methods for setting appropriate expectations before the initial use of an AI-based system. In this work, we use a Scheduling Assistant - an AI system for automated meeting request detection in free-text email - to study the impact of several methods of expectation setting. We explore two versions of this system with the same 50% level of accuracy of the AI component but each designed with a different focus on the types of errors to avoid (avoiding False Positives vs. False Negatives). We show that such different focus can lead to vastly different subjective perceptions of accuracy and acceptance. Further, we design expectation adjustment techniques that prepare users for AI imperfections and result in a significant increase in acceptance.

Conversational agents stand to play an important role in supporting behavior change and well-being in many domains. With users able to interact with conversational agents through both text and voice, understanding how designing for these channels supports behavior change is important. To begin answering this question, we designed a conversational agent for the workplace that supports workers' activity journaling and self-learning through reflection. Our agent, named Robota, combines chat-based communication as a Slack Bot and voice interaction through a personal device using a custom Amazon Alexa Skill. Through a 3-week controlled deployment, we examine how voice-based and chat-based interaction affect workers' reflection and support self-learning. We demonstrate that, while many current technical limitations exist, adding dedicated mobile voice interaction separate from the already busy chat modality may further enable users to step back and reflect on their work. We conclude with discussion of the implications of our findings to design of workplace self-tracking systems specifically and to behavior-change systems in general.

Effective communication of activities and progress in the workplace is crucial for the success of many modern organizations. In this paper, we extend current research on workplace communication and uncover opportunities for technology to support effective work activity reporting. We report on three studies: With a survey of 68 knowledge workers followed by 14 in-depth interviews, we investigated the perceived benefits of different types of progress reports and an array of challenges at three stages: Collection, Composition, and Delivery. We show an important interplay between written and face-to-face reporting, and highlight the importance of tailoring a report to its audience. We then present results from an analysis of 722 reports composed by 361 U.S.-based knowledge workers, looking at the influence of the audience on a report's language. We conclude by discussing opportunities for future technologies to assist both employees and managers in collecting, interpreting, and reporting progress in the workplace.

Despite much recent progress in conversational systems, the vision of a truly "intelligent" agent is still far from being realized. Part of the reason is that current applications focus on offering conversational alternatives to GUI supported tasks. In my research I am focusing on application of dialogue-based interaction in area of self-learning in health behavior change, domain in which conversation can offer unique value. Yet, to be effective in this domain a conversational system needs to be "intelligent". In my research I define aspects of intelligence crucial for supporting long-term self-learning in behavior change through conversation and address the technical as well as design challenges of enabling such intelligent applications.

Donation-based support for open, peer production projects such as Wikipedia is an important mechanism for preserving their integrity and independence. For this reason understanding donation behavior and incentives is crucial in this context. In this work, using a dataset of aggregated donation information from Wikimedia's 2015 fund-raising campaign, representing nearly 1 million pages from English and French language versions of Wikipedia, we explore the relationship between the properties of contents of a page and the number of donations on this page. Our results suggest the existence of a reciprocity mechanism, meaning that articles that provide more utility value attract a higher rate of donation. We discuss these and other findings focusing on the impact they may have on the design of banner-based fundraising campaigns. Our findings shed more light on the mechanisms that lead people to donate to Wikipedia and the relation between properties of contents and donations.

Machine learning (ML) has become increasingly influential to human society, yet the primary advancements and applications of ML are driven by research in only a few computational disciplines. Even applications that affect or analyze human behaviors and social structures are often developed with limited input from experts outside of computational fields. Social scientists—experts trained to examine and explain the complexity of human behavior and interactions in the world—have considerable expertise to contribute to the development of ML applications for human-generated data, and their analytic practices could benefit from more human-centered ML methods. Although a few researchers have highlighted some gaps between ML and social sciences [51, 57, 70], most discussions only focus on quantitative methods. Yet many social science disciplines rely heavily on qualitative methods to distill patterns that are challenging to discover through quantitative data. One common analysis method for qualitative data is qualitative coding . In this article, we highlight three challenges of applying ML to qualitative coding. Additionally, we utilize our experience of designing a visual analytics tool for collaborative qualitative coding to demonstrate the potential in using ML to support qualitative coding by shifting the focus to identifying ambiguity. We illustrate dimensions of ambiguity and discuss the relationship between disagreement and ambiguity. Finally, we propose three research directions to ground ML applications for social science as part of the progression toward human-centered machine learning.

Qualitative coding offers the potential to obtain deep insights into social media, but the technique can be inconsistent and hard to scale. Researchers using qualitative coding impose structure on unstructured data through “codes” that represent categories for analysis. Our visual analytics interface, Aeonium, supports human insight in collaborative coding through visual overviews of codes assigned by multiple researchers and distributions of important keywords and codes. The underlying machine learning model highlights ambiguity and inconsistency. Our goal was not to reduce qualitative coding to a machine-solvable problem, but rather to bolster human understanding gained from coding and reinterpreting the data collaboratively. We conducted an experimental study with 39 participants who coded tweets using our interface. In addition to increased understanding of the topic, participants reported that Aeonium's collaborative coding functionality helped them reflect on their own interpretations. Feedback from participants demonstrates that visual analytics can help facilitate rich qualitative analysis and suggests design implications for future exploration.

A distance cartogram (DC) is a technique that alters distances between a user-specified origin and the other locations in a map with respect to travel time. With DC, users can weigh the relative travel time costs between the origin and potential destinations at a glance because travel times are projected in a linearly interpolated time space from the origin. Such glance-ability is known to be useful for travelers who are mindful of travel time when finding their travel destinations. When constructing DC, however, uneven urban traffic conditions introduce excessive distortion and challenge user intuition. In addition, there has been little research focusing on DC's user interaction design. To tackle these challenges and realize the potential of DC as an interactive decision-making support tool, we derive a set of useful interactions through two formative studies and devise two novel techniques called Geo-contextual Anchoring Projection and Scalable Road-network Construction. We develop an interactive map system using these techniques and evaluate this system by comparing it against an equidistant map (EM), a widely used conventional layout that preserves the geographical reality. Based on the analysis of user behavior and qualitative feedback, we identify several benefits of using DC itself and of the interaction techniques we derived. We also analyze the specific reasons behind these identified benefits.

Although we may complain about meetings, they are an essential part of an information worker's work life. Consequently, busy people spend a significant amount of time scheduling meetings. We present Calendar.help, a system that provides fast, efficient scheduling through structured workflows. Users interact with the system via email, delegating their scheduling needs to the system as if it were a human personal assistant. Common scheduling scenarios are broken down using well-defined workflows and completed as a series of microtasks that are automated when possible and executed by a human otherwise. Unusual scenarios fall back to a trained human assistant executing an unstructured macrotask. We describe the iterative approach we used to develop Calendar.help, and share the lessons learned from scheduling thousands of meetings during a year of real-world deployments. Our findings provide insight into how complex information tasks can be broken down into repeatable components that can be executed efficiently to improve productivity.

Social systems and applications often rely on message triggers to promote, remind and even persuade people to perform certain actions. However, repeated exposure to these triggers can lead to boredom, annoyance and decreased engagement. While existing research suggests that diversification of trigger contents may mitigate these issues, no systematic way of introducing it has been proposed. This paper proposes two message diversification strategies based on the use of cognitive spaces: 1) target-diverse -- using concepts cognitively close to the targeted action; and 2) self-diverse -- using concepts cognitively close to the message's recipient. Through a controlled experiment we found that the self-diverse strategy reduces annoyance and boredom from repeated exposure and that both strategies increase perceived informativeness and helpfulness of the triggers. In a subsequent 2-week long field deployment focused on assessing the effects of the self-diverse strategy, we found that this strategy results in higher activity completion through supporting awareness, providing more information, and making the triggers more personally relevant. These diverse triggers are perceived as motivators rather than simple reminders. We conclude with insights on how to design and generate diverse messages.

Designing effective incentives is a challenge across many social computing contexts, from attracting crowdworkers to sustaining online contributions. However, one aspect of incentivizing that has been understudied is its impact on participation bias, as different incentives may attract different subsets of the population to participate. In this paper, we present two empirical studies in the crowdworking context that show that the incentive offered influence who participates in the task. Using the Basic Human Values, we found that a lottery reward attracted participants who held stronger openness-to-change values while a charity reward attracted those with stronger self-transcendence orientation. Further, we found that participation self-selection resulted in differences in the task outcomes. Through attracting more self-directed individuals, the lottery reward resulted in more ideas generated in a brainstorming task. Design implications include utilizing rewards to target desired participants and using diverse incentives to improve participation diversity.

Due to the growing pace of life, stress became one of the major factors causing health problems. We have developed a framework for measuring stress in real-life conditions continuously and unobtrusively. In order to provide meaningful, useful and actionable information, we present stress information, derived from sensor measurements, in the context of person's activities. In this paper, we describe our framework, discuss how we address arising challenges and evaluate our approach on basis of the field studies we have conducted. The main results of the evaluation are that the results of long-term measurements of stress reveal people information about their behavioral patterns that they perceive as meaningful and useful, and trigger their ideas about behavioral changes necessary to achieve a better stress balance.