Information workers increasingly struggle with productivity challenges in modern workplaces, facing difficulties in managing time and effectively utilizing workplace analytics data for behavioral improvement. Despite the availability of productivity metrics through enterprise tools, workers often fail to translate this data into actionable insights. We present a comprehensive, user-centric approach to address these challenges through AI-based productivity agents tailored to users' needs. Utilizing a two-phase method, we first conducted a survey with 363 participants, exploring various aspects of productivity, communication style, agent approach, personality traits, personalization, and privacy. Drawing on the survey insights, we developed a GPT-4 powered personalized productivity agent that utilizes telemetry data gathered via Viva Insights from information workers to provide tailored assistance. We compared its performance with alternative productivity-assistive tools, such as dashboard and narrative, in a study involving 40 participants. Our findings highlight the importance of user-centric design, adaptability, and the balance between personalization and privacy in AI-assisted productivity tools. By building on these insights, our work provides important guidance for developing more effective productivity solutions, ultimately leading to optimized efficiency and user experiences for information workers.

The future of work is rapidly changing, with remote and hybrid settings blurring the boundaries between professional and personal life. To understand how work rhythms vary across different work settings, we conducted a month-long study of 65 software developers, collecting anonymized computer activity data as well as daily ratings for perceived stress, productivity, and work setting. In addition to confirming the double-peak pattern of activity at 10:00 am and 2:00 pm observed in prior research, we observed a significant third peak around 9:00 pm. This third peak was associated with higher perceived productivity during remote days but increased stress during onsite and hybrid days, highlighting a nuanced interplay between work demands and work settings. Additionally, we found strong correlations between computer activity, productivity, and stress, including an inverted U-shaped relationship where productivity peaked at around six hours of computer activity before declining on more active days. These findings provide new insights into evolving work rhythms and highlight the impact of different work settings on productivity and stress.

We present AImagery, an AI-powered immersive relaxation experience tailored to user preferences and physiological feedback. In a study with 32 participants, half of them experienced a multisensory experience with scent, biofeedback, and a personalized AI audio story based on their heart rate, self-reported mood and custom scenery. The experimental group showed a significant anxiety reduction for those with moderate to high anxiety, as opposed to the control group (non-guided meditation/baseline resting control condition). User feedback was positive and received high ratings for enjoyment, immersion and, to a lesser extend, sleepiness. Our results highlight the potential of multisensory AI-driven relaxation tools for those with elevated anxiety.

A critical factor in the success of many decision support systems is the accurate modeling of user preferences. Psychology research has demonstrated that users often develop their preferences during the elicitation process, highlighting the pivotal role of system-user interaction in developing personalized systems. This paper introduces a novel approach, combining Large Language Models (LLMs) with Constraint Programming to facilitate interactive decision support. We study this hybrid framework through the lens of meeting scheduling, a time-consuming daily activity faced by a multitude of information workers. We conduct three studies to evaluate the novel framework, including a diary study to characterize contextual scheduling preferences, a quantitative evaluation of the system’s performance, and a user study to elicit insights with a technology probe that encapsulates our framework. Our work highlights the potential for a hybrid LLM and optimization approach for iterative preference elicitation, and suggests design considerations for building systems that support human-system collaborative decision-making processes.

Workplace stress has been increasing in recent decades and has worsened by the unique demands imposed by COVID-19 and the new remote/hybrid work settings. High-stress working conditions can be detrimental to the health and wellness of workers and can lead to significant business costs in terms of productivity loss and medical expenses. An essential step toward managing stress involves finding comfortable ways to sense workers and recognizing stress as soon as it happens. This work explores the potential value of using pervasive sensors such as keyboards, webcams, and behavioral data such as calendar and e-mail activity to passively assess individual stress levels of work in real-life. In particular, we collected a large corpus of such data from 46 remote information workers over one month and asked them to self-report their stress levels and other relevant factors several times a day. Analysis of the data demonstrates that passive sensors can effectively detect both triggers and manifestations of workplace stress and that having access to prior data of the worker is critical for developing well-performing stress recognition models. Furthermore, we provide qualitative feedback capturing workers' preferences in workplace stress monitoring.

Workplace stress-reduction interventions have produced mixed results due to engagement and adherence barriers. Leveraging technology to integrate such interventions into the workday may address these barriers and help mitigate the mental, physical, and monetary effects of workplace stress. To inform the design of a workplace stress-reduction intervention system, we conducted a four-week longitudinal study with 86 participants, examining the effects of intervention type and timing on usage, stress reduction impact, and user preferences. We compared three intervention types and two delivery timing conditions: Pre-scheduled (PS) by users and Just-in-time (JIT) prompted by the system-identified user stress-levels. We found JIT participants completed significantly more interventions than PS participants, but post-intervention and study-long stress reduction was not significantly different between conditions. Participants rated low-effort interventions highest, but high-effort interventions reduced the most stress. Participants felt JIT provided accountability but desired partial agency over timing. We present type and timing implications.

The ability to monitor audience reactions is critical when delivering presentations. However, current videoconferencing platforms offer limited solutions to support this. This work leverages recent advances in affect sensing to capture and facilitate communication of relevant audience signals. Using an exploratory survey (N=175), we assessed the most relevant audience responses such as confusion, engagement, and head-nods. We then implemented AffectiveSpotlight, a Microsoft Teams bot that analyzes facial responses and head gestures of audience members and dynamically spotlights the most expressive ones. In a within-subjects study with 14 groups (N=117), we observed that the system made presenters significantly more aware of their audience, speak for a longer period of time, and self-assess the quality of their talk more similarly to the audience members, compared to two control conditions (randomly-selected spotlight and default platform UI). We provide design recommendations for future affective interfaces for online presentations based on feedback from the study.

Video-conferencing is essential for many companies, but its limitations in conveying social cues can lead to ineffective meetings. We present MeetingCoach, an intelligent post-meeting feedback dashboard that summarizes contextual and behavioral meeting information. Through an exploratory survey (N=120), we identified important signals (e.g., turn taking, sentiment) and used these insights to create a wireframe dashboard. The design was evaluated with in situ participants (N=16) who helped identify the components they would prefer in a post-meeting dashboard. After recording video-conferencing meetings of eight teams over four weeks, we developed an AI system to quantify the meeting features and created personalized dashboards for each participant. Through interviews and surveys (N=23), we found that reviewing the dashboard helped improve attendees’ awareness of meeting dynamics, with implications for improved effectiveness and inclusivity. Based on our findings, we provide suggestions for future feedback system designs of video-conferencing meetings.

Traces of searcher behaviour, such as query reformulation or clicks, are commonly used to evaluate a running search engine. The underlying expectation is that these behaviours are proxies for something more important, such as relevance, utility, or satisfaction. Affective computing technology gives us the tools to help confirm some of these expectations, by examining visceral expressive responses during search sessions. However, work to date has only studied small populations in laboratory settings and with a limited number of contrived search tasks. In this study, we analysed longitudinal, in-situ, search behaviours of 152 information workers, over the course of several weeks while simultaneously tracking their facial expressions. Results from over 20,000 search sessions and 45,000 queries allow us to observe that indeed affective expressions are consistent with, and complementary to, existing "click-based'' metrics. On a query-level, searches that result in a short dwell time are associated with a decrease in smiles (expressions of "happiness'') and that if a query is reformulated the results of the reformulation are associated with an increase in smiling---suggesting a positive outcome as people converge on the information they need. On a session-level, sessions that feature reformulations are more commonly associated with fewer smiles and more furrowed brows (expressions of "anger/frustration''). Similarly, sessions with short-dwell clicks are also associated with fewer smiles. These data provide an insight into visceral aspects of search experience and present a new dimension for evaluating engine performance.

The ability to regulate our emotions plays an important role in our psychological and physical health. Regulating emotions influences how and when emotions are expressed. We performed a large scale, longitudinal observational study to investigate the effect of emotion regulation ability on expressed affect. We found that expression of negative affect increased throughout the day. For people who suppress emotion this increase is slower that for those who do not. For those with stronger cognitive reappraisal abilities, though not significant, there was a trend for higher positive affect and negative affect increased significantly less steeply, suggesting that they might experience more positive and less negative affect. These results reflect some of the first results based on large scale, continuous tracking of behavioral expression of emotion longitudinally. Our results demonstrate the need to carefully consider the time of day and emotion regulation ability, in addition to gender and age, when attempting to automatically infer affective states for facial behavior.

Through an iterative design process using Wizard of Oz (WOz) prototypes, we designed a video calling application for people with Autism Spectrum Disorder. Our Video Calling for Autism prototype provided an Expressiveness Mirror that gave feedback to autistic people on how their facial expressions might be interpreted by their neurotypical conversation partners. This feedback was in the form of emojis representing six emotions and a bar indicating the amount of overall expressiveness demonstrated by the user. However, when we built a working prototype and conducted a user study with autistic participants, their negative feedback caused us to reconsider how our design process led to a prototype that they did not find useful. We reflect on the design challenges around developing AI technology for an autistic user population, how Wizard of Oz prototypes can be overly optimistic in representing AI-driven prototypes, how autistic research participants can respond differently to user experience prototypes of varying fidelity, and how designing for people with diverse abilities needs to include that population in the development process.

Current research on building intelligent agents for aiding with productivity and focus in the workplace is quite limited, despite the ubiquity of information workers across the globe. In our work, we present a productivity agent which helps users schedule and block out time on their calendar to focus on important tasks, monitor and intervene with distractions, and reflect on their daily mood and goals in a single, standalone application. We created two different prototype versions of our agent: a text-based (TB) agent with a similar UI to a standard chatbot, and a more emotionally expressive virtual agent (VA) that employs a video avatar and the ability to detect and respond appropriately to users' emotions. We evaluated these two agent prototypes against an existing product (control) condition through a three-week, within subjects study design with 40 participants, across different work roles in a large organization. We found that participants scheduled 134% more time with the TB prototype, and 110% more time with the VA prototype for focused tasks compared to the control condition. Users reported that they felt more satisfied and productive with the VA agent. However, The perception of anthropomorphism in the VA was polarized, with several participants suggesting that the human appearance was unnecessary. We discuss important insights from our work for the future design of conversational agents for productivity, wellbeing, and focus in the workplace.

The delivery of mental health interventions via ubiquitous devices has shown much promise. A conversational chatbot is a promising oracle for delivering appropriate just-in-time interventions. However, designing emotionally-aware agents, specially in this context, is under-explored. Furthermore, the feasibility of automating the delivery of just-in-time mHealth interventions via such an agent has not been fully studied. In this paper, we present the design and evaluation of EMMA (EMotion-Aware mHealth Agent) through a two-week long human-subject experiment with N=39 participants. EMMA provides emotionally appropriate micro-activities in an empathetic manner. We show that the system can be extended to detect a user's mood purely from smartphone sensor data. Our results show that our personalized machine learning model was perceived as likable via self-reports of emotion from users. Finally, we provide a set of guidelines for the design of emotion-aware bots for mHealth.

A natural conversational interface that allows longitudinal symptom tracking would be extremely valuable in health/wellness applications. However, the task of designing emotionally-aware agents for behavior change is still poorly understood. In this paper, we present the design and evaluation of an emotion-aware chatbot that conducts experience sampling in an empathetic manner. We evaluate it through a human-subject experiment with N=39 participants over the course of a week. Our results show that extraverts preferred the emotion-aware chatbot significantly more than introverts. Also, participants reported a higher percentage of positive mood reports when interacting with the empathetic bot. Finally, we provide guidelines for the design of emotion-aware chatbots for potential use in mHealth contexts.

Conversational agents have the potential to support users in many tasks. However, support for productivity and well-being in the workplace has received little attention. We present the first design of a conversational system that supports information workers with multiple work-related goals, informed by a survey of the current and potential use of conversational agents in the workplace. The goals of this research include the evaluation of using an agent for scheduling and prioritizing tasks, switching tasks, providing break reminders, dealing with social media distractions and for end of the day reflection on tasks accomplished. We deployed a chat-based intelligent agent, named Amber, in a field study with 24 information workers over the course of 6 days. We present our preliminary findings from the field study and discuss implications for the design of future workplace conversational agents.

Nonvisually Accessible Video Calling (NAVC) is a prototype that detects visual conversation cues in a video call and uses audio cues to convey them to a user who is blind or low-vision. NAVC uses audio cues inspired by movie soundtracks to convey Attention, Agreement, Disagreement, Happiness, Thinking, and Surprise. When designing NAVC, we partnered with people who are blind or low-vision through a user-centered design process that included need-finding interviews and design reviews. To evaluate NAVC, we conducted a user study with 16 participants. The study provided feedback on the NAVC prototype and showed that the participants could easily discern some cues, like Attention and Agreement, but had trouble distinguishing others. The accuracy of the prototype in detecting conversation cues emerged as a key concern, especially in avoiding false positives and in detecting negative emotions, which tend to be masked in social conversations. This research identified challenges and design opportunities in using AI models to enable accessible video calling.

Mental health disorders are a leading cause of disability worldwide. Although evidence-based psychotherapy is effective, engagement from such programs can be low. Mobile apps have the potential to help engage and support people in their therapy. We developed Pocket Skills, a mobile web app based on Dialectical Behavior Therapy (DBT). Pocket Skills teaches DBT via a conversational agent modeled on Marsha Linehan, who developed DBT. We examined the feasibility of Pocket Skills in a 4-week field study with 73 individuals enrolled in psychotherapy. After the study, participants reported decreased depression and anxiety and increased DBT skills use. We present a model based on qualitative findings of how Pocket Skills supported DBT. Pocket Skills helped participants engage in their DBT and practice and implement skills in their environmental context, which enabled them to see the results of using their DBT skills and increase their self-efficacy. We discuss the design implications of these findings for future mobile mental health systems.

The development of strong social and emotional skills is central to personal wellbeing. Increasingly, these skills are being taught in schools through well researched curricula. Such social-emotional learning (SEL) curricula are most effective if reinforced by parents, thus transferring the skills into everyday contexts. Traditional SEL programs have however had limited success in engaging parents, and we argue that technology might be able to help bridge this school-home divide. Through interviews with SEL experts we identified central design considerations for technology and SEL content: the reliance on experiential learning and the need to scaffold the parents in scaffolding the interaction for their children. This informed the design of a technology probe comprising a magnet card and online SEL activities, deployed in a school and via Mturk. The results provide a nuanced understanding of how technology-based interventions could bridge the school-home gap in real-world settings and support at-home reinforcement of children's social-emotional skills.

Behavior modification in health is difficult, as habitual behaviors are extremely well-learned, by definition. This research is focused on building a persuasive system for behavior modification around emotional eating. In this paper, we make strides towards building a just-in-time support system for emotional eating in three user studies. The first two studies involved participants using a custom mobile phone application for tracking emotions, food, and receiving interventions. We found lots of individual differences in emotional eating behaviors and that most participants wanted personalized interventions, rather than a pre-determined intervention. Finally, we also designed a novel, wearable sensor system for detecting emotions using a machine learning approach. This system consisted of physiological sensors which were placed into women's brassieres. We tested the sensing system and found positive results for emotion detection in this mobile, wearable system.

At ICSE 2010, the Code Bubbles team from Brown University and the Code Canvas team from Microsoft Research presented similar ideas for new user experiences for an integrated development environment. Since then, the two teams formed a collaboration, along with the Microsoft Visual Studio team, to release Debugger Canvas, an industrial version of the Code Bubbles paradigm. With Debugger Canvas, a programmer debugs her code as a collection of code bubbles, annotated with call paths and variable values, on a two-dimensional pan-and-zoom surface. In this experience report, we describe new user interface ideas, describe the rationale behind our design choices, evaluate the performance overhead of the new design, and provide user feedback based on lab participants, post-release usage data, and a user survey and interviews. We conclude that the code bubbles paradigm does scale to existing customer code bases, is best implemented as a mode in the existing user experience rather than a replacement, and is most useful when the user has a long or complex call paths, a large or unfamiliar code base, or complex control patterns, like factories or dynamic linking.

The user interfaces of today's development environments have a "bento box" design that partitions information into separate areas. This design makes it difficult to stay oriented in the open documents and to synthesize information shown in different areas. Code Canvas takes a new approach by providing an infinite zoomable surface for software development. A canvas both houses editable forms of all of a project's documents and allows multiple layers of visualization over those documents. By uniting the content of a project and information about it onto a single surface, Code Canvas is designed to leverage spatial memory to keep developers oriented and to make it easy to synthesize information.