As large language models (LLMs) are used in complex writing workflows, users engage in multi-turn interactions to steer generations to better fit their needs.Rather than passively accepting output, users actively refine, explore, and co-construct text.We conduct a largescale analysis of this collaborative behavior for users engaged in writing tasks in the wild with two popular AI assistants, Bing Copilot and WildChat.Our analysis goes beyond simple task classification or satisfaction estimation common in prior work and instead characterizes how users interact with LLMs through the course of a session.We identify prototypical behaviors in how users interact with LLMs in prompts following their original request.We refer to these as Prototypical Human-AI Collaboration Behaviors (PATHs) and find that a small group of PATHs explain a majority of the variation seen in user-LLM interaction.These PATHs span users revising intents, exploring texts, posing questions, adjusting style or injecting new content.Next, we find statistically significant correlations between specific writing intents and PATHs, revealing how users' intents shape their collaboration behaviors.We conclude by discussing the implications of our findings on LLM alignment.1

As processes around hybrid work, spatially distant collaborations, and work-life boundaries grow increasingly complex, managing workers' schedules for synchronous meetings has become a critical aspect of building successful global teams. However, gaps remain in our understanding of workers' scheduling preferences and practices, which we aim to fill in this large-scale, mixed-methods study of individuals' calendars in a multinational organization. Using interviews with eight participants, survey data from 165 respondents, and telemetry data from millions of meetings scheduled by 211 thousand workers, we characterize scheduling preferences, practices, and their relationship with each other and organizational factors. We find that temporal preferences can be broadly classified as either cyclical, such as suitability of certain days, or relational, such as dispersed meetings, at various time scales. Furthermore, our results suggest that these preferences are disconnected from actual practice--albeit with several notable exceptions--and that individual differences are associated with factors like meeting load, time-zones, importance of meetings to job function, and job titles. We discuss key themes for our findings, along with the implications for calendar and scheduling systems and socio-technical systems more broadly.

A critical factor in the success of many decision support systems is the accurate modeling of user preferences. Psychology research has demonstrated that users often develop their preferences during the elicitation process, highlighting the pivotal role of system-user interaction in developing personalized systems. This paper introduces a novel approach, combining Large Language Models (LLMs) with Constraint Programming to facilitate interactive decision support. We study this hybrid framework through the lens of meeting scheduling, a time-consuming daily activity faced by a multitude of information workers. We conduct three studies to evaluate the novel framework, including a diary study to characterize contextual scheduling preferences, a quantitative evaluation of the system’s performance, and a user study to elicit insights with a technology probe that encapsulates our framework. Our work highlights the potential for a hybrid LLM and optimization approach for iterative preference elicitation, and suggests design considerations for building systems that support human-system collaborative decision-making processes.

Organizations are becoming increasingly distributed and many need to collaborate synchronously over great geographical distances. Despite a rich body of literature on spatially-distanced meetings, gaps remain in our understanding of temporally-distanced meetings. Here, we characterize cross time zone collaborations by analyzing 20 million meetings scheduled at a multinational corporation, Microsoft, supported by a survey on how 130 employees perceive their scheduling needs. We find that cross time zone meetings are closely associated with scheduling patterns around early morning and late evening hours, which are challenging and discordant with employees’ stated temporal preferences. Additionally, the burdens of meeting across time boundaries are asymmetrically distributed among workers at different levels of the organization and different geolocations. Nonetheless, we further observe evidence that cross time zone attendees are organizationally distant and diverse, suggesting that addressing these challenges by limiting meetings would disafford employees the opportunities to connect. We conclude by sharing opportunities for facilitating cross time zone meetings that foster healthier global collaborations.

Motives or goals are recognized in psychology literature as the most fundamental drive that explains and predicts why people do what they do, including when they browse the web. Although providing enormous value, these higher-ordered goals are often unobserved, and little is known about how to leverage such goals to assist people’s browsing activities. This paper proposes to take a new approach to address this problem, which is fulfilled through a novel neural framework, Goal-directed Web Browsing (GoWeB). We adopt a psychologically-sound taxonomy of higher-ordered goals and learn to build their representations in a structure-preserving manner. Then we incorporate the resulting representations for enhancing the experiences of common activities people perform on the web. Experiments on large-scale data from Microsoft Edge web browser show that GoWeB significantly outperforms competitive baselines for in-session web page recommendation, re-visitation classification, and goal-based web page grouping. A follow-up analysis further characterizes how the variety of human motives can affect the difference observed in human behavioral patterns.

Despite the centrality of email in the daily routines of knowledge workers, fundamental aspects of its usage are still poorly understood. We are particularly interested in understanding one aspect of email management, email triage, the process of going through unhandled email and deciding what to do with them. In this paper we investigate the email triage behavior by presenting interview and survey results that characterize user behavior and needs. The results highlight current challenges and enhance our understanding of how the triage process can be more effectively supported.

Email triage involves going throughunhandled emails and deciding what to do with them. This familiar process can become increasingly challenging as the number of unhandled email grows. During a triage session, users commonlydefer handling emails that they cannot immediately deal with to later. These deferred emails, are often related to tasks that are postponed until the user has more time or the right information to deal with them. In this paper, through qualitative interviews and a large-scale log analysis, we study when and whatenterprise email users tend to defer. We found that users are more likely to defer emails when handling them involves replying, reading carefully, or clicking on links and attachments. We also learned that the decision to defer emails depends on many factors such as user's workload and the importance of the sender. Our qualitative results suggested that deferring is very common, and our quantitative log analysis confirms that 12% of triage sessions and 16% of daily active users had at least one deferred email on weekdays. We also discuss severaldeferral strategies such as marking emails as unread and flagging that are reported by our interviewees, and illustrate how such patterns can be also observed in user logs. Inspired by the characteristics of deferred emails and contextual factors involved in deciding if an email should be deferred, we train a classifier for predicting whether a recently triaged email is actually deferred. Our experimental results suggests that deferral can be classified with modest effectiveness. Overall, our work provides novel insights about how users handle their emails and how deferral can be modeled.

In information retrieval and information visualization, hierarchies are a common tool to structure information into topics or facets, and network visualizations such as knowledge graphs link related concepts within a domain. In this paper, we explore a multi-layer extension to knowledge graphs, hierarchical knowledge graphs (HKGs), that combines hierarchical and network visualizations into a unified data representation. Through interaction logs, we show that HKGs preserve the benefits of single-layer knowledge graphs at conveying domain knowledge while incorporating the sense-making advantages of hierarchies for knowledge seeking tasks. Specially, this paper describes our algorithm to construct these visualizations, analyzes interaction logs to quantitatively demonstrate performance parity with networks and performance advantages over hierarchies, and synthesizes data from interaction logs, inter- views, and thinkalouds on a testbed data set to demonstrate the utility of the unified hierarchy+network structure in our HKGs.

In exploratory search, how information is presented to the user and how the user interacts with the presented information heavily influence the user's success. In this paper, we examine two different spatial representations of search results: knowledge graphs and hierarchical trees. Through interaction logs we show that knowledge graphs can effectively reduce the need to read source content with no reduction in the quality of the information gathered by the user. Through qualitative interviews and thinkalounds we explore factors that influence user perception of different search results representations including biases, task, perceived structure of the data, and problem-solving approach. Overall, these results enhance our understanding of the role each of these representations can play in information seeking.