As processes around hybrid work, spatially distant collaborations, and work-life boundaries grow increasingly complex, managing workers' schedules for synchronous meetings has become a critical aspect of building successful global teams. However, gaps remain in our understanding of workers' scheduling preferences and practices, which we aim to fill in this large-scale, mixed-methods study of individuals' calendars in a multinational organization. Using interviews with eight participants, survey data from 165 respondents, and telemetry data from millions of meetings scheduled by 211 thousand workers, we characterize scheduling preferences, practices, and their relationship with each other and organizational factors. We find that temporal preferences can be broadly classified as either cyclical, such as suitability of certain days, or relational, such as dispersed meetings, at various time scales. Furthermore, our results suggest that these preferences are disconnected from actual practice--albeit with several notable exceptions--and that individual differences are associated with factors like meeting load, time-zones, importance of meetings to job function, and job titles. We discuss key themes for our findings, along with the implications for calendar and scheduling systems and socio-technical systems more broadly.

A critical factor in the success of many decision support systems is the accurate modeling of user preferences. Psychology research has demonstrated that users often develop their preferences during the elicitation process, highlighting the pivotal role of system-user interaction in developing personalized systems. This paper introduces a novel approach, combining Large Language Models (LLMs) with Constraint Programming to facilitate interactive decision support. We study this hybrid framework through the lens of meeting scheduling, a time-consuming daily activity faced by a multitude of information workers. We conduct three studies to evaluate the novel framework, including a diary study to characterize contextual scheduling preferences, a quantitative evaluation of the system’s performance, and a user study to elicit insights with a technology probe that encapsulates our framework. Our work highlights the potential for a hybrid LLM and optimization approach for iterative preference elicitation, and suggests design considerations for building systems that support human-system collaborative decision-making processes.

Organizations are becoming increasingly distributed and many need to collaborate synchronously over great geographical distances. Despite a rich body of literature on spatially-distanced meetings, gaps remain in our understanding of temporally-distanced meetings. Here, we characterize cross time zone collaborations by analyzing 20 million meetings scheduled at a multinational corporation, Microsoft, supported by a survey on how 130 employees perceive their scheduling needs. We find that cross time zone meetings are closely associated with scheduling patterns around early morning and late evening hours, which are challenging and discordant with employees’ stated temporal preferences. Additionally, the burdens of meeting across time boundaries are asymmetrically distributed among workers at different levels of the organization and different geolocations. Nonetheless, we further observe evidence that cross time zone attendees are organizationally distant and diverse, suggesting that addressing these challenges by limiting meetings would disafford employees the opportunities to connect. We conclude by sharing opportunities for facilitating cross time zone meetings that foster healthier global collaborations.

The public is increasingly concerned about the practices of large technology companies with regards to privacy and many other issues. To force changes in these practices, there have been growing calls for “data strikes.” These new types of collective action would seek to create leverage for the public by starving business-critical models (e.g. recommender systems, ranking algorithms) of much-needed training data. However, little is known about how data strikes would work, let alone how effective they would be. Focusing on the important commercial domain of recommender systems, we simulate data strikes under a wide variety of conditions and explore how they can augment traditional boycotts. Our results suggest that data strikes can be effective and that users have more power in their relationship with technology companies than they do with other companies. However, our results also highlight important trade-offs and challenges that must be considered by potential organizers.

This article introduces Cartograph, a visualization system that harnesses the vast world knowledge encoded within Wikipedia to create thematic maps of almost any data. Cartograph extends previous systems that visualize non-spatial data using geographic approaches. Although these systems required data with an existing semantic structure, Cartograph unlocks spatial visualization for a much larger variety of datasets by enhancing input datasets with semantic information extracted from Wikipedia. Cartograph’s map embeddings use neural networks trained on Wikipedia article content and user navigation behavior. Using these embeddings, the system can reveal connections between points that are unrelated in the original datasets but are related in meaning and therefore embedded close together on the map. We describe the design of the system and key challenges we encountered. We present findings from two user studies exploring design choices and use of the system.

Peer production communities create valuable content such as software, encyclopedia articles, and map data. As part of the creation process, these communities define production standards for their content, e.g., semantic and syntactic requirements. We carried out a study in OpenStreetMap to investigate the role of that community's standards for geographic metadata. We found that most applied metadata was consistent with the community's standards; however, we also found that the standards identified many opportunities for applying metadata that were not achieved. In addition, when we situated the standards in the context of OpenStreetMap's data model, we found a significant amount of ambiguity; the syntax allowed only one value, but everyday meaning -- and the standards themselves -- called for multiple values. Our results suggest significant opportunities for OpenStreetMap to produce additional valuable open source content to power applications.

In addition to encyclopedia articles and software, peer production communities produce structured data, e.g., Wikidata and OpenStreetMap's metadata. Structured data from peer production communities has become increasingly important due to its use by computational applications, such as CartoCSS, MapBox, and Wikipedia infoboxes. However, this structured data is usable by applications only if it follows standards. We did an interview study focused on OpenStreetMap's knowledge production processes to investigate how -- and how successfully -- this community creates and applies its data standards. Our study revealed a fundamental tension between the need to produce structured data in a standardized way and OpenStreetMap's tradition of contributor freedom. We extracted six themes that manifested this tension and three overarching concepts, correctness, community, and code, which help make sense of and synthesize the themes. We also offered suggestions for improving OpenStreetMap's knowledge production processes, including new data models, sociotechnical tools, and community practices (e.g. stronger leadership).

This paper introduces Cartograph, a visualization system that harnesses the vast amount of world knowledge encoded within Wikipedia to create thematic maps of almost any data. Cartograph extends previous systems that visualize non-spatial data using geographic approaches. While these systems required data with an existing semantic structure, Cartograph unlocks spatial visualization for a much larger variety of datasets by enhancing input datasets with semantic information extracted from Wikipedia. Cartograph's map embeddings use neural networks trained on Wikipedia article content and user navigation behavior. Using these embeddings, the system can reveal connections between points that are unrelated in the original data sets, but are related in meaning and therefore embedded close together on the map. We describe the design of the system and key challenges we encountered, and we present findings from an exploratory user study

Localness is an oft-cited benefit of volunteered geographic information (VGI). This study examines whether localness is a constant, universally shared benefit of VGI, or one that varies depending on the context in which it is produced. Focusing on articles about geographic entities (e.g. cities, points of interest) in 79 language editions of Wikipedia, we examine the localness of both the editors working on articles and the sources of the information they cite. We find extensive geographic inequalities in localness, with the degree of localness varying with the socioeconomic status of the local population and the health of the local media. We also point out the key role of language, showing that information in languages not native to a place tends to be produced and sourced by non-locals. We discuss the implications of this work for our understanding of the nature of VGI and highlight a generalizable technical contribution: an algorithm that determines the home country of the original publisher of online content.

In just a few years, crowdsourcing markets like Mechanical Turk have become the dominant mechanism for for building "gold standard" datasets in areas of computer science ranging from natural language processing to audio transcription. The assumption behind this sea change - an assumption that is central to the approaches taken in hundreds of research projects - is that crowdsourced markets can accurately replicate the judgments of the general population for knowledge-oriented tasks. Focusing on the important domain of semantic relatedness algorithms and leveraging Clark's theory of common ground as a framework, we demonstrate that this assumption can be highly problematic. Using 7,921 semantic relatedness judgements from 72 scholars and 39 crowdworkers, we show that crowdworkers on Mechanical Turk produce significantly different semantic relatedness gold standard judgements than people from other communities. We also show that algorithms that perform well against Mechanical Turk gold standard datasets do significantly worse when evaluated against other communities' gold standards. Our results call into question the broad use of Mechanical Turk for the development of gold standard datasets and demonstrate the importance of understanding these datasets from a human-centered point-of-view. More generally, our findings problematize the notion that a universal gold standard dataset exists for all knowledge tasks.

Tobler's First Law of Geography (TFL) is one of the key reasons why "spatial is special". The law, which states that "everything is related to everything else, but near things are more related than distant things", is central to the management, presentation, and analysis of geographic information. However, despite the importance of TFL, we have a limited general understanding of its domain-neutral properties. In this paper, we leverage recent advances in the natural language processing domain of semantic relatedness estimation to, for the first time, robustly evaluate the extent to which relatedness between spatial entities decreases over distance in a domain-neutral fashion. Our results reveal that, in general, TFL can indeed be considered a globally recognized domain-neutral property of geographic information but that there is a distance beyond which being nearer, on average, no longer means being more related.

Wikipedia is known for serving humans' informational needs. Over the past decade, the encyclopedic knowledge encoded in Wikipedia has also powerfully served computer systems. Leading algorithms in artificial intelligence, natural language processing, data mining, geographic information science, and many other fields analyze the text and structure of articles to build computational models of the world.

We ask what kinds of sources Wikipedians value most and compare Wikipedia's stated policy on sources to what we observe in practice. We find that primary data sources developed by alternative publishers are both popular and persistent, despite policies that present such sources as inferior to scholarly secondary sources. We also find that Wikipedians make almost equal use of information produced by associations such as nonprofits as from scholarly publishers, with a significant portion coming from government information sources. Our findings suggest the rise of new influential sources of information on the Web but also reinforce the traditional geographic patterns of scholarly publication. This has a significant effect on the goal of Wikipedians to represent "the sum of all human knowledge."

Users have come to rely on automated route finding services for driving, public transit, walking, and bicycling. Current state of the art route finding algorithms typically rely on objective factors like time and distance; they do not consider subjective preferences that also influence route quality. This paper addresses that need. We introduce a new framework for evaluating edge rating prediction techniques in transportation networks and use it to explore ten families of prediction algorithms in Cyclopath, a geographic wiki that provides route finding services for bicyclists. Overall, we find that personalized algorithms predict more accurately than non-personalized ones, and we identify two algorithms with low error and excellent coverage, one of which is simple enough to be implemented in thin clients like web browsers. These results suggest that routing systems can generate better routes by collecting and analyzing users' subjective preferences.

Most recommender systems assume user ratings accurately represent user preferences. However, prior research shows that user ratings are imperfect and noisy. Moreover, this noise limits the measurable predictive power of any recommender system. We propose an information theoretic framework for quantifying the preference information contained in ratings and predictions. We computationally explore the properties of our model and apply our framework to estimate the efficiency of different rating scales for real world datasets. We then estimate how the amount of information predictions give to users is related to the scale ratings are collected on. Our findings suggest a tradeoff in rating scale granularity: while previous research indicates that coarse scales (such as thumbs up / thumbs down) take less time, we find that ratings with these scales provide less predictive value to users. We introduce a new measure, preference bits per second, to quantitatively reconcile this tradeoff.

This article introduces the tag genome, a data structure that extends the traditional tagging model to provide enhanced forms of user interaction. Just as a biological genome encodes an organism based on a sequence of genes, the tag genome encodes an item in an information space based on its relationship to a common set of tags. We present a machine learning approach for computing the tag genome, and we evaluate several learning models on a ground truth dataset provided by users. We describe an application of the tag genome called Movie Tuner which enables users to navigate from one item to nearby items along dimensions represented by tags. We present the results of a 7-week field trial of 2,531 users of Movie Tuner and a survey evaluating users’ subjective experience. Finally, we outline the broader space of applications of the tag genome.

The Macademia website promotes faculty collaboration and research by visualizing faculty research interests as a dynamic "constellation."Semantic similarity inference algorithms power the site's visualization, allowing users to spatially browse related research interests, and researchers who have those interests.

Tags help users understand a rich information space, by showing them specific text annotations for each item in the space and enabling them to search by these annotations. Often, however, users may wish to move from one item to other items that are similar overall, but that differ in key characteristics. For example, a user who loves Pulp Fiction might want to see a similar movie, but might be in a mood for a less "dark" movie. This paper introduces Movie Tuner, a novel interface that supports navigation from one item to nearby items along dimensions represented by tags. Movie Tuner is based on a data structure called the tag genome, which is described in separate work. The tag genome encodes each item's relationship to a common set of tags by applying machine learning algorithms to user-contributed content. The present paper discusses our design of Movie Tuner, including algorithms for navigating to new items and for suggesting tags for navigation. We present the results of a 7-week field trial of 2,531 users of Movie Tuner, and of a survey evaluating users' subjective experience.

Netflix.com uses star ratings, Digg.com uses up/down votes and Facebook uses a "like" but not a "dislike" button. Despite the popularity and diversity of these rating scales, research offers little guidance for designers choosing between them.

In this paper we introduce tag expression, a novel form of preference elicitation that combines elements from tagging and rating systems. Tag expression enables users to apply affect to tags to indicate whether the tag describes a reason they like, dislike, or are neutral about a particular item. We present a user interface for applying affect to tags, as well as a technique for visualizing the overall community's affect. By analyzing 27,773 tag expressions from 553 users entered in a 3-month period, we empirically evaluate our design choices. We also present results of a survey of 97 users that explores users' motivations in tagging and measures user satisfaction with tag expression.

Many websites use tags as a mechanism for improving item metadata through collective user effort. Users of tagging systems often apply far more tags to an item than a system can display. These tags can range in quality from tags that capture a key facet of an item, to those that are subjective, irrelevant, or misleading. In this paper we explore tag selection algorithms that choose the tags that sites display. Based on 225,000 ratings and survey responses, we conduct offline analyses of 21 tag selection algorithms. We select the three best performing algorithms from our offline analysis, and deploy them live on the MovieLens website to 5,695 users for three months. Based on our results, we offer tagging system designers advice about tag selection algorithms.

While recommender systems tell users what items they might like, explanations of recommendations reveal why they might like them. Explanations provide many benefits, from improving user satisfaction to helping users make better decisions. This paper introduces tagsplanations, which are explanations based on community tags. Tagsplanations have two key components: tag relevance, the degree to which a tag describes an item, and tag preference, the user's sentiment toward a tag. We develop novel algorithms for estimating tag relevance and tag preference, and we conduct a user study exploring the roles of tag relevance and tag preference in promoting effective tagsplanations. We also examine which types of tags are most useful for tagsplanations.

Tagging has emerged as a powerful mechanism that enables users to find, organize, and understand online entities. Recommender systems similarly enable users to efficiently navigate vast collections of items. Algorithms combining tags with recommenders may deliver both the automation inherent in recommenders, and the flexibility and conceptual comprehensibility inherent in tagging systems. In this paper we explore tagommenders, recommender algorithms that predict users' preferences for items based on their inferred preferences for tags. We describe tag preference inference algorithms based on users' interactions with tags and movies, and evaluate these algorithms based on tag preference ratings collected from 995 MovieLens users. We design and evaluate algorithms that predict users' ratings for movies based on their inferred tag preferences. Our tag-based algorithms generate better recommendation rankings than state-of-the-art algorithms, and they may lead to flexible recommender systems that leverage the characteristics of items users find most important.

Recommender systems try to address the "new user problem" by quickly and painlessly learning user preferences so that users can begin receiving recommendations as soon as possible. We take an expanded perspective on the new user experience, seeing it as an opportunity to elicit valuable contributions to the community and shape subsequent user behavior. We conducted a field experiment in MovieLens where we imposed additional work on new users: not only did they have to rate movies, they also had to enter varying numbers of tags. While requiring more work led to fewer users completing the entry process, the benefits were significant: the remaining users produced a large volume of tags initially, and continued to enter tags at a much higher rate than a control group. Further, their rating behavior was not depressed. Our results suggest that careful design of the initial user experience can lead to significant benefits for an online community.

Many online communities use tags - community selected words or phrases - to help people find what they desire. The quality of tags varies widely, from tags that capture akey dimension of an entity to those that are profane, useless, or unintelligible. Tagging systems must often select a subset of available tags to display to users due to limited screen space. Because users often spread tags they have seen, selecting good tags not only improves an individual's view of tags, it also encourages them to create better tags in the future. We explore implicit (behavioral) and explicit (rating) mechanisms for determining tag quality. Based on 102,056 tag ratings and survey responses collected from 1,039 users over 100 days, we offer simple suggestions to designers of online communities to improve the quality of tags seen by their users.

In support of social interaction and information sharing, online communities commonly provide interfaces for users to form or interact with groups. For example, a user of the social music recommendation site last.fm might join the "First Wave Punk" group to discuss his or her favorite band (The Clash) and listen to playlists generated by fellow fans. Clustering techniques provide the potential to automatically discover groups of users who appear to share interests. We explore this idea by describing algorithms for clustering users of an online community and automatically describing the resulting user groups. We designed these techniques for use in an online recommendation system with no pre-existing group functionality, which led us to develop an "activity-balanced clustering" algorithm that considers both user activity and user interests in forming clusters.

As the number of alerts generated by collaborative applications grows, users receive more unwanted alerts. FeedMe is a general alert management system based on XML feed protocols such as RSS and ATOM. In addition to traditional rule-based alert filtering, FeedMe uses techniques from machine-learning to infer alert preferences based on user feedback. In this paper, we present and evaluate a new collaborative naive Bayes filtering algorithm. Using FeedMe, we collected alert ratings from 33 users over 29 days. We used the data to design and verify the accuracy of the filtering algorithm and provide insights into alert prediction.

A tagging community's vocabulary of tags forms the basis for social navigation and shared expression.We present a user-centric model of vocabulary evolution in tagging communities based on community influence and personal tendency. We evaluate our model in an emergent tagging system by introducing tagging features into the MovieLens recommender system.We explore four tag selection algorithms for displaying tags applied by other community members. We analyze the algorithms 'effect on vocabulary evolution, tag utility, tag adoption, and user satisfaction.

In today's data-rich networked world, people express many aspects of their lives online. It is common to segregate different aspects in different places: you might write opinionated rants about movies in your blog under a pseudonym while participating in a forum or web site for scholarly discussion of medical ethics under your real name. However, it may be possible to link these separate identities, because the movies, journal articles, or authors you mention are from a sparse relation space whose properties (e.g., many items related to by only a few users) allow re-identification. This re-identification violates people's intentions to separate aspects of their life and can have negative consequences; it also may allow other privacy violations, such as obtaining a stronger identifier like name and address.This paper examines this general problem in a specific setting: re-identification of users from a public web movie forum in a private movie ratings dataset. We present three major results. First, we develop algorithms that can re-identify a large proportion of public users in a sparse relation space. Second, we evaluate whether private dataset owners can protect user privacy by hiding data; we show that this requires extensive and undesirable changes to the dataset, making it impractical. Third, we evaluate two methods for users in a public forum to protect their own privacy, suppression and misdirection. Suppression doesn't work here either. However, we show that a simple misdirection strategy works well: mention a few popular items that you haven't rated.