Ying-Chun Lin, Jennifer Neville, Jack Stokes, Longqi Yang, Tara Safavi, Mengting Wan, Scott Counts, Siddharth Suri, Reid Andersen, Xiaofeng Xu, Deepak Gupta, Sujay Kumar Jauhar, Xia Song, Georg Buscher, Saurabh Tiwary, Brent Hecht, Jaime Teevan. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.

Systemic property dispossession from minority groups has often been carried out in the name of technological progress. In this paper, we identify evidence that the current paradigm of large language models (LLMs) likely continues this long history. Examining common LLM training datasets, we find that a disproportionate amount of content authored by Jewish Americans is used for training without their consent. The degree of over-representation ranges from around 2x to around 6.5x. Given that LLMs may substitute for the paid labor of those who produced their training data, they have the potential to cause even more substantial and disproportionate economic harm to Jewish Americans in the coming years. This paper focuses on Jewish Americans as a case study, but it is probable that other minority communities (e.g., Asian Americans, Hindu Americans) may be similarly affected and, most importantly, the results should likely be interpreted as a “canary in the coal mine” that highlights deep structural concerns about the current LLM paradigm whose harms could soon affect nearly everyone. We discuss the implications of these results for the policymakers thinking about how to regulate LLMs as well as for those in the AI field who are working to advance LLMs. Our findings stress the importance of working together towards alternative LLM paradigms that avoid both disparate impacts and widespread societal harms.

Hypergraphs provide an effective abstraction for modeling multi-way group interactions among nodes, where each hyperedge can connect any number of nodes. Different from most existing studies which leverage statistical dependencies, we study hypergraphs from the perspective of causality. Specifically, we focus on the problem of individual treatment effect (ITE) estimation on hypergraphs, aiming to estimate how much an intervention (e.g., wearing face covering) would causally affect an outcome (e.g., COVID-19 infection) of each individual node. Existing works on ITE estimation either assume that the outcome of one individual should not be influenced by the treatment of other individuals (i.e., no interference), or assume the interference only exists between connected individuals in an ordinary graph. We argue that these assumptions can be unrealistic on real-world hypergraphs, where higher-order interference can affect the ITE estimations due to group interactions. We investigate high-order interference modeling, and propose a new causality learning framework powered by hypergraph neural networks. Extensive experiments on real-world hypergraphs verify the superiority of our framework over existing baselines.

Making recommendations for users in diverse organizations (orgs) is a challenging task for workplace social platforms such as Microsoft Teams and Slack. The current industry-standard model training approaches either use data from all organizations to maximize information or train organization-specific models to minimize noise. Our real-world experiments show that both approaches are poorly suited for the multi-org recommendation setting where different organizations’ interaction patterns vary in their generalizability. We introducetargeted training, which improves on standard practices by automatically selecting a subset of orgs for model development whose data are cleanest and best represent global trends. We demonstrate how and when targeted training improves over global training through theoretical analysis and simulation. Our experiments on large-scale datasets from Microsoft Teams, SharePoint, Stack Exchange, DBLP, and Reddit show that in many cases targeted training can improve mean average precision (MAP) across orgs by 10–15% over global training, is more robust to orgs with lower data quality, and generalizes better to unseen orgs. Our training framework is applicable to a wide range of inductive recommendation models, from simple regression models to graph neural networks (GNNs).

Hypergraphs provide an effective abstraction for modeling multi-way group interactions among nodes, where each hyperedge can connect any number of nodes. Different from most existing studies which leverage statistical dependencies, we study hypergraphs from the perspective of causality. Specifically, in this paper, we focus on the problem of individual treatment effect (ITE) estimation on hypergraphs, aiming to estimate how much an intervention (e.g., wearing face covering) would causally affect an outcome (e.g., COVID-19 infection) of each individual node. Existing works on ITE estimation either assume that the outcome on one individual should not be influenced by the treatment assignments on other individuals (i.e., no interference), or assume the interference only exists between pairs of connected individuals in an ordinary graph. We argue that these assumptions can be unrealistic on real-world hypergraphs, where higher-order interference can affect the ultimate ITE estimations due to the presence of group interactions. In this work, we investigate high-order interference modeling, and propose a new causality learning framework powered by hypergraph neural networks. Extensive experiments on real-world hypergraphs verify the superiority of our framework over existing baselines.

Virtual meetings are critical for remote work because of the need for synchronous collaboration in the absence of in-person interactions. In-meeting multitasking is closely linked to people’s productivity and wellbeing. However, we currently have limited understanding of multitasking in remote meetings and its potential impact. In this paper, we present what we believe is the most comprehensive study of remote meeting multitasking behavior through an analysis of a large-scale telemetry dataset collected from February to May 2020 of U.S. Microsoft employees and a 715-person diary study. Our results demonstrate that intrinsic meeting characteristics such as size, length, time, and type, significantly correlate with the extent to which people multitask, and multitasking can lead to both positive and negative outcomes. Our findings suggest important best-practice guidelines for remote meetings (e.g., avoid important meetings in the morning) and design implications for productivity tools (e.g., support positive remote multitasking).

Motives or goals are recognized in psychology literature as the most fundamental drive that explains and predicts why people do what they do, including when they browse the web. Although providing enormous value, these higher-ordered goals are often unobserved, and little is known about how to leverage such goals to assist people’s browsing activities. This paper proposes to take a new approach to address this problem, which is fulfilled through a novel neural framework, Goal-directed Web Browsing (GoWeB). We adopt a psychologically-sound taxonomy of higher-ordered goals and learn to build their representations in a structure-preserving manner. Then we incorporate the resulting representations for enhancing the experiences of common activities people perform on the web. Experiments on large-scale data from Microsoft Edge web browser show that GoWeB significantly outperforms competitive baselines for in-session web page recommendation, re-visitation classification, and goal-based web page grouping. A follow-up analysis further characterizes how the variety of human motives can affect the difference observed in human behavioral patterns.

Millions of people use mobile map applications like Google Maps on a regular basis. However, despite these applications' ubiquity, the literature contains very little information about how these applications are used in the real world. As such, many researchers and practitioners seeking to improve mobile map applications may not be able to identify important challenges and may miss major opportunities for innovation. To address this paucity of usage information, we collected and analysed data during unsupervised usage of Google Maps by replacing the standard application with a wrapped version called MapRecorder. In two studies we recorded data from locals and tourists using our application and collected over 580 minutes of actual application usage from 34 users, spanning 555 unique sessions. We identify typical usage scenarios, observe a large amount of map exploration and elucidate generalisable interaction patterns.

The rise of geotargeted online advertising has disrupted the business model of local journalism, but it remains ambiguous whether online advertising platforms can effectively reach local audiences. To address this ambiguity, we present a focused study auditing the positional accuracy of geotargeted display advertisements on Google. We measure the frequency and severity of geotargeting errors by targeting display ads to random ZIP codes across the United States, collecting self-reported location information from users who click on the advertisement. We find evidence that geotargeting errors are common, but minor in terms of advertising goals. While 41% of respondents lived outside the target ZIP code, only 11% lived outside the target county, and only 2% lived outside the target state. We also present details regarding a high volume of suspicious clicks in our data, which made the cost per sample extremely expensive. The paper concludes by discussing implications for advertisers, the business of local journalism, and future research.

A growing body of work has highlighted the important role that Wikipedia's volunteer-created content plays in helping search engines achieve their core goal of addressing the information needs of hundreds of millions of people. In this paper, we report the results of an investigation into the incidence of Wikipedia links in search engine results pages (SERPs). Our results extend prior work by considering three U.S. search engines, simulating both mobile and desktop devices, and using a spatial analysis approach designed to study modern SERPs that are no longer just "ten blue links". We find that Wikipedia links are extremely common in important search contexts, appearing in 67-84% of desktop SERPs for common and trending queries, but less often for medical queries. Furthermore, we observe that Wikipedia links often appear in "Knowledge Panel" SERP elements and are in positions visible to users without scrolling, although Wikipedia appears less often and in less prominent positions on mobile devices. Our findings reinforce the complementary notions that (1) Wikipedia content and research has major impact outside of the Wikipedia domain and (2) powerful technologies like search engines are highly reliant on free content created by volunteers.

Tech users currently have limited ability to act on concerns regarding the negative societal impacts of large tech companies. However, recent work suggests that users can exert leverage using their role in the generation of valuable data, for instance by withholding their data contributions to intelligent technologies. We propose and evaluate a new means to exert this type of leverage against tech companies: "conscious data contribution" (CDC). Users who participate in CDC exert leverage against a target tech company by contributing data to technologies operated by a competitor of that company. Using simulations, we find that CDC could be highly effective at reducing the gap in intelligent technologies performance between an incumbent and their competitors. In some cases, just 20% of users contributing data they have produced to a small competitor could help that competitor get 80% of the way towards the original company's best-case performance. We discuss the implications of CDC for policymakers, tech designers, and researchers.

Even though a restaurant may receive different ratings across review platforms, people often see only one rating during a local search (e.g. 'best burgers near me'). In this paper, we examine the differences in ratings between two commonly used review platforms-Google Maps and Yelp. We found that restaurant ratings on Google Maps are, on average, 0.7 stars higher than those on Yelp, with the increase being driven in large part by higher ratings for chain restaurants on Google Maps. We also found extensive diversity i¬¬n top-ranked restaurants by geographic region across platforms. For example, for a given metropolitan area, there exists little overlap in its top ten lists of restaurants on Google Maps and Yelp. Our results problematize the use of a single review platform in local search and have implications for end users of ratings and local search technologies. We outline concrete design recommendations to improve communication of restaurant evaluation and discuss the potential causes for the divergence we observed.

Millions of people worldwide contribute content to peer production repositories that serve human information needs and provide vital world knowledge to prominent artificial intelligence systems. Yet, extreme gender participation disparities exist in which men significantly outnumber women. A central concern has been that due to self-focus bias, these disparities can lead to corresponding gender content disparities, in which content of interest to men is better represented than content of interest to women. This paper investigates the relationship between participation and content disparities in OpenStreetMap. We replicate findings that women are dramatically under-represented as OSM contributors, and observe that men and women contribute different types of content and do so about different places. However, the character of these differences confound simple narratives about self-focus bias: we find that on a proportional basis, men produced a higher proportion of contributions in feminized spaces compared to women, while women produced a higher proportion of contributions in masculinized spaces compared to men.

Group projects are an essential component of teaching user interface (UI) design. We identified six challenges in transferring traditional group projects into the context of Massive Open Online Courses: managing dropout, avoiding free-riding, appropriate scaffolding, cultural and time zone differences, and establishing common ground. We present a case study of the design of a group project for a UI Design MOOC, in which we implemented technical tools and social structures to cope with the above challenges. Based on survey analysis, interviews, and team chat data from the students over a six-month period, we found that our socio-technical design addressed many of the obstacles that MOOC learners encountered during remote collaboration. We translate our findings into design implications for better group learning experiences at scale.

The public is increasingly concerned about the practices of large technology companies with regards to privacy and many other issues. To force changes in these practices, there have been growing calls for “data strikes.” These new types of collective action would seek to create leverage for the public by starving business-critical models (e.g. recommender systems, ranking algorithms) of much-needed training data. However, little is known about how data strikes would work, let alone how effective they would be. Focusing on the important commercial domain of recommender systems, we simulate data strikes under a wide variety of conditions and explore how they can augment traditional boycotts. Our results suggest that data strikes can be effective and that users have more power in their relationship with technology companies than they do with other companies. However, our results also highlight important trade-offs and challenges that must be considered by potential organizers.

Researchers and the media have become increasingly interested in protest users, or people who change (protest use) or stop (protest non-use) their use of a company's products because of the company's values and/or actions. Past work has extensively engaged with the phenomenon of technology non-use but has not focused on non-use (nor changed use) in the context of protest. With recent research highlighting the potential for protest users to exert leverage against technology companies, it is important for technology stakeholders to understand the prevalence of protest users, their motivations, and the specific tactics they currently use. In this paper, we report the results of two surveys (n = 463 and n = 398) of representative samples of American web users that examine if, how, and why people have engaged in protest use and protest non-use of the products of five major technology companies. We find that protest use and protest non-use are relatively common, with 30% of respondents in 2019 reporting they were protesting at least one major tech company. Furthermore, we identify that protest users' most common motivations were (1) concerns about business models that profit from user data and (2) privacy; and the most common tactics were (1) stopping use and (2) leveraging ad blockers. We also identify common challenges and roadblocks faced by active and potential protest users, which include (1) losing social connections and (2) the lack of alternative products. Our results highlight the growing importance of protest users in the technology ecosystem and the need for further social computing research into this phenomenon. We also provide concrete design implications for existing and future technologies to support or account for protest use and protest non-use.

This article introduces Cartograph, a visualization system that harnesses the vast world knowledge encoded within Wikipedia to create thematic maps of almost any data. Cartograph extends previous systems that visualize non-spatial data using geographic approaches. Although these systems required data with an existing semantic structure, Cartograph unlocks spatial visualization for a much larger variety of datasets by enhancing input datasets with semantic information extracted from Wikipedia. Cartograph’s map embeddings use neural networks trained on Wikipedia article content and user navigation behavior. Using these embeddings, the system can reveal connections between points that are unrelated in the original datasets but are related in meaning and therefore embedded close together on the map. We describe the design of the system and key challenges we encountered. We present findings from two user studies exploring design choices and use of the system.

Many applications of geotagged content are predicated on the concept of localness (e.g., local restaurant recommendation, mining social media for local perspectives on an issue). However, definitions of who is a "local" in a given area are typically informal and ad-hoc and, as a result, approaches for localness assessment that have been used in the past have not been formally validated. In this paper, we begin the process of addressing these gaps in the literature. Specifically, we (1) formalize definitions of "local" using themes identified in a 30-paper literature review, (2) develop the first ground truth localness dataset consisting of 132 Twitter users and 58,945 place-tagged tweets, and (3) use this dataset to evaluate existing localness assessment approaches. Our results provide important methodological guidance to the large body of research and practice that depends on the concept of localness and suggest means by which localness assessment can be improved.

Volunteered Geographic Information (VGI), such as contributions to OpenStreetMap and geotagged Wikipedia articles, is often assumed to be produced locally. However, recent work has found that peer-produced VGI is frequently contributed by non-locals. We evaluate this approach across hundreds of content types from Wikipedia, OpenStreetMap, and eBird, and show that these models can describe more than 90% of "VGI flows" for some content types. Our findings advance geographic HCI theory, suggesting some spatial mechanisms underpinning VGI production. We also discuss design implications that can help (a) human and algorithmic consumers of VGI evaluate the perspectives it contains and (b) address geographic coverage variations in these platforms (e.g. via more effective volunteer recruitment strategies).

The extensive Wikipedia literature has largely considered Wikipedia in isolation, outside of the context of its broader Internet ecosystem. Very recent research has demonstrated the significance of this limitation, identifying critical relationships between Google and Wikipedia that are highly relevant to many areas of Wikipedia-based research and practice. This paper extends this recent research beyond search engines to examine Wikipedia's relationships with large-scale online communities, Stack Overflow and Reddit in particular. We find evidence of consequential, albeit unidirectional relationships. Wikipedia provides substantial value to both communities, with Wikipedia content increasing visitation, engagement, and revenue, but we find little evidence that these websites contribute to Wikipedia in return. Overall, these findings highlight important connections between Wikipedia and its broader ecosystem that should be considered by researchers studying Wikipedia. Critically, our results also emphasize the key role that volunteer-created Wikipedia content plays in improving other websites, even contributing to revenue generation.

The evolution of contributor behavior in peer production communities over time has been a subject of substantial interest in the social computing community. In this paper, we extend this literature to the geographic domain, exploring contribution behavior in OpenStreetMap using a spatiotemporal lens. In doing so, we observe a geographic version of a 'born, not made' phenomenon: throughout their lifespans, contributors are relatively consistent in the places and types of places that they edit. We show how these 'born, not made' trends may help explain the urban and socioeconomic coverage biases that have been observed in OpenStreetMap. We also discuss how our findings can help point towards solutions to these biases.

Peer production communities create valuable content such as software, encyclopedia articles, and map data. As part of the creation process, these communities define production standards for their content, e.g., semantic and syntactic requirements. We carried out a study in OpenStreetMap to investigate the role of that community's standards for geographic metadata. We found that most applied metadata was consistent with the community's standards; however, we also found that the standards identified many opportunities for applying metadata that were not achieved. In addition, when we situated the standards in the context of OpenStreetMap's data model, we found a significant amount of ambiguity; the syntax allowed only one value, but everyday meaning -- and the standards themselves -- called for multiple values. Our results suggest significant opportunities for OpenStreetMap to produce additional valuable open source content to power applications.

Data visualizations in news articles (e.g., maps, line graphs, bar charts) greatly enrich the content of news articles and result in well-established improvements to reader comprehension. However, existing systems that generate news data visualiza-tions either require substantial manual effort or are limited to very specific types of data visualizations, thereby greatly re-stricting the number of news articles that can be enhanced. To address this issue, we define a new problem: given a news ar-ticle, retrieve relevant visualizations that already exist on the web. We show that this problem is tractable through a new system, VizByWiki, that mines contextually relevant data visualizations from Wikimedia Commons, the central file reposi-tory for Wikipedia. Using a novel ground truth dataset, we show that VizByWiki can successfully augment as many as 48% of popular online news articles with news visualizations. We also demonstrate that VizByWiki can automatically rank visualizations according to their usefulness with reasonable accuracy ([email protected] of 0.82). To facilitate further advances on our "news visualization retrieval problem", we release our ground truth dataset and make our system and its source code publicly available.

In many traditional labor markets, women earn less on average compared to men. However, it is unclear whether this discrepancy persists in the online gig economy, which bears important differences from the traditional labor market (e.g., more flexible work arrangements, shorter-term engagements, reputation systems). In this study, we collected self-determined hourly bill rates from the public profiles of 48,019 workers in the United States (48.8% women) on Upwork, a popular gig work platform. The median female worker set hourly bill rates that were 74% of the median man's hourly bill rates, a gap than cannot be entirely explained by online and offline work experience, education level, and job category. However, in some job categories, we found evidence of a more complex relationship between gender and earnings: women earned more overall than men by working more hours, outpacing the effect of lower hourly bill rates. To better support equality in the rapidly growing gig economy, we encourage continual evaluation of the complex gender dynamics on these platforms and discuss whose responsibility it is to address inequalities.

Emoji are popular in digital communication, but they are rendered differently on different viewing platforms (e.g., iOS, Android). It is unknown how many people are aware that emoji have multiple renderings, or whether they would change their emoji-bearing messages if they could see how these messages render on recipients' devices. We developed software to expose the multi-rendering nature of emoji and explored whether this increased visibility would affect how people communicate with emoji. Through a survey of 710 Twitter users who recently posted an emoji-bearing tweet, we found that at least 25% of respondents were unaware that the emoji they posted could appear differently to their followers. Additionally, after being shown how one of their tweets rendered across platforms, 20% of respondents reported that they would have edited or not sent the tweet. These statistics reflect millions of potentially regretful tweets shared per day because people cannot see emoji rendering differences across platforms. Our results motivate the development of tools that increase the visibility of emoji rendering differences across platforms, and we contribute our cross-platform emoji rendering software to facilitate this effort.

GrabYourWallet, #boycottNRA and other online boycott campaigns have attracted substantial public interest in recent months. However, a number of significant challenges are preventing online boycotts from reaching their potential. In particular, complex webs of brands and subsidiaries can make it difficult for participants to conform to the goals of a boycott. Similarly, participants and organizers have limited visibility into a boycott's progress. This affects their ability to use sociotechnical innovations from social computing to incentivize participation. To address these challenges, this paper makes a system contribution: a new boycott tool called Out of Site . Out of Site uses lightweight automation to remove obstacles to successful online boycotts. We describe the design challenges associated with Out of Site and report results from two phases of deployment with the GrabYourWallet and Stop Animal Testing boycott communities. Our findings highlight the potential of boycott-assisting technologies and inform the design of this new class of technologies. Finally, like is the case for many systems in social computing, while we designed Out of Site for pro-social uses, there are a number of easily predictable ways in which the system can be leveraged for anti-social purposes (e.g. exacerbating filter bubble issues, empowering boycotts of businesses owned by racial, ethnic, and religious minorities). As such, we developed for this project a new, very straightforward design approach that treats preventing these anti-social uses as a top-tier design concern. This approach stands in contrast to the status quo of ignoring potential anti-social uses and/or considering them to be a secondary design priority. We discuss how our simple approach may help other research projects reduce their potential negative impacts with minimal burden.

The widespread popularity of Pokémon GO presents the first opportunity to observe the geographic effects of location-based gaming at scale. This paper reports the results of a mixed methods study of the geography of Pokémon GO that includes a five-country field survey of 375 Pokémon GO players and a large scale geostatistical analysis of game elements. Focusing on the key geographic themes of places and movement, we find that the design of Pokémon GO reinforces existing geographically-linked biases (e.g. the game advantages urban areas and neighborhoods with smaller minority populations), that Pokémon GO may have instigated a relatively rare large-scale shift in global human mobility patterns, and that Pokémon GO has geographically-linked safety risks, but not those typically emphasized by the media. Our results point to geographic design implications for future systems in this space such as a means through which the geographic biases present in Pokémon GO may be counteracted.

In addition to encyclopedia articles and software, peer production communities produce structured data, e.g., Wikidata and OpenStreetMap's metadata. Structured data from peer production communities has become increasingly important due to its use by computational applications, such as CartoCSS, MapBox, and Wikipedia infoboxes. However, this structured data is usable by applications only if it follows standards. We did an interview study focused on OpenStreetMap's knowledge production processes to investigate how -- and how successfully -- this community creates and applies its data standards. Our study revealed a fundamental tension between the need to produce structured data in a standardized way and OpenStreetMap's tradition of contributor freedom. We extracted six themes that manifested this tension and three overarching concepts, correctness, community, and code, which help make sense of and synthesize the themes. We also offered suggestions for improving OpenStreetMap's knowledge production processes, including new data models, sociotechnical tools, and community practices (e.g. stronger leadership).

Much research has shown that social media platforms have substantial population biases. However, very little is known about how these population biases affect the many algorithms that rely on social media data. Focusing on the case study of geolocation inference algorithms and their performance across the urban-rural spectrum, we establish that these algorithms exhibit significantly worse performance for underrepresented populations (i.e. rural users). We further establish that this finding is robust across both text- and network-based algorithm designs. However, we also show that some of this bias can be attributed to the design of algorithms themselves rather than population biases in the underlying data sources. For instance, in some cases, algorithms perform badly for rural users even when we substantially overcorrect for population biases by training exclusively on rural data. We discuss the implications of our findings for the design and study of social media-based algorithms.

Catastrophic incidents associated with GPS devices and other personal navigation technologies are sufficiently common that these incidents have been given a colloquial nickname: "Death by GPS". While there is a significant body of work on the use of personal navigation technologies in everyday scenarios, no research has examined these technologies' roles in catastrophic incidents. In this paper, we seek to address this gap in the literature. Borrowing techniques from public health research and communication studies, we construct a corpus of 158 detailed news reports of unique catastrophic incidents associated with personal navigation technologies. We then identify key themes in these incidents and the roles that navigation technologies played in them, e.g. missing road characteristics data contributed to over 25% of these incidents. With the goal of reducing casualties associated with personal navigation technologies, we outline implications for design and research that emerge from our results, e.g. advancing "space usage rule" mapping, incorporating weather information in routing, and improving visual and audio instructions in complex situations.

Wikipedia-based studies and systems frequently assume that no two articles describe the same concept. However, in this paper, we show that this article-as-concept assumption is problematic due to editors' tendency to split articles into parent articles and sub-articles when articles get too long for readers (e.g. "Portland, Oregon" and "History of Portland, Oregon" in the English Wikipedia). In this paper, we present evidence that this issue can have significant impacts on Wikipedia-based studies and systems and introduce the sub-article matching problem. The goal of the sub-article matching problem is to automatically connect sub-articles to parent articles to help Wikipedia-based studies and systems retrieve complete information about a concept. We then describe the first system to address the sub-article matching problem. We show that, using a diverse feature set and standard machine learning techniques, our system can achieve good performance on most of our ground truth datasets, significantly outperforming baseline approaches.

This paper introduces Cartograph, a visualization system that harnesses the vast amount of world knowledge encoded within Wikipedia to create thematic maps of almost any data. Cartograph extends previous systems that visualize non-spatial data using geographic approaches. While these systems required data with an existing semantic structure, Cartograph unlocks spatial visualization for a much larger variety of datasets by enhancing input datasets with semantic information extracted from Wikipedia. Cartograph's map embeddings use neural networks trained on Wikipedia article content and user navigation behavior. Using these embeddings, the system can reveal connections between points that are unrelated in the original data sets, but are related in meaning and therefore embedded close together on the map. We describe the design of the system and key challenges we encountered, and we present findings from an exploratory user study

Landmark-based navigation systems have proven benefits relative to traditional turn-by-turn systems that use street names and distances. However, one obstacle to the implementation of landmark-based navigation systems is the complex challenge of selecting salient local landmarks at each decision point for each user. In this paper, we present Pharos, a novel system that extends turn-by-turn navigation instructions using a single global landmark (e.g. the Eiffel Tower, the Burj Khalifa, municipal TV towers) rather than multiple, hard-to-select local landmarks. We first show that our approach is feasible in a large number of cities around the world through the use of computer vision to select global landmarks. We then present the results of a study demonstrating that by including global landmarks in navigation instructions, users navigate more confidently and build a more accurate mental map of the navigated area than using turn-by-turn instructions.

While popular digital maps support an unprecedented number of use cases, new reference map customization tools have been created for purposes for which those maps fall short. With the goal of informing the design of this new class of cartographic tools, we present the first study of naturalistic ("bespoke") map customization behavior. Through a mixed methods and mixed-media approach involving a survey, the analysis of a corpus of customized maps, and an interview with a power user, we find that bespoke map customization is a relatively common activity and identify frequent use cases as well as map customization strategies. We discuss these use cases and strategies in detail, and propose design implications for future customization tools, such as the use of templates for common use cases, adaptability for various customization styles and the support of multimedia interaction.

The sharing economy has quickly become a very prominent subject of research in the broader computing literature and the in human--computer interaction (HCI) literature more specifically. When other computing research areas have experienced similarly rapid growth (e.g. human computation, eco-feedback technology), early stage literature reviews have proved useful and influential by identifying trends and gaps in the literature of interest and by providing key directions for short- and long-term future work. In this paper, we seek to provide the same benefits with respect to computing research on the sharing economy. Specifically, following the suggested approach of prior computing literature reviews, we conducted a systematic review of sharing economy articles published in the Association for Computing Machinery Digital Library to investigate the state of sharing economy research in computing. We performed this review with two simultaneous foci: a broad focus toward the computing literature more generally and a narrow focus specifically on HCI literature. We collected a total of 112 sharing economy articles published between 2008 and 2017 and through our analysis of these papers, we make two core contributions: (1) an understanding of the computing community's contributions to our knowledge about the sharing economy, and specifically the role of the HCI community in these contributions (i.e. what has been done ) and (2) a discussion of under-explored and unexplored aspects of the sharing economy that can serve as a partial research agenda moving forward (i.e. what is next to do ).

As the gig economy continues to grow and freelance work moves online, five-star reputation systems are becoming more and more common. At the same time, there are increasing accounts of race and gender bias in evaluations of gig workers, with negative impacts for those workers. We report on a series of four Mechanical Turk-based studies in which participants who rated simulated gig work did not show race- or gender bias, while manipulation checks showed they reliably distinguished between low- and high-quality work. Given prior research, this was a striking result. To explore further, we used a Bayesian approach to verify absence of ratings bias (as opposed to merely not detecting bias). This Bayesian test let us identify an upper- bound: if any bias did exist in our studies, it was below an average of 0.2 stars on a five-star scale. We discuss possible interpretations of our results and outline future work to better understand the results.

Despite the geographically situated nature of most sharing economy tasks, little attention has been paid to the role that geography plays in the sharing economy. In this article, we help to address this gap in the literature by examining how four key principles from human geography—distance decay, structured variation in population density, mental maps, and “the Big Sort” (spatial homophily)—manifest in sharing economy platforms. We find that these principles interact with platform design decisions to create systemic biases in which the sharing economy is significantly more effective in dense, high socioeconomic status (SES) areas than in low-SES areas and the suburbs. We further show that these results are robust across two sharing economy platforms: UberX and TaskRabbit. In addition to highlighting systemic sharing economy biases, this article more fundamentally demonstrates the importance of considering well-known geographic principles when designing and studying sharing economy platforms.

Wikipedia articles about places, OpenStreetMap features, and other forms of peer-produced content have become critical sources of geographic knowledge for humans and intelligent technologies. In this paper, we explore the effectiveness of the peer production model across the rural/urban divide, a divide that has been shown to be an important factor in many online social systems. We find that in both Wikipedia and OpenStreetMap, peer-produced content about rural areas is of systematically lower quality, is less likely to have been produced by contributors who focus on the local area, and is more likely to have been generated by automated software agents (i.e. "bots"). We then codify the systemic challenges inherent to characterizing rural phenomena through peer production and discuss potential solutions.

Geotagged tweets and other forms of social media volunteered geographic information (VGI) are becoming increasingly critical to many applications and scientific studies. An important assumption underlying much of this research is that social media VGI is "local", or that its geotags correspond closely with the general home locations of its contributors. We demonstrate through a study on three separate social media communities (Twitter, Flickr, Swarm) that this localness assumption holds in only about 75% of cases. In addition, we show that the geographic contours of localness follow important sociodemographic trends, with social media in, for instance, rural areas and older areas, being substantially less local in character (when controlling for other demographics). We demonstrate through a case study that failure to account for non-local social media VGI can lead to misrepresentative results in social media VGI-based studies. Finally, we compare the methods for determining localness, finding substantial disagreement in certain cases, and highlight new best practices for social media VGI-based studies and systems.

The lack of certain types of geographic data prevents the development of location-aware technologies in a number of important domains. One such type of "unmapped" geographic data is space usage rules (SURs), which are defined as geographically-bound activity restrictions (e.g. "no dogs", "no smoking", "no fishing", "no skateboarding"). Researchers in the area of human-computer interaction have recently begun to develop techniques for the automated mapping of SURs with the aim of supporting activity planning systems (e.g. one-touch "Can I Smoke Here?" apps, SUR-aware vacation planning tools). In this paper, we present a novel SUR mapping technique -- SPtP -- that outperforms state-of-the-art approaches by 30% for one of the most important components of the SUR mapping pipeline: associating a point observation of a SUR (e.g. a 'no smoking' sign) with the corresponding polygon in which the SUR applies (e.g. the nearby park or the entire campus on which the sign is located). This paper also contributes a series of new SUR benchmark datasets to help further research in this area.

Providing pedestrian navigation instructions on small screens is a challenging task due to limited screen space. As image-based approaches for navigation have been successfully proven to outperform map-based navigation on mobile devices, we propose to bring image-based navigation to smartwatches. We contribute a straightforward pipeline to easily create image-based indoor navigation instructions that allow users to freely navigate in indoor environments without any localization infrastructure and with minimal user input on the smartwatch. In a user study, we show that our approach outperforms the current state-of-the art application in terms of task completion time, perceived task load and perceived usability. In addition, we did not find an indication that there is a need to provide explicit directional instructions for image-based navigation on small screens.

Recent work has identified the lack of space usage rule (SUR) data -- e.g. "no smoking", "no campfires" -- as an important limitation of online/mobile maps that presents risks to user safety and the environment. In order to address this limitation, a large-scale means of mapping SURs must be developed. In this paper, we introduce and motivate the problem of mapping space usage rules and take the first steps towards identifying solutions. We show how computer vision can be employed to identify SUR indicators in the environment (e.g. "No Smoking" signs) with reasonable accuracy and describe techniques that can assign each rule to the appropriate geographic feature.

Localness is an oft-cited benefit of volunteered geographic information (VGI). This study examines whether localness is a constant, universally shared benefit of VGI, or one that varies depending on the context in which it is produced. Focusing on articles about geographic entities (e.g. cities, points of interest) in 79 language editions of Wikipedia, we examine the localness of both the editors working on articles and the sources of the information they cite. We find extensive geographic inequalities in localness, with the degree of localness varying with the socioeconomic status of the local population and the health of the local media. We also point out the key role of language, showing that information in languages not native to a place tends to be produced and sourced by non-locals. We discuss the implications of this work for our understanding of the nature of VGI and highlight a generalizable technical contribution: an algorithm that determines the home country of the original publisher of online content.

In just a few years, crowdsourcing markets like Mechanical Turk have become the dominant mechanism for for building "gold standard" datasets in areas of computer science ranging from natural language processing to audio transcription. The assumption behind this sea change - an assumption that is central to the approaches taken in hundreds of research projects - is that crowdsourced markets can accurately replicate the judgments of the general population for knowledge-oriented tasks. Focusing on the important domain of semantic relatedness algorithms and leveraging Clark's theory of common ground as a framework, we demonstrate that this assumption can be highly problematic. Using 7,921 semantic relatedness judgements from 72 scholars and 39 crowdworkers, we show that crowdworkers on Mechanical Turk produce significantly different semantic relatedness gold standard judgements than people from other communities. We also show that algorithms that perform well against Mechanical Turk gold standard datasets do significantly worse when evaluated against other communities' gold standards. Our results call into question the broad use of Mechanical Turk for the development of gold standard datasets and demonstrate the importance of understanding these datasets from a human-centered point-of-view. More generally, our findings problematize the notion that a universal gold standard dataset exists for all knowledge tasks.

Mobile crowdsourcing markets (e.g., Gigwalk and TaskRabbit) offer crowdworkers tasks situated in the physical world (e.g., checking street signs, running household errands). The geographic nature of these tasks distinguishes these markets from online crowdsourcing markets and raises new, fundamental questions. We carried out a controlled study in the Chicago metropolitan area aimed at addressing two key questions: (1) What geographic factors influence whether a crowdworker will be willing to do a task? (2) What geographic factors influence how much compensation a crowdworker will demand in order to do a task? Quantitative modeling shows that travel distance to the location of the task and the socioeconomic status (SES) of the task area are important factors. Qualitative analysis enriches our modeling, with workers mentioning safety and difficulties getting to a location as key considerations. Our results suggest that low-SES areas are currently less able to take advantage of the benefits of mobile crowdsourcing markets. We discuss the implications of our study for these markets, as well as for "sharing economy" phenomena like UberX, which have many properties in common with mobile crowdsourcing markets.

Peer production communities have been proven to be successful at creating valuable artefacts, with Wikipedia as a prime example. However, a number of studies have shown that work in these communities tends to be of uneven quality and certain content areas receive more attention than others. In this paper, we examine the efficacy of a range of targeted strategies to increase the quality of under-attended content areas in peer production communities. Mining data from five quality improvement projects in the English Wikipedia, the largest peer production community in the world, we show that certain types of strategies (e.g. creating artefacts from scratch) have better quality outcomes than others (e.g. improving existing artefacts), even if both are done by a similar cohort of participants. We discuss the implications of our findings for Wikipedia as well as other peer production communities.

Map applications for smartwatches present new challenges in cartography, a domain in which large display sizes have significant advantages. In this paper, we introduce StripeMaps, a system that adapts the mobile web design technique of linearization for displaying maps on the small screens of smartwatches. Just as web designers simplify multiple column desktop websites into a single column for easier navigation on mobile devices, StripeMaps transforms any two-dimensional route map into a one-dimensional "stripe". Through a user study, we show that this simplification allows StripeMaps to outperform both traditional mobile map interfaces and turn-by-turn directions for pedestrian navigation using smartwatches. In addition to introducing StripeMaps, this paper also has a secondary contribution. It contains the first empirical comparison of different approaches for pedestrian smartwatch navigation and illuminates their pros and cons.

Despite the large and growing prominence of online and mobile maps, they have not been broadly and systematically examined with a lens informed by traditional cartography. Using an approach rooted in cartographic theory and a unique dataset of 382 publicly-displayed local maps, we identify the collective wisdom of hundreds of cartographers with respect to a number of cartographic design decisions. We compare our findings to the approaches taken in popular online and mobile map platforms and develop suggestions for incorporating the collective wisdom of cartographers into these systems. Our suggestions include the adoption of location-aware cartography, in which cartographic approaches are intelligently varied based on the type of location being viewed. We provide mockup designs of online and mobile maps that implement our suggestions and discuss means by which the surprising gap between online and mobile maps and traditional cartography may be bridged.

Interactive visualizations add rich, data-based context to online news articles. Geographic maps are currently the most prevalent form of these visualizations. Unfortunately, designers capable of producing high-quality, customized geovisualizations are scarce. We present NewsViews, a novel automated news visualization system that generates interactive, annotated maps without requiring professional designers. NewsViews' maps support trend identification and data comparisons relevant to a given news article. The NewsViews system leverages text mining to identify key concepts and locations discussed in articles (as well as potential annotations), an extensive repository of 'found' databases, and techniques adapted from cartography to identify and create visually 'interesting' thematic maps. In this work, we develop and evaluate key criteria in automatic, annotated, map generation and experimentally validate the key features for successful representations (e.g., relevance to context, variable selection, 'interestingness' of representation and annotation quality).

In status message question asking (SMQA), members of social networking sites make use of status messages to express information needs to friends and contacts. We present findings from a laboratory study that examined 82 participants' SMQA behaviors in the broader context of online information seeking. When given the option of using a search engine and/or a social network, participants leveraged SMQA for 20% of their information needs, most often posing a question to their network in addition to issuing a query. We show the important roles played by the specificity of the information need and the perceived audience of a given network on routing decisions. We then demonstrate that routing decisions have varied effects on participants' satisfaction, information value, and trust of outcomes. In addition to highlighting the complementary advantages and disadvantages of search and SMQA, our findings suggest that search engines can better address a meaningful portion of people's information needs by integrating SMQA capabilities into their systems.

Tobler's First Law of Geography (TFL) is one of the key reasons why "spatial is special". The law, which states that "everything is related to everything else, but near things are more related than distant things", is central to the management, presentation, and analysis of geographic information. However, despite the importance of TFL, we have a limited general understanding of its domain-neutral properties. In this paper, we leverage recent advances in the natural language processing domain of semantic relatedness estimation to, for the first time, robustly evaluate the extent to which relatedness between spatial entities decreases over distance in a domain-neutral fashion. Our results reveal that, in general, TFL can indeed be considered a globally recognized domain-neutral property of geographic information but that there is a distance beyond which being nearer, on average, no longer means being more related.

Thanks to rapid advances in technologies like GPS and Wi-Fi positioning, smartphone users are able to determine their location almost everywhere they go. This is not true, however, of people who are traveling in underground public transportation networks, one of the few types of high-traffic areas where smartphones do not have access to accurate position information. In this paper, we introduce the problem of underground transport positioning on smartphones and present SubwayPS, an accelerometer-based positioning technique that allows smartphones to determine their location substantially better than baseline approaches, even deep beneath city streets. We highlight several immediate applications of positioning in subway networks in domains ranging from mobile advertising to mobile maps and present MetroNavigator, a proof-of-concept smartphone and smartwatch app that notifies users of upcoming points-of-interest and alerts them when it is time to get ready to exit the train.

Twitter, Facebook, Pinterest and many other online communities ask their users to populate a location field in their user profiles. The information that is entered into this field has many uses in both industry and academia, with location field data providing valuable geographic context for operators of online communities and playing key roles in numerous research projects. However, despite the importance of location field entries, we know little about how to design location fields effectively. In this paper, we report the results of the first controlled study of the design of location fields in user profiles. After presenting a survey of location field design decisions in use across many online communities, we show that certain design decisions can lead to more granular location information or a higher percentage of users that fill out the field, but that there is a trade-off between granularity and the percent of non-empty fields. We also add context to previous work that found that location fields tend to have a high rate of non-geographic information (e.g. Location: "Justin Bieber's Heart"), showing that this result may be site-specific rather than endemic to all location fields. Finally, we provide evidence that verifying users' location field entries against a database of known-valid locations can eliminate toponym (place name) ambiguity and any non-geographic location field entries while at the same time having little effect on field population rate and granularity.

Wikipedia is known for serving humans' informational needs. Over the past decade, the encyclopedic knowledge encoded in Wikipedia has also powerfully served computer systems. Leading algorithms in artificial intelligence, natural language processing, data mining, geographic information science, and many other fields analyze the text and structure of articles to build computational models of the world.

We present Omnipedia, a system that allows Wikipedia readers to gain insight from up to 25 language editions of Wikipedia simultaneously. Omnipedia highlights the similarities and differences that exist among Wikipedia language editions, and makes salient information that is unique to each language as well as that which is shared more widely. We detail solutions to numerous front-end and algorithmic challenges inherent to providing users with a multilingual Wikipedia experience. These include visualizing content in a language-neutral way and aligning data in the face of diverse information organization strategies. We present a study of Omnipedia that characterizes how people interact with information using a multilingual lens. We found that users actively sought information exclusive to unfamiliar language editions and strategically compared how language editions defined concepts. Finally, we briefly discuss how Omnipedia generalizes to other domains facing language barriers.

Exploratory search, in which a user investigates complex concepts, is cumbersome with today's search engines. We present a new exploratory search approach that generates interactive visualizations of query concepts using thematic cartography (e.g. choropleth maps, heat maps). We show how the approach can be applied broadly across both geographic and non-geographic contexts through explicit spatialization, a novel method that leverages any figure or diagram -- from a periodic table, to a parliamentary seating chart, to a world map -- as a spatial search environment. We enable this capability by introducing explanatory semantic relatedness measures. These measures extend frequently-used semantic relatedness measures to not only estimate the degree of relatedness between two concepts, but also generate human-readable explanations for their estimates by mining Wikipedia's text, hyperlinks, and category structure. We implement our approach in a system called Atlasify, evaluate its key components, and present several use cases.

Little research exists on one of the most common, oldest, and most utilized forms of online social geographic information: the 'location' field found in most virtual community user profiles. We performed the first in-depth study of user behavior with regard to the location field in Twitter user profiles. We found that 34% of users did not provide real location information, frequently incorporating fake locations or sarcastic comments that can fool traditional geographic information tools. When users did input their location, they almost never specified it at a scale any more detailed than their city. In order to determine whether or not natural user behaviors have a real effect on the 'locatability' of users, we performed a simple machine learning experiment to determine whether we can identify a user's location by only looking at what that user tweets. We found that a user's country and state can in fact be determined easily with decent accuracy, indicating that users implicitly reveal location information, with or without realizing it. Implications for location-based services and privacy are discussed.

While applications for mobile devices have become extremely important in the last few years, little public information exists on mobile application usage behavior. We describe a large-scale deployment-based research study that logged detailed application usage information from over 4,100 users of Android-powered mobile devices. We present two types of results from analyzing this data: basic descriptive statistics and contextual descriptive statistics. In the case of the former, we find that the average session with an application lasts less than a minute, even though users spend almost an hour a day using their phones. Our contextual findings include those related to time of day and location. For instance, we show that news applications are most popular in the morning and games are at night, but communication applications dominate through most of the day. We also find that despite the variety of apps available, communication applications are almost always the first used upon a device's waking from sleep. In addition, we discuss the notion of a virtual application sensor, which we used to collect the data.

This study explores language's fragmenting effect on user-generated content by examining the diversity of knowledge representations across 25 different Wikipedia language editions. This diversity is measured at two levels: the concepts that are included in each edition and the ways in which these concepts are described. We demonstrate that the diversity present is greater than has been presumed in the literature and has a significant influence on applications that use Wikipedia as a source of world knowledge. We close by explicating how knowledge diversity can be beneficially leveraged to create "culturally-aware applications" and "hyperlingual applications".

The "localness" of participation in repositories of user-generated content (UGC) with geospatial components has been cited as one of UGC's greatest benefits. However, the degree of localness in major UGC repositories such as Flickr and Wikipedia has never been examined. We show that over 50 percent of Flickr users contribute local information on average, and over 45 percent of Flickr photos are local to the photographer. Across four language editions of Wikipedia, however, we find that participation is less local. We introduce the spatial content production model (SCPM) as a possible factor in the localness of UGC, and discuss other theoretical and applied implications.

Virtual globes have progressed from little-known technology to broadly popular software in a mere few years. We investigated this phenomenon through a survey and discovered that, while virtual globes are en vogue, their use is restricted to a small set of tasks so simple that they do not involve any spatial thinking. Spatial thinking requires that users ask "what is where" and "why"; the most common virtual globe tasks only include the "what". Based on the results of this survey, we have developed a multi-touch virtual globe derived from an adapted virtual globe paradigm designed to widen the potential uses of the technology by helping its users to inquire about both the "what is where" and "why" of spatial distribution. We do not seek to provide users with full GIS (geographic information system) functionality, but rather we aim to facilitate the asking and answering of simple "why" questions about general topics that appeal to a wide virtual globe user base.