Measuring and assessing the impact and “fairness’’ of recommendation algorithms is central to responsible recommendation efforts. However, the complexity of fairness definitions and the proliferation of fairness metrics in research literature have led to a complex decision-making space. This environment makes it challenging for practitioners to operationalize and pick metrics that work within their unique context. This suggests that practitioners require more decision-making support, but it is not clear what type of support would be beneficial. We conducted a literature review of 24 papers to gather metrics introduced by the research community for measuring fairness in recommendation and ranking systems. We organized these metrics into a ‘decision-tree style’ support framework designed to help practitioners scope fairness objectives and identify fairness metrics relevant to their recommendation domain and application context. To explore the feasibility of this approach, we conducted 15 semi-structured interviews using this framework to assess which challenges practitioners may face when scoping fairness objectives and metrics for their system, and which further support may be needed beyond such tools.

Calls to action to implement evaluation of fairness and bias into industry systems are increasing at a rapid rate. The research community has attempted to meet these demands by producing ethical principles and guidelines for AI, but few of these documents provide guidance on how to implement these principles in real world settings. Without readily available standardized and practice-tested approaches for evaluating fairness in recommendation systems, industry practitioners, who are often not experts, may easily run into challenges or implement metrics that are potentially poorly suited to their specific applications. When evaluating recommendations, practitioners are well aware they should evaluate their systems for unintended algorithmic harms, but the most important, and unanswered question, is how? In this talk, we will present practical challenges we encountered in addressing algorithmic responsibility in recommendation systems, which also present research opportunities for the RecSys community. This talk will focus on the steps that need to happen before bias mitigation can even begin.

In order to support fairness-forward thinking by machine learning (ML) practitioners, fairness researchers have created toolkits that aim to transform state-of-the-art research contributions into easily-accessible APIs. Despite these efforts, recent research indicates a disconnect between the needs of practitioners and the tools offered by fairness research. By engaging 20 ML practitioners in a simulated scenario in which they utilize fairness toolkits to make critical decisions, this work aims to utilize practitioner feedback to inform recommendations for the design and creation of fair ML toolkits. Through the use of survey and interview data, our results indicate that though fair ML toolkits are incredibly impactful on users’ decision-making, there is much to be desired in the design and demonstration of fairness results. To support the future development and evaluation of toolkits, this work offers a rubric that can be used to identify critical components of Fair ML toolkits.

Large and ever-evolving technology companies continue to invest more time and resources to incorporate responsible Artificial Intelligence (AI) into production-ready systems to increase algorithmic accountability. This paper examines and seeks to offer a framework for analyzing how organizational culture and structure impact the effectiveness of responsible AI initiatives in practice. We present the results of semi-structured qualitative interviews with practitioners working in industry, investigating common challenges, ethical tensions, and effective enablers for responsible AI initiatives. Focusing on major companies developing or utilizing AI, we have mapped what organizational structures currently support or hinder responsible AI initiatives, what aspirational future processes and structures would best enable effective initiatives, and what key elements comprise the transition from current work practices to the aspirational future.

It can be difficult for user researchers to explore how people might interact with interactive systems in everyday contexts; time and space limitations make it hard to be present everywhere that technology is used. Digital music services are one domain where designing for context is important given the myriad places people listen to music. One novel method to help design researchers embed themselves in everyday contexts is through remote-controlled speech agents. This paper describes a practitioner-centered case study of music service interaction researchers using a remote-controlled speech agent, called DJ Bot, to explore people's music interaction in the car and the home. DJ Bot allowed the team to conduct remote user research and contextual inquiry and to quickly explore new interactions. However, challenges using a remote speech-agent arose when adapting DJ Bot from the constrained environment of the car to the unconstrained home environment.

It remains unknown whether personalized recommendations increase or decrease the diversity of content people consume. We present results from a randomized field experiment on Spotify testing the effect of personalized recommendations on consumption diversity. In the experiment, both control and treatment users were given podcast recommendations, with the sole aim of increasing podcast consumption. Treatment users' recommendations were personalized based on their music listening history, whereas control users were recommended popular podcasts among users in their demographic group. We find that, on average, the treatment increased podcast streams by 28.90%. However, the treatment also decreased the average individual-level diversity of podcast streams by 11.51%, and increased the aggregate diversity of podcast streams by 5.96%, indicating that personalized recommendations have the potential to create patterns of consumption that are homogenous within and diverse across users, a pattern reflecting Balkanization. Our results provide evidence of an "engagement-diversity trade-off" when recommendations are optimized solely to drive consumption: while personalized recommendations increase user engagement, they also affect the diversity of consumed content. This shift in consumption diversity can affect user retention and lifetime value, and impact the optimal strategy for content producers. We also observe evidence that our treatment affected streams from sections of Spotify's app not directly affected by the experiment, suggesting that exposure to personalized recommendations can affect the content that users consume organically. We believe these findings highlight the need for academics and practitioners to continue investing in personalization methods that explicitly take into account the diversity of content recommended.

Voice interfaces often struggle with specific types of named content. Domain-specific terminology and naming may push the bounds of standard language, especially in domains like music where artistic creativity extends beyond the music itself. Artists may name themselves with symbols (e.g. M S C RA) that most standard automatic speech recognition (ASR) systems cannot transcribe. Voice interfaces also experience difficulty surfacing content whose titles include non-standard spellings, symbols or other ASCII characters in place of English letters, or are written using a non-standard dialect. We present a generalizable method to detect content that current voice interfaces underserve by leveraging differences in engagement across input modalities. Using this detection method, we develop a typology of content types and linguistic practices that can make content hard to surface. Finally, we present a process using crowdsourced annotations to make underserved content more accessible.

Many free mobile applications are supported by advertising. Ads can greatly affect user perceptions and behavior. In mobile apps, ads often follow a “native” format: they are designed to conform in both format and style to the actual content and context of the application. Clicking on the ad leads users to a second destination, outside of the hosting app, where the unified experience provided by native ads within the app is not necessarily reflected by the landing page the user arrives at. Little is known about whether and how this type of mobile ads is impacting user experience. In this paper, we use both quantitative and qualitative methods to study the impact of two design decisions for the landing page of a native ad on the user experience: (i) native ad style (following the style of the application) versus a non‐native ad style; and (ii) pages with multimedia versus static pages. We found considerable variability in terms of user experience with mobile ad landing pages when varying presentation style and multimedia usage, especially interaction between presence of video and ad style (native or non‐native). We also discuss insights and recommendations for improving the user experience with mobile native ads.

Most devices today are developed adhering to a one-user paradigm. Yet within households, couples are often sharing devices and accounts. In this paper we take an in-depth look at sharing practices and preferences of cohabiting couples, and discuss the nuances of existing practices surrounding accounts and devices. We present a qualitative interview and diary study with ten couples, consisting of 20 individual interviews, and individual 8-day diaries. Dichotomous access models do not reflect the sharing practices of our couples; in which intent, access, and utilization all characterized sharing behaviors. We present a detailed description of the intentional and unintentional sharing practices our participants used in their day to day interactions and discuss the different challenges that particularly one type of content pose in terms of issues of privacy. We discuss implications for accounts and devices based on the ways in which content was shared and hidden among collocated couples. We provide a structured account of these sharing practices to inform the design of multi-user settings within future technologies.

Emojis are an extremely common occurrence in mobile communications, but their meaning is open to interpretation. We investigate motivations for their usage in mobile messaging in the US. This study asked 228 participants for the last time that they used one or more emojis in a conversational message, and collected that message, along with a description of the emojis' intended meaning and function. We discuss functional distinctions between: adding additional emotional or situational meaning, adjusting tone, making a message more engaging to the recipient, conversation management, and relationship maintenance. We discuss lexical placement within messages, as well as social practices. We show that the social and linguistic function of emojis are complex and varied, and that supporting emojis can facilitate important conversational functions.

Native advertising, ads that are highly cohesive with actual content in format and style, is a pervasive online trend. We report on a two-stage Mechanical Turk experiment exploring the effects of perceived quality of native ads on perceptions of site quality. First, a set of native ads was rated (N=98) for perceived annoyance and trust. Second, we compare four conditions of an aggregation of news headlines (N=237): 'no ads', 'low quality ads', 'high quality ads + content-relevant', 'high-quality ads + not content-relevant'. Our results indicate that native ads, which in isolation have been rated as high quality, could still have a negative effect on perceived site credibility and perceived site quality if they are too content-relevant. In addition to the effect of ad quality alone (e.g. non-annoying, trustworthy ads), there is an additional impact for ads that are too similar to content; avoiding confusion is important for quality perceptions.

An overwhelming variety of communication channels are available to consumers. Here, we present an overview of the aspects that need to be accounted for when intimate partners select a communication channel. We present interviews with 10 cohabiting couples (20 participants) and an 8-day diary study of communication and coordination. Using reported instances of within-couple communication, triggered by relationship-oriented or practical household needs, we identify why particular channels are chosen or sequenced. Extending media richness critiques, we identify additional factors that influence communication choice such as intimate knowledge of the others' habits, possibilities to add emotional meaning, and couples' shared needs as an identifiable unit. We also extend the notion of network effects on channel choice, and discuss the ecology of channel, networks, devices and device settings involved between partners. Finally, channel choice is not an all-or-nothing game; multiple channels can, and must, co-exist.

Geotagged data can be used to describe regions in the world and discover local themes. However, not all data produced within a region is necessarily specifically descriptive of that area. To surface the content that is characteristic for a region, we present the geographical hierarchy model (GHM), a probabilistic model based on the assumption that data observed in a region is a random mixture of content that pertains to different levels of a hierarchy. We apply the GHM to a dataset of 8 million Flickr photos in order to discriminate between content (i.e. tags) that specifically characterizes a region (e.g. neighborhood) and content that characterizes surrounding areas or more general themes. Knowledge of the discriminative and non-discriminative terms used throughout the hierarchy enables us to quantify the uniqueness of a given region and to compare similar but distant regions. Our evaluation demonstrates that our model improves upon traditional Naive Bayes classification by 47% and hierarchical TF-IDF by 27%. We further highlight the differences and commonalities with human reasoning about what is locally characteristic for a neighborhood, distilled from ten interviews and a survey that covered themes such as time, events, and prior regional knowledge.

In the 1960s, Lynch's 'The Image of the City' explored what impression US city neighborhoods left on its inhabitants. The scale of urban perception studies until recently was considerably constrained by the limited number of study participants. We here present a crowdsourcing project that aims to investigate, at scale, which visual aspects of London neighborhoods make them appear beautiful, quiet, and/or happy. We collect votes from over 3.3K individuals and translate them into quantitative measures of urban perception. In so doing, we quantify each neighborhood's aesthetic capital. By then using state-of-the-art image processing techniques, we determine visual cues that may cause a street to be perceived as being beautiful, quiet, or happy. We identify effects of color, texture and visual words. For example, the amount of greenery is the most positively associated visual cue with each of three qualities; by contrast, broad streets, fortress-like buildings, and council houses tend to be associated with the opposite qualities (ugly, noisy, and unhappy).

Online services provide a range of opportunities for understanding human behaviour through the large aggregate data sets that their operation collects. Yet the data sets they collect do not unproblematically model or mirror the world events. In this paper we use data from Foursquare, a popular location check-in service, to argue for the importance of analysing social media as a communicative rather than representational system. Drawing on logs of all Foursquare check-ins over eight weeks we highlight four features of Foursquare's use: the relationship between attendance and check-ins, event check-ins, commercial incentives to check-in, and lastly humorous check-ins These points show how large data analysis is affected by the end user uses to which social networks are put.

With the launch of 'app stores' on several mobile platforms and the great uptake of smartphones among the general population, researchers have begun utilising these distribution channels to deploy research software to large numbers of users. Previous Research In The Large workshops have sought to establish base-line practice in this area. We have seen the use of app stores as being successful as a methodology for gathering large amounts of data, leading to design implications, but we have yet to explore the full potential for this data's use and interpretation. How is it possible to leverage the practices of large-scale research, beyond the current approaches, to more directly inform future designs? We propose that the time is right to re-energise discussions on large-scale research, looking further than the basic methodological issues and assessing the potential for informing the design of new mobile software.

This paper explores the issues that arise in the context of the migration of a robotic pet between different embodiments and the associated design challenges. In the following, we describe the perceptions that a group of children have of a dinosaur character crossing the boundary between its robotic embodiment (the Pleo commercial pet), and its virtual counterpart on a mobile phone. We analyse the children's perceptions of, as well as emotional reactions to, the migration of this character, and show how seemingly subtle variations in the migration process can affect the children's perception on the character and its embodiments. Among other findings, gaps in the migration process, or perceived unresponsiveness, appeared to be accompanied by anxiety in the participating children. Based on our results, we point to yet unsolved design challenges for migration in interactions with embodied characters, and offer insights for migration implementation.

In building location-based services, it is important to present information in ways that fit with how individuals view and navigate the city. We conducted an adaptation of the 1970s Mental Maps study by Stanley Milgram in order to better understand differences in people's views of the city based on their backgrounds and technology use. We correlated data from a demographic questionnaire with the map data from our participants to perform a first-of-its-kind statistical analysis on differences in hand-drawn city maps. We describe our study, findings, and design implications for location-based services.

This paper describes a case study of the initial reactions to a transport robot running in a semi-public hospital environment. The robot was transporting goods and samples for an orthopedic department, moving within and between different departments for 13 days, and was used by the staff for five days. Based on this case, we discuss how the robot was perceived by staff and visitors and purpose an initial framework, a utopian model, describing four different perspectives; an alien, a machine, a worker and as a work partner. This has been derived from interviews, questionnaires and observation, and exemplifies different qualities that the robot was ascribed. We discuss how the perspectives may mutually co-exist and change, and are affected by time and familiarity with the robot at work.

Mobile, local recommendations are on the rise. Surprisingly however, research addressing user perceptions of local recommendations and local differences when interacting with such recommendation services is yet scarce. Location-based recommendation services are mostly evaluated from a `recommendation systems' standpoint, with limited experiential insights from users and limited focus on local differences that may apply. This workshop focuses on the local, personal user experience, and provides a forum to exchange experiences, insights and strategies in personalizing local mobile applications and generating local recommendations that fit local user needs.

This late breaking report describes two explorations of effects of using social media in human-robot interaction. The first is an exploration of how 'autonomous creatures' can use information shared via social awareness streams by implementing a Nabaztag to use information from its 'friends' on location-sharing service foursquare. The second is an informal analysis of tweets sent to an existing robot-associated twitter account as a case. We show parallels to prior research and discuss questions that these simple explorations pose for the future of robots and social media.

This work in progress discusses a persistent myth about robots, namely that 'future robots will be universal solutions', or in other words that robots should tackle many complex tasks and situations. In our approach we consider whether this is a case of posing robots as solutions to wicked problems or if robots can be considered wicked design problems in themselves. At the same time we make an argument for adopting a research through design approach. Our stance suggests that by viewing robots as composed of design materials we can sensitively address and in the long run perhaps even avoid wicked problems related to robotics.

With the proliferation of app stores and the advancement of mobile devices, research that might have only been tested with a dozen participants in the past can now be released to millions. This offers huge opportunities, but also requires adaptations of existing methods in dealing with large deployments and making sense of large data sets. This workshop provides a forum for researchers to exchange experiences and strategies for wide distribution of applications as well as gathering and analyzing large scale data sets.

Location-sharing services have a long history in research, but have only recently become available for consumers. Most popular commercial location-sharing services differ from previous research efforts in important ways: they use manual 'check-ins' to pair user location with semantically named venues rather than tracking; venues are visible to all users; location is shared with a potentially very large audience; and they employ incentives. By analysis of 20 in-depth interviews with foursquare users and 47 survey responses, we gained insight into emerging social practices surrounding location-sharing. We see a shift from privacy issues and data deluge, to more performative considerations in sharing one's location. We discuss performance aspects enabled by check-ins to public venues, and show emergent, but sometimes conflicting norms (not) to check-in.

Mobile services can provide users with information relevant to their current circumstances. Distant services in turn can acquire local information from people in an area of interest. Socially expressive agent behaviour has been suggested as a way to build reciprocal relationships and to increase user response to such requests. This between-subject, Wizard-of-Oz experiment aimed to investigate the potential of such behaviours. 44 participants performed a search task in an urgent context while being interrupted by a mobile agent that both provided and requested information. The socially expressive behaviour shown in this study did not increase compliance to requests; it instead reduced trust in provided information and compliance to warnings. It also negatively impacted the affective experience of users scoring lower on empathy as a personality trait. Inappropriate social expressiveness can have serious consequences; we here elaborate on the reasons for our negative results.

Empathy has great potential in human-robot interaction. However, the challenging nature of assessing the user's emotional state points to the importance of also understanding the effects of empathic behaviours incongruent with users' affective experience. A 3x2 between-subject video-based survey experiment (N=133) was conducted with empathic robot behaviour (empathically accurate, neutral, inaccurate) and valence of the situation (positive, negative) as dimensions. Trust decreased when empathic responses were incongruent with the affective state of the user. However, in the negative valence condition, reported perceived empathic abilities were greater when the robot responded as if the situation were positive.

This workshop aims to explore different approaches and challenges in studying playfulness as a mode of interacting with mobile technology. Researchers, designers and developers with interest in this theme are welcome to participate in a full day activity of demos, presentations and discussions. In particular, our emphasis is on how to introduce, explore and understand playful interaction in mobile applications used in the wild.

Even though adaptive (trainable) spam filters are a common example of systems that make (semi-)autonomous decisions on behalf of the user, trust in these filters has been underexplored. This paper reports a study of usage of spam filters in the daily workplace and user behaviour in training these filters (N=43). User observation, interview and survey techniques were applied to investigate attitudes towards two types of filters: a user-adaptive (trainable) and a rule-based filter. While many of our participants invested extensive effort in training their filters, training did not influence filter trust. Instead, the findings indicate that users' filter awareness and understanding seriously impacts attitudes and behaviour. Specific examples of difficulties related to awareness of filter activity and adaptivity are described showing concerns relevant to all adaptive and (semi-)autonomous systems that rely on explicit user feedback.

Despite robots' embodiment, the effect of physical contact or touch and its interaction with robots' autonomous behaviour has been a mostly overlooked aspect of human-robot interaction. This video-based, 2x2 between-subject survey experiment (N=119) found that touch and proactiveness interacted in their effects on perceived machine-likeness and dependability. Attitude towards robots in general also interacted with the effects of touch. Results show the value of further exploring the combination of physical aspects of human-robot interaction and proactiveness.

In this paper, we present ongoing research concerning the interaction between users and autonomous mobile agents in the environmental monitoring domain. The overarching project, DIADEM, deals with developing a system that detects potentially hazardous situations in populated industrial areas using input from both a distributed sensor network and humans through mobile devices. We propose a model of interaction with the gas detection system where concerned citizens communicate with a mobile agent to inform the gas monitoring system about unusual smells via their mobile phones. Next, we present a preliminary user requirements analysis based on 40 phone calls from members of the public to an environmental monitoring agency. Finally, we introduce measures to study the delicate long-term social relationship between users and the gas monitoring system.

State of the art web search systems enable aggregation of information from many sources. Users are challenged to assess the reliability of information from different sources. We report on an empirical user study on the effect of displaying credibility ratings of multiple cultural heritage sources (e.g. museum websites, art blogs) on users' search performance and selection. The results of our online interactive study (N=122) show that when explicitly presenting these ratings, people become significantly more confident in their selection of information from aggregated results.

Abstract Embodied social agents are programmed to display human‐like social behaviour to increase intuitiveness of interacting with these agents. It is yet unclear what the differences in peoples' responses are to different types of agents' social behaviours. One example is touch. Despite robots' physical embodiment and increasing autonomy, the effect of communicative touch has been a mostly overlooked aspect of human‐robot interaction. This video‐based, 2 × 2 between‐subject survey experiment (N = 119) found that the combination of touch and proactivity influenced whether people saw the robot as machine‐like and dependable. Furthermore, participants' attitude toward robots in general was found to influence perceived closeness between a human and a robot. Results show that communicative touch could be considered a more appropriate behaviour for proactive agents rather than reactive agents. Also, people that are generally more positive towards robots find robots that interact by touch less machine‐like. These effects illustrate that careful consideration is necessary when incorporating social behaviours in agents' physical interaction design. Copyright © 2009 John Wiley & Sons, Ltd.

The increasing availability of (digital) cultural heritage artefacts offers great potential for increased access to art content, but also necessitates tools to help users deal with such abundance of information. User-adaptive art recommender systems aim to present their users with art content tailored to their interests. These systems try to adapt to the user based on feedback from the user on which artworks he or she finds interesting. Users need to be able to depend on the system to competently adapt to their feedback and find the artworks that are most interesting to them. This paper investigates the influence of transparency on user trust in and acceptance of content-based recommender systems. A between-subject experiment ( N = 60) evaluated interaction with three versions of a content-based art recommender in the cultural heritage domain. This recommender system provides users with artworks that are of interest to them, based on their ratings of other artworks. Version 1 was not transparent, version 2 explained to the user why a recommendation had been made and version 3 showed a rating of how certain the system was that a recommendation would be of interest to the user. Results show that explaining to the user why a recommendation was made increased acceptance of the recommendations. Trust in the system itself was not improved by transparency. Showing how certain the system was of a recommendation did not influence trust and acceptance. A number of guidelines for design of recommender systems in the cultural heritage domain have been derived from the study’s results.