Measuring and assessing the impact and “fairness’’ of recommendation algorithms is central to responsible recommendation efforts. However, the complexity of fairness definitions and the proliferation of fairness metrics in research literature have led to a complex decision-making space. This environment makes it challenging for practitioners to operationalize and pick metrics that work within their unique context. This suggests that practitioners require more decision-making support, but it is not clear what type of support would be beneficial. We conducted a literature review of 24 papers to gather metrics introduced by the research community for measuring fairness in recommendation and ranking systems. We organized these metrics into a ‘decision-tree style’ support framework designed to help practitioners scope fairness objectives and identify fairness metrics relevant to their recommendation domain and application context. To explore the feasibility of this approach, we conducted 15 semi-structured interviews using this framework to assess which challenges practitioners may face when scoping fairness objectives and metrics for their system, and which further support may be needed beyond such tools.

Calls to action to implement evaluation of fairness and bias into industry systems are increasing at a rapid rate. The research community has attempted to meet these demands by producing ethical principles and guidelines for AI, but few of these documents provide guidance on how to implement these principles in real world settings. Without readily available standardized and practice-tested approaches for evaluating fairness in recommendation systems, industry practitioners, who are often not experts, may easily run into challenges or implement metrics that are potentially poorly suited to their specific applications. When evaluating recommendations, practitioners are well aware they should evaluate their systems for unintended algorithmic harms, but the most important, and unanswered question, is how? In this talk, we will present practical challenges we encountered in addressing algorithmic responsibility in recommendation systems, which also present research opportunities for the RecSys community. This talk will focus on the steps that need to happen before bias mitigation can even begin.

Many free mobile applications are supported by advertising. Ads can greatly affect user perceptions and behavior. In mobile apps, ads often follow a “native” format: they are designed to conform in both format and style to the actual content and context of the application. Clicking on the ad leads users to a second destination, outside of the hosting app, where the unified experience provided by native ads within the app is not necessarily reflected by the landing page the user arrives at. Little is known about whether and how this type of mobile ads is impacting user experience. In this paper, we use both quantitative and qualitative methods to study the impact of two design decisions for the landing page of a native ad on the user experience: (i) native ad style (following the style of the application) versus a non‐native ad style; and (ii) pages with multimedia versus static pages. We found considerable variability in terms of user experience with mobile ad landing pages when varying presentation style and multimedia usage, especially interaction between presence of video and ad style (native or non‐native). We also discuss insights and recommendations for improving the user experience with mobile native ads.

Geotagged data can be used to describe regions in the world and discover local themes. However, not all data produced within a region is necessarily specifically descriptive of that area. To surface the content that is characteristic for a region, we present the geographical hierarchy model (GHM), a probabilistic model based on the assumption that data observed in a region is a random mixture of content that pertains to different levels of a hierarchy. We apply the GHM to a dataset of 8 million Flickr photos in order to discriminate between content (i.e. tags) that specifically characterizes a region (e.g. neighborhood) and content that characterizes surrounding areas or more general themes. Knowledge of the discriminative and non-discriminative terms used throughout the hierarchy enables us to quantify the uniqueness of a given region and to compare similar but distant regions. Our evaluation demonstrates that our model improves upon traditional Naive Bayes classification by 47% and hierarchical TF-IDF by 27%. We further highlight the differences and commonalities with human reasoning about what is locally characteristic for a neighborhood, distilled from ten interviews and a survey that covered themes such as time, events, and prior regional knowledge.

Mobile, local recommendations are on the rise. Surprisingly however, research addressing user perceptions of local recommendations and local differences when interacting with such recommendation services is yet scarce. Location-based recommendation services are mostly evaluated from a `recommendation systems' standpoint, with limited experiential insights from users and limited focus on local differences that may apply. This workshop focuses on the local, personal user experience, and provides a forum to exchange experiences, insights and strategies in personalizing local mobile applications and generating local recommendations that fit local user needs.

State of the art web search systems enable aggregation of information from many sources. Users are challenged to assess the reliability of information from different sources. We report on an empirical user study on the effect of displaying credibility ratings of multiple cultural heritage sources (e.g. museum websites, art blogs) on users' search performance and selection. The results of our online interactive study (N=122) show that when explicitly presenting these ratings, people become significantly more confident in their selection of information from aggregated results.