The latitude of acceptance, or the range of recommendations users find acceptable, plays a critical role in recommender system effectiveness, yet how these acceptance boundaries evolve through user interaction remains poorly understood. This study investigates preference dynamics in gaming recommender systems with experienced players. We developed Pesto, an interactive tool that simulates Pokémon battles and visualizes how users’ latitude of acceptance shifts as they engage with different recommendations. Through a controlled within-user study with experienced players (N=10), we examine how exposure to varied recommendations influences the evolution of users’ acceptance latitude. Our study provides preliminary evidence that this approach can increase item acceptance without sacrificing user satisfaction. We also find a persistent upper bound on maximally acceptable recommendations across all participants. This work further suggests several concrete design implications based on a novel interactive visualization.

Explanations in recommender systems are important for enhancing transparency, trust, and system understanding. However, as research interest in interactive and chat-based explanations grows, the absence of standardized, reusable interface components limits their systematic evaluation. We introduce the Framework for Interactive Recommender Explanations (FIRE) to bridge this gap. FIRE provides an open-source library of modular explanation styles with varying levels of interactivity — spanning from static lists and draggable charts to conversational agents. To test these modalities in online user studies, FIRE integrates a supporting serverless experiment engine equipped with dynamic routing, crowdsourcing integration, and complete interaction logging. Currently configured for group recommenders but built for broader applicability, FIRE helps to standardize the evaluation of the explanation layer, thereby driving reproducible experimental design.

As algorithmic decision-making becomes more prevalent, ensuring these systems align with human perceptions of fairness becomes critical. While user studies usually can provide ground truth for fairness assessments, the emergence of Large Language Models (LLMs) as judges offers a potential path toward scalable and real-time evaluation. This study investigates whether LLMs can accurately replicate human fairness distributions using a dataset of human evaluations regarding group recommendation fairness. We evaluate the performance of base models as well as a fine-tuned model against human ground truth. Our findings reveal that base models often produce biased judgments that fail to capture the variety of human responses. Additionally, our results demonstrate that fine-tuning LLMs on human judgment data significantly improves their ability to mirror human fairness assessments on unseen data. This work highlights the limitations of off-the-shelf models and the promise of fine-tuning for automating algorithmic fairness evaluation. Our methodology can be extended beyond fairness to evaluate relevant subjective judgments such as user satisfaction to ground and adapt algorithmic outcomes.

Large vision-language models (VLMs) enable intuitive visual search using natural language queries. However, improving their performance often requires fine-tuning and scaling to larger model variants. In this work, we propose a mechanism inspired by traditional text-based search to improve retrieval performance at inference time: relevance feedback. While relevance feedback can serve as an alternative to fine-tuning, its model-agnostic design also enables use with fine-tuned VLMs. Specifically, we introduce and evaluate four feedback strategies for VLM-based retrieval. First, we revise classical pseudo-relevance feedback (PRF), which refines query embeddings based on top-ranked results. To address its limitations, we propose generative relevance feedback (GRF), which uses synthetic captions for query refinement. Furthermore, we introduce an attentive feedback summarizer (AFS), a custom transformer-based model that integrates multimodal fine-grained features from relevant items. Finally, we simulate explicit feedback using ground-truth captions as an upper-bound baseline. Experiments on Flickr30k and COCO with the VLM backbones show that GRF, AFS, and explicit feedback improve retrieval performance by 3–5% in MRR@5 for smaller VLMs, and 1–3% for larger ones, compared to retrieval with no feedback. Moreover, AFS, similarly to explicit feedback, mitigates query drift and is more robust than GRF in iterative, multi-turn retrieval settings. Our findings demonstrate that relevance feedback can consistently enhance retrieval across VLMs and open up opportunities for interactive and adaptive visual search.

Social choice-based aggregation strategies are often used in group recommender systems to aggregate individual preferences or recommendations. However, previous works evaluating group recommenders with user studies found that the diversity of the group members’ preferences impacts the effectiveness of the strategies. In this paper, we highlight and address the methodological limitations of those previous works. Specifically, the methodologies we introduce demonstrate the following three novelties: 1) We evaluated the strategies from the viewpoint of an “internal evaluator”; 2) We introduced a novel methodology for modeling a fictional but realistic group with specific preference profiles for the group members, defining scenarios with concrete users and items, that are still mapped to specific group configurations; 3) We evaluated the understanding of the participants, by measuring how well they can successfully apply the aggregation strategy to a new scenario. To do this we performed a randomized controlled trial (n=444) using a mixed design with two between-subject factors (the used aggregation strategy and the presented explanation type ), and a within-subject factor (the group configuration ). Our results, with friend groups, showed significant differences in the effectiveness of the aggregation strategies depending on the specific group configuration (i.e., depending on the internal diversity of group members’ preferences), with noticeable differences between evaluations of what is good for the group – external evaluation – and what is good for the participant – internal evaluation . We conclude with methodological implications for group recommender systems.

In this paper, I describe lessons learned in nearly 20 years of evaluating explanation interfaces with people. In my work, I have observed that tailoring the explanation to the user (e.g., a domain expert or a layperson) and task (e.g., decision support or model improvement), or context (e.g., under time pressure) is necessary for meaningful assessment of explanation quality (e.g., correct decisions or better understanding). Learning from trends in empirical research methods (natural language processing, information retrieval, and machine learning), I further argue that this is an issue for both human-computer interaction and machine learning. Real-world factors influencing the performance of systems are at best implicitly encoded when inferring probabilities from observations, and at worst, no longer applicable to the settings for which they are used. In seeking a balanced approach between this reality and a pragmatic, data-driven and (by necessity) reductionist approach, I make constructive suggestions for evaluating the quality of explainable artificial intelligence.

In the absence of interaction history, image recommendations often depend on content-based approaches. Prompted by user queries in natural language, such systems rank items based on the similarity between textual and visual features. However, these approaches typically rely on static queries and do not offer alternative feedback mechanisms. In this paper, we present VisualReF: an interactive image retrieval prototype that introduces visual relevance feedback through fine-grained user annotations. Built on vision-language models (VLMs) for retrieval, our system allows users to label relevant and irrelevant regions in retrieved images. These regions are captioned using a generative vision-language model to refine the query vector. Our work bridges the gap between conventional static image retrieval and interactive, user-guided search by introducing visual relevance feedback. Finally, our prototype contributes to the field of visual recommendation by empowering researchers with practical tools for: (i) collecting region-level visual relevance signals from users, (ii) supporting integration of human feedback into interactive search pipelines, and (iii) explaining how the relevance feedback model perceives user input.

Large Language Models (LLMs) are increasingly being implemented as joint decision-makers and explanation generators for Group Recommender Systems (GRS). In this paper, we evaluate these recommendations and explanations by comparing them to social choice-based aggregation strategies. Our results indicate that LLM-generated recommendations often resembled those produced by Additive Utilitarian (ADD) aggregation. However, the explanations typically referred to averaging ratings (resembling but not identical to ADD aggregation). Group structure, uniform or divergent, did not impact the recommendations. Furthermore, LLMs regularly claimed additional criteria such as user or item similarity, diversity, or used undefined popularity metrics or thresholds. Our findings have important implications for LLMs in the GRS pipeline as well as standard aggregation strategies. Additional criteria in explanations were dependent on the number of ratings in the group scenario, indicating potential inefficiency of standard aggregation methods at larger item set sizes. Additionally, inconsistent and ambiguous explanations undermine transparency and explainability, which are key motivations behind the use of LLMs for GRS.

Group Recommender Systems (GRS) employing social choice-based aggregation strategies have previously been explored in terms of perceived consensus, fairness, and satisfaction. At the same time, the impact of textual explanations has been examined, but the results suggest a low effectiveness of these explanations. However, user understanding remains fairly unexplored, even if it can contribute positively to transparent GRS. This is particularly interesting to study in more complex or potentially unfair scenarios when user preferences diverge, such as in a minority scenario (where group members have similar preferences, except for a single member in a minority position). In this paper, we analyzed the impact of different types of explanations on user understanding of group recommendations. We present a randomized controlled trial (n = 271) using two between-subject factors: (i) the aggregation strategy (additive, least misery, and approval voting), and (ii) the modality of explanation (no explanation, textual explanation, or multimodal explanation). We measured both subjective (self-perceived by the user) and objective understanding (performance on model simulation, counterfactuals and error detection). In line with recent findings on explanations for machine learning models, our results indicate that more detailed explanations, whether textual or multimodal, did not increase subjective or objective understanding. However, we did find a significant effect of aggregation strategies on both subjective and objective understanding. These results imply that when constructing GRS, practitioners need to consider that the choice of aggregation strategy can influence the understanding of users. Post-hoc analysis also suggests that there is value in analyzing performance on different tasks, rather than through a single aggregated metric of understanding.

Information access systems, such as Google News or YouTube, increasingly employ algorithms to rank diverse content such as music, recipes, and news articles. Acknowledging the influential role of these algorithms as gatekeepers to online content, the research community is increasingly exploring ‘beyond-accuracy’ metrics. However, deciding what norms and values are relevant and should be prioritized when designing and evaluating information access systems is a challenging task. This tutorial aims to cultivate normative thinking and decision-making in the design and evaluation of information access systems. The tutorial comprises two key components. The first part involves a lecture on the foundational principles of normative thinking, emphasizing the importance of reflecting on the desired state of a system rather than its current state. The second part is an interactive session where participants engage in group discussions, applying normative thinking to a specific use case. Participants analyze the system’s usage, stakeholders, and relevant norms and values and address potential conflicts between stakeholders and/or values. Through a point-allocation exercise, participants represent stakeholders and advocate for specific values, fostering a deeper understanding of normative decision-making in the context of information access systems.

Recommender systems are among the most widely used applications of artificial intelligence. Their use can have far-reaching consequences for users, stakeholders, and society at large. In this second edition of the NORMalize workshop, we once again seek to advance the research agenda of normative thinking, considering the norms and values that underpin recommender systems, as well as to introduce the concept to a broader audience. We aim to bring together a growing community of researchers and practitioners across disciplines who want to think about the norms and values that should be considered in the design and evaluation of recommender systems, and to further educate them on how to reflect on, prioritise, and operationalise such norms and values. NORMalize 2024 is a half-day workshop consisting of a combination of paper presentations and an interactive session, building upon its successful full-day run last year at RecSys’23.

When people use web search engines to find information on debated topics, the search results they encounter can influence opinion formation and practical decision-making with potentially far-reaching consequences for the individual and society. However, current web search engines lack support for information-seeking strategies that enable responsible opinion formation, e.g., by mitigating confirmation bias and motivating engagement with diverse viewpoints. We conducted two preregistered user studies to test the benefits and risks of an intervention aimed at confirmation bias mitigation. In the first study, we tested the effect of warning labels, warning of the risk of confirmation bias, combined with obfuscations, hiding selected search results per default. We observed that obfuscations with warning labels effectively reduce engagement with search results. These initial findings did not allow conclusions about the extent to which the reduced engagement was caused by the warning label (reflective nudging element) versus the obfuscation (automatic nudging element). If obfuscation was the primary cause, this would raise concerns about harming user autonomy. We thus conducted a follow-up study to test the effect of warning labels and obfuscations separately. According to our findings, obfuscations run the risk of manipulating behavior instead of guiding it, while warning labels without obfuscations (purely reflective) do not exhaust processing capacities but encourage users to actively choose to decrease engagement with attitude-confirming search results. Therefore, given the risks and unclear benefits of obfuscations and potentially other automatic nudging elements to guide engagement with information, we call for prioritizing interventions that aim to enhance human cognitive skills and agency instead.

Abstract Social choice aggregation strategies have been proposed as an explainable way to generate recommendations to groups of users. However, it is not trivial to determine the best strategy to apply for a specific group. Previous work highlighted that the performance of a group recommender system is affected by the internal diversity of the group members’ preferences. However, few of them have empirically evaluated how the specific distribution of preferences in a group determines which strategy is the most effective. Furthermore, only a few studies evaluated the impact of providing explanations for the recommendations generated with social choice aggregation strategies, by evaluating explanations and aggregation strategies in a coupled way. To fill these gaps, we present two user studies (N=399 and N=288) examining the effectiveness of social choice aggregation strategies in terms of users’ fairness perception, consensus perception, and satisfaction. We study the impact of the level of (dis-)agreement within the group on the performance of these strategies. Furthermore, we investigate the added value of textual explanations of the underlying social choice aggregation strategy used to generate the recommendation. The results of both user studies show no benefits in using social choice-based explanations for group recommendations. However, we find significant differences in the effectiveness of the social choice-based aggregation strategies in both studies. Furthermore, the specific group configuration (i.e., various scenarios of internal diversity) seems to determine the most effective aggregation strategy. These results provide useful insights on how to select the appropriate aggregation strategy for a specific group based on the level of (dis-)agreement within the group members’ preferences.

Abstract When deciding where to visit next while traveling in a group, people have to make a trade-off in an interactive group recommender system between (a) disclosing their personal information to explain and support their arguments about what places to visit or to avoid (e.g., this place is too expensive for my budget) and (b) protecting their privacy by not disclosing too much. Arguably, this trade-off crucially depends on who the other group members are and how cooperative one aims to be in making the decision. This paper studies how an individual’s personality, trust in group, and general privacy concern as well as their preference scenario and the task design serve as antecedents to their trade-off between disclosure benefit and privacy risk when disclosing their personal information (e.g., their current location, financial information, etc.) in a group recommendation explanation. We aim to design a model which helps us understand the relationship between risk and benefit and their moderating factors on final information disclosure in the group. To create realistic scenarios of group decision making where users can control the amount of information disclosed, we developed . This chat-bot agent generates natural language explanations to help group members explain their arguments for suggestions to the group in the tourism domain [more specifically, the initial POI options were selected from the category of “Food” in Amsterdam (see Sect. 3.2 for the details)]. To understand the dynamics between the factors mentioned above and information disclosure, we conducted an online, between-subjects user experiment that involved 278 participants who were exposed to either a competitive task (i.e., instructed to convince the group to visit or skip a recommended place) or a cooperative task (i.e., instructed to reach a decision in the group). Results show that participants’ personality and whether their preferences align with the majority affect their general privacy concern perception. This, in turn, affects their trust in the group, which affects their perception of privacy risk and disclosure benefit when disclosing personal information in the group, which ultimately influences the amount of personal information they disclose. A surprising finding was that the effect of privacy risk on information disclosure is different for different types of tasks: privacy risk significantly impacts information disclosure when the task of finding a suitable destination is framed competitively but not when it is framed cooperatively. These findings contribute to a better understanding of the moderating factors of information disclosure in group decision making and shed new light on the role of task design on information disclosure. We conclude with design recommendations for developing explanations in group decision-making systems. Further, we propose a theory of user modeling that shows what factors need to be considered when generating such group explanations automatically.

Recent research claims that information cues and system attributes of algorithmic decision-making processes affect decision subjects’ fairness perceptions. However, little is still known about how these factors interact. This paper presents a user study (N = 267) investigating the individual and combined effects of explanations, human oversight, and contestability on informational and procedural fairness perceptions for high- and low-stakes decisions in a loan approval scenario. We find that explanations and contestability contribute to informational and procedural fairness perceptions, respectively, but we find no evidence for an effect of human oversight. Our results further show that both informational and procedural fairness perceptions contribute positively to overall fairness perceptions but we do not find an interaction effect between them. A qualitative analysis exposes tensions between information overload and understanding, human involvement and timely decision-making, and accounting for personal circumstances while maintaining procedural consistency. Our results have important design implications for algorithmic decision-making processes that meet decision subjects’ standards of justice.

One way to help users navigate debated topics online is to apply stance detection in web search. Automatically identifying whether search results are against, neutral, or in favor could facilitate diversification efforts and support interventions that aim to mitigate cognitive biases. To be truly useful in this context, however, stance detection models not only need to make accurate (cross-topic) predictions but also be sufficiently explainable to users when applied to search results – an issue that is currently unclear. This paper presents a study into the feasibility of using current stance detection approaches to assist users in their web search on debated topics. We train and evaluate 10 stance detection models using a stance-annotated data set of 1204 search results. In a preregistered user study (N = 291), we then investigate the quality of stance detection explanations created using different explainability methods and explanation visualization techniques. The models we implement predict stances of search results across topics with satisfying quality (i.e., similar to the state-of-the-art for other data types). However, our results reveal stark differences in explanation quality (i.e., as measured by users’ ability to simulate model predictions and their attitudes towards the explanations) between different models and explainability methods. A qualitative analysis of textual user feedback further reveals potential application areas, user concerns, and improvement suggestions for such explanations. Our findings have important implications for the development of user-centered solutions surrounding web search on debated topics.

A common criteria for Explainable AI (XAI) is to support users in establishing appropriate trust in the AI – rejecting advice when it is incorrect, and accepting advice when it is correct. Previous findings suggest that explanations can cause an over-reliance on AI (overly accepting advice). Explanations that evoke appropriate trust are even more challenging for decision-making tasks that are difficult for humans and AI. For this reason, we study decision-making by non-experts in the high-uncertainty domain of stock trading. We compare the effectiveness of three different explanation styles (influenced by inductive, abductive, and deductive reasoning) and the role of AI confidence in terms of a) the users’ reliance on the XAI interface elements (charts with indicators, AI prediction, explanation), b) the correctness of the decision (task performance), and c) the agreement with the AI’s prediction. In contrast to previous work, we look at interactions between different aspects of decision-making, including AI correctness, and the combined effects of AI confidence and explanations styles. Our results show that specific explanation styles (abductive and deductive) improve the user’s task performance in the case of high AI confidence compared to inductive explanations. In other words, these styles of explanations were able to invoke correct decisions (for both positive and negative decisions) when the system was certain. In such a condition, the agreement between the user’s decision and the AI prediction confirms this finding, highlighting a significant agreement increase when the AI is correct. This suggests that both explanation styles are suitable for evoking appropriate trust in a confident AI.

Recommender systems are among the most widely used applications of artificial intelligence. Since they are so widely used, it is important that we, as practitioners and researchers, think about the impact these systems may have on users, society, and other stakeholders. To that effect, the NORMalize workshop seeks to introduce normative thinking, to consider the norms and values that underpin recommender systems in the recommender systems community. The objective of NORMalize is to bring together a growing community of researchers and practitioners across disciplines who want to think about the norms and values that should be considered in the design and evaluation of recommender systems; and further educate them on how to reflect on, prioritise, and operationalise such norms and values. NORMalize offers a comprehensive program designed to cater to both the norm-curious and the norm-active. The morning session is on-site and features a lecture on normative thinking and an interactive workshop. The afternoon is a hybrid program focused on the dissemination of results. NORMalize publishes proceedings, as well as a technical report that summarises the outcomes of the interactive morning session.

Increasingly taking place in online spaces, modern political conversations are typically perceived to be unproductively affirming---siloed in so called "echo chambers" of exclusively like-minded discussants. Yet, to date we lack sufficient means to measure viewpoint diversity in conversations. To this end, in this paper, we operationalize two viewpoint metrics proposed for recommender systems and adapt them to the context of social media conversations. This is the first study to apply these two metrics (Representation and Fragmentation) to real world data and to consider the implications for online conversations specifically. We apply these measures to two topics---daylight savings time (DST), which serves as a control, and the more politically polarized topic of immigration. We find that the diversity scores for both Fragmentation and Representation are lower for immigration than for DST. Further, we find that while pro-immigrant views receive consistent pushback on the platform, anti-immigrant views largely operate within echo chambers. We observe less severe yet similar patterns for DST. Taken together, Representation and Fragmentation paint a meaningful and important new picture of viewpoint diversity.

Existing eXplainable Artificial Intelligence (XAI) techniques support people in interpreting AI advice. However, although previous work evaluates the users’ understanding of explanations, factors influencing the decision support are largely overlooked in the literature. This article addresses this gap by studying the impact of user uncertainty , AI correctness , and the interaction between AI uncertainty and explanation logic-styles for classification tasks. We conducted two separate studies: one requesting participants to recognize handwritten digits and one to classify the sentiment of reviews. To assess the decision making, we analyzed the task performance, agreement with the AI suggestion, and the user’s reliance on the XAI interface elements. Participants make their decision relying on three pieces of information in the XAI interface (image or text instance, AI prediction, and explanation). Participants were shown one explanation style (between-participants design) according to three styles of logical reasoning (inductive, deductive, and abductive). This allowed us to study how different levels of AI uncertainty influence the effectiveness of different explanation styles. The results show that user uncertainty and AI correctness on predictions significantly affected users’ classification decisions considering the analyzed metrics. In both domains (images and text), users relied mainly on the instance to decide. Users were usually overconfident about their choices, and this evidence was more pronounced for text. Furthermore, the inductive style explanations led to overreliance on the AI advice in both domains—it was the most persuasive, even when the AI was incorrect. The abductive and deductive styles have complex effects depending on the domain and the AI uncertainty levels.

Research in the area of human information interaction (HII) typically represents viewpoints on debated topics in a binary fashion, as either against or in favor of a given topic (e.g., the feminist movement). This simple taxonomy, however, greatly reduces the latent richness of viewpoints and thereby limits the potential of research and practical applications in this field. Work in the communication sciences has already demonstrated that viewpoints can be represented in much more comprehensive ways, which could enable a deeper understanding of users’ interactions with debated topics online. For instance, a viewpoint’s stance usually has a degree of strength (e.g., mild or strong), and, even if two viewpoints support or oppose something to the same degree, they may use different logics of evaluation (i.e., underlying reasons). In this paper, we draw from communication science practice to propose a novel, two-dimensional way of representing viewpoints that incorporates a viewpoint’s stance degree as well as its logic of evaluation. We show in a case study of tweets on debated topics how our proposed viewpoint label can be obtained via crowdsourcing with acceptable reliability. By analyzing the resulting data set and conducting a user study, we further show that the two-dimensional viewpoint representation we propose allows for more meaningful analyses and diversification interventions compared to current approaches. Finally, we discuss what this novel viewpoint label implies for HII research and how obtaining it may be made cheaper in the future.

Well-annotated data is a prerequisite for good Natural Language Processing models.Too often, though, annotation decisions are governed by optimizing time or annotator agreement.We make a case for nuanced efforts in an interdisciplinary setting for annotating offensive online speech.Detecting offensive content is rapidly becoming one of the most important real-world NLP tasks.However, most datasets use a single binary label, e.g., for hate or incivility, even though each concept is multi-faceted.This modeling choice severely limits nuanced insights, but also performance.We show that a more fine-grained multi-label approach to predicting incivility and hateful or intolerant content addresses both conceptual and performance issues.We release a novel dataset of over 40,000 tweets about immigration from the US and UK, annotated with six labels for different aspects of incivility and intolerance.Our dataset not only allows for a more nuanced understanding of harmful speech online, models trained on it also outperform or match performance on benchmark datasets.Warning: This paper contains examples of hateful language some readers might find offensive.

Recent research has demonstrated that cognitive biases such as the confirmation bias or the anchoring effect can negatively affect the quality of crowdsourced data. In practice, however, such biases go unnoticed unless specifically assessed or controlled for. Task requesters need to ensure that task workflow and design choices do not trigger workers’ cognitive biases. Moreover, to facilitate the reuse of crowdsourced data collections, practitioners can benefit from understanding whether and which cognitive biases may be associated with the data. To this end, we propose a 12-item checklist adapted from business psychology to combat cognitive biases in crowdsourcing. We demonstrate the practical application of this checklist in a case study on viewpoint annotations for search results. Through a retrospective analysis of relevant crowdsourcing research that has been published at HCOMP in 2018, 2019, and 2020, we show that cognitive biases may often affect crowd workers but are typically not considered as potential sources of poor data quality. The checklist we propose is a practical tool that requesters can use to improve their task designs and appropriately describe potential limitations of collected data. It contributes to a body of efforts towards making human-labeled data more reliable and reusable.

In web search on debated topics, algorithmic and cognitive biases strongly influence how users consume and process information. Recent research has shown that this can lead to a search engine manipulation effect (SEME): when search result rankings are biased towards a particular viewpoint, users tend to adopt this favored viewpoint. To better understand the mechanisms underlying SEME, we present a pre-registered, 5 x 3 factorial user study investigating whether order effects (i.e., users adopting the viewpoint pertaining to higher-ranked documents) can cause SEME. For five different debated topics, we evaluated attitude change after exposing participants with mild pre-existing attitudes to search results that were overall viewpoint-balanced but reflected one of three levels of algorithmic ranking bias. We found that attitude change did not differ across levels of ranking bias and did not vary based on individual user differences. Our results thus suggest that order effects may not be an underlying mechanism of SEME. Exploratory analyses lend support to the presence of exposure effects (i.e., users adopting the majority viewpoint among the results they examine) as a contributing factor to users' attitude change. We discuss how our findings can inform the design of user bias mitigation strategies.

Explanations can help users to better understand why items have been recommended. Additionally, explanations for group recommender systems need to consider further goals than single-user recommender systems. For example, we need to balance group members’ need for privacy with their need for transparency, since a transparent explanation might pose a privacy hazard. In an online experiment with real groups (n=114 participants: 38 groups of size 3), we seek to understand which factors influence people’s privacy concerns when a single explanation is presented to a group in the tourism domain. In particular, we study the direct effects of three factors on privacy concern: a) group members’ personality (using the ‘Big Five’ personality traits), b) specific preference scenarios (i.e., having minority or majority preferences compared to two other group members), c) the type of relationship they have in the group (i.e., loosely coupled heterogeneous, versus tightly coupled homogeneous). We find that for personality two traits, Extroversion, and Agreeableness, each significantly affects the privacy concern. Moreover, having the minority or majority preferences in the group, as well as the type of relationship people have in the group, have a strong and significant influence on participants’ privacy concern. These results suggest that explanations presented to groups need to be adapted to all three factors (personality, type of relationship, and preference scenario) when considering the privacy concern of users.

Psychological factors such aspersonality, emotions, social connections, anddecision biasescan significantly affect the outcome of a decision process. These factors are also prevalent in the existing literature related to the inclusion of psychological aspects in recommender system development.Personalityandemotionsof users have strong connections with their interests and decision-making behavior. Hence, integrating these factors into recommender systems can help to better predict users’ item preferences and increase the satisfaction with recommended items. In scenarios where decisions are made by groups (e.g., selecting a tourism destination to visit with friends),group compositionandsocial connectionsamong group members can affect the outcome of a group decision.Decision biasesoften occur in a recommendation process, since users usually apply heuristics when making a decision. These biases can result in low-quality decisions. In this article, we provide a rigorous review of existing research on the influence of the mentioned psychological factors on recommender systems. These factors are not only considered in single-user recommendation scenarios but, importantly, also in group recommendation ones, where groups of users are involved in a decision-making process. We include working examples to provide a deeper understanding of how to take into account these factors in recommendation processes. The provided examples go beyond single-user recommendation scenarios by also considering specific aspects of group recommendation settings.

For group recommendations, one objective is to recommend an ordered set of items, a top-N, to a group such that each individual recommendation is relevant for everyone. A common way to do this is to select items on which the group can agree, using so-called ‘aggregation strategies’. One weakness of these aggregation strategies is that they select items independently of each other. They therefore cannot guarantee properties such as fairness, that apply to the set of recommendations as a whole.

Video summaries or highlights are a compelling alternative for exploring and contextualizing unprecedented amounts of video material. However, the summarization process is commonly automatic, non-transparent and potentially biased towards particular aspects depicted in the original video. Therefore, our aim is to help users like archivists or collection managers to quickly understand which summaries are the most representative for an original video. In this paper, we present empirical results on the utility of different types of visual explanations to achieve transparency for end users on how representative video summaries are, with respect to the original video. We consider four types of video summary explanations, which use in different ways the concepts extracted from the original video subtitles and the video stream, and their prominence. The explanations are generated to meet target user preferences and express different dimensions of transparency: concept prominence, semantic coverage, distance and quantity of coverage. In two user studies we evaluate the utility of the visual explanations for achieving transparency for end users. Our results show that explanations representing all of the dimensions have the highest utility for transparency, and consequently, for understanding the representativeness of video summaries.

Adaptive and personalized systems have become pervasive technologies which are gradually playing an increasingly important role in our daily lives. Indeed, we are now used to interact every day with algorithms that help us in several scenarios, ranging from services that suggest us music to be listened to or movies to be watched, to personal assistants able to proactively support us in complex decision-making tasks.

Selecting reviewers for code changes is a critical step for an efficient code review process. Recent studies propose automated reviewer recommendation algorithms to support developers in this task. However, the evaluation of recommendation algorithms, when done apart from their target systems and users (i.e., code review tools and change authors), leaves out important aspects: perception of recommendations, influence of recommendations on human choices, and their effect on user experience. This study is the first to evaluate a reviewer recommender in vivo. We compare historical reviewers and recommendations for over 21,000 code reviews performed with a deployed recommender in a company environment and set out to measure the influence of recommendations on users' choices, along with other performance metrics. Having found no evidence of influence, we turn to the users of the recommender. Through interviews and a survey we find that, though perceived as relevant, reviewer recommendations rarely provide additional value for the respondents. We confirm this finding with a larger study at another company. The confirmation of this finding brings up a case for more user-centric approaches to designing and evaluating the recommenders. Finally, we investigate information needs of developers during reviewer selection and discuss promising directions for the next generation of reviewer recommendation tools. Preprint: https://doi.org/10.5281/zenodo.1404814.

Music preferences are likely to depend on contextual characteristics such as location and activity. However, most recommender systems do not allow users to adapt recommendations to their current context. We therefore built ContextPlay, a context-aware music recommender that enables user control for both contextual characteristics and music preferences. By conducting a mixed-design study (N=114) with four typical scenarios of music listening, we investigate the effect of controlling contextual characteristics in a music recommender system on four aspects: perceived quality, diversity, effectiveness, and cognitive load. Compared to our baseline which only allows to specify music preferences, having additional control for context leads to higher perceived quality and does not increase cognitive load. We also find that the contexts of mood, weather, and location tend to influence user perception of the system. Moreover, we found that users are more likely to modify contexts and their profile during relaxing activities.

Personality impacts all areas of our lives; it governs who we are and how we react to life’s challenges. Personalized systems that adapt to end users should take into account the user’s personality to perform well. Several methodologies (e.g. User-as-Wizard, indirect studies) that use personality adaptation require first for personality to be conveyed to the participant; this has few validated approaches. Furthermore, measuring personality is often time consuming, prone to response bias (e.g. using questionnaires) or data intensive (e.g. using behaviour or text mining). This paper presents a methodology for creating and validating stories to convey psychological traits and for using such stories with a personality slider scale to measure these traits. We present the validation of the scale and evaluate its reliability. To evidence the validity of the methodology, we outline studies where the stories and scale have been effectively applied (in recommender systems, intelligent tutoring systems, and persuasive systems).

Previous research has found that enabling users to control the recommendation process increases user satisfaction. However, providing additional controls also increases cognitive load, and different users have different needs for control. Therefore, in this study, we investigate the effect of two personal characteristics: musical sophistication and visual memory capacity. We designed a visual user interface, on top of a commercial music recommender, with different controls: interactions with recommendations (i.e., the output of a recommender system), the user profile (i.e., the top listened songs), and algorithm parameters (i.e., weights in an algorithm). We created eight experimental settings with combinations of these three user controls and conducted a between-subjects study (N=240), to explore the effect on cognitive load and recommendation acceptance for different personal characteristics. We found that controlling recommendations is the most favorable single control element. In addition, controlling user profile and algorithm parameters was the most beneficial setting with multiple controls. Moreover, the participants with high musical sophistication perceived recommendations to be of higher quality, which in turn lead to higher recommendation acceptance. However, we found no effect of visual working memory on either cognitive load or recommendation acceptance. This work contributes an understanding of how to design control that hits the sweet spot between the perceived quality of recommendations and acceptable cognitive load.

When recommendations become increasingly personalized, users are often presented with a narrower range of content. To mitigate this issue, diversity-enhanced user interfaces for recommender systems have in the past found to be effective in increasing overall user satisfaction with recommendations. However, users may have different requirements for diversity, and consequently different visualization requirements. In this paper, we evaluate two visual user interfaces, SimBub and ComBub, to present the diversity of a music recommender system from different perspectives. SimBub is a baseline bubble chart that shows music genres and popularity by color and size, respectively. In addition, ComBub visualizes selected audio features along the X and Y axis in a more advanced and complex visualization. Our goal is to investigate how individual traits such as musical sophistication (MS) and visual memory (VM) influence the satisfaction of the visualization for perceived music diversity, overall usability, and support to identify blind-spots. We hypothesize that music experts, or people with better visual memory, will perceive higher diversity in ComBub than SimBub. A within-subjects user study (N=83) is conducted to compare these two visualizations. Results of our study show that participants with high MS and VM tend to perceive significantly higher diversity from ComBub compared to SimBub. In contrast, participants with low MS perceived significantly higher diversity from SimBub than ComBub; however, no significant result is found for the participants with low VM. Our research findings show the necessity of considering individual traits while designing diversity-aware interfaces.

As intelligent interactive systems, recommender systems focus on determining predictions that fit the wishes and needs of users. Still, a large majority of recommender systems research focuses on accuracy criteria and much less attention is paid to how users interact with the system, and in which way the user interface has an influence on the selection behavior of the users. Consequently, it is important to look beyond algorithms. The main goals of the IntRS workshop are to analyze the impact of user interfaces and interaction design, and to explore human interaction with recommender systems from a human decision making perspective. Methodologies for evaluating these aspects are also within the scope of the workshop.

This paper presents an exploratory study of the perceptions users have of diversity and ordering in playlist recommendations. There is a match between the diversification approach used in the system, and importance that users placed on the item properties. Surprisingly, participants had no expectations of the songs being in a particular order in a playlist. We discuss possible explanations for this finding, refining the research agenda to consider which ordering choices are perceptible to users, and influence user satisfaction.

We explore how the problem of uncertainty and imprecision in natural language generation (NLG) could be addressed through the use of fuzzy sets. We propose bringing together standard empirical procedures for knowledge acquisition in NLG and computing with words/perceptions related techniques (with a special focus on linguistic description of data) to address an open challenge in NLG: the generation of geographical referring expressions. Following this methodology, we present an exploratory experiment which provides some insights about how human subjects refer to geographical expressions and discuss how the obtained results might relate to the use of fuzzy sets.

As intelligent interactive systems, recommender systems focus on determining predictions that fit the wishes and needs of users. Still, a large majority of recommender systems research focuses on accuracy criteria and much less attention is paid to how users interact with the system, and in which way the user interface has an influence on the selection behavior of the users. Consequently, it is important to look beyond algorithms. The main goals of the IntRS workshop are to analyze the impact of user interfaces and interaction design, and to explore human interaction with recommender systems. Methodologies for evaluating these aspects are also within the scope of the workshop.

Many interactive systems in today’s world can be viewed as providing advice to their users. Commercial examples include recommender systems, satellite navigation systems, intelligent personal assistants on smartphones, and automated checkout systems in supermarkets. We will call these systems that support people in making choices and decisions artificial advice givers (AAGs) : They propose and evaluate options while involving their human users in the decision-making process. This special issue addresses the challenge of improving the interaction between artificial and human agents. It answers the question of how an agent of each type (human and artificial) can influence and understand the reasoning, working models, and conclusions of the other agent by means of novel forms of interaction. To address this challenge, the articles in the special issue are organized around three themes: (a) human factors to consider when designing interactions with AAGs (e.g., over- and under-reliance, overestimation of the system’s capabilities), (b) methods for supporting interaction with AAGs (e.g., natural language, visualization, and argumentation), and (c) considerations for evaluating AAGs (both criteria and methodology for applying them).

It has been claimed that computational models of argumentation provide support for complex decision making activities in part due to the close alignment between their semantics and human intuition. In this paper we assess this claim by means of an experiment: people's evaluation of formal arguments — presented in plain English — is compared to the conclusions obtained from argumentation semantics. Our results show a correspondence between the acceptability of arguments by human subjects and the justification status prescribed by the formal theory in the majority of the cases. However, post-hoc analyses show that there are some significant deviations, which appear to arise from implicit knowledge regarding the domains in which evaluation took place. We argue that in order to create argumentation systems, designers must take implicit domain specific knowledge into account.

As an interactive intelligent system, recommender systems are developed to give predictions that match users preferences. Since the emergence of recommender systems, a large majority of research focuses on objective accuracy criteria and less attention has been paid to how users interact with the system and the efficacy of interface designs from the end-user perspective. The field has reached a point where it is ready to look beyond algorithms, into users interactions, decision making processes and overall experience. Accordingly, the goals of this workshop (Int[email protected]) are to explore the human aspects of recommender systems, with a particular focus on the impact of interfaces and interaction design on decision-making and user experiences with recommender systems, and to explore methodologies to evaluate these human aspects of the recommendation process that go beyond traditional automated approaches.

No abstract available.

Currently available commercial Augmentative and Alternative Communication (AAC) technology makes little use of computing power to improve the access to words and phrases for personal narrative, an essential part of social interaction. In this paper, we describe the development and evaluation of a mobile phone application to enable data collection for a personal narrative system for children with severe speech and physical impairments (SSPI). Based on user feedback from the previous project "How was School today?" we developed a modular system where school staff can use a mobile phone to track interaction with people and objects and user location at school. The phone also allows taking digital photographs and recording voice message sets by both school staff and parents/carers at home. These sets can be played back by the child for immediate narrative sharing similar to established AAC device interaction using sequential voice recorders. The mobile phone sends all the gathered data to a remote server. The data can then be used for automatic narrative generation on the child's PC based communication aid. Early results from the ongoing evaluation of the application in a special school with two participants and school staff show that staff were able to track interactions, record voice messages and take photographs. Location tracking was less successful, but was supplemented by timetable information. The participating children were able to play back voice messages and show photographs on the mobile phone for interactive narrative sharing using both direct and switch activated playback options.

Recent research in Augmented and Alternative Communication (AAC) has begun to make use of Natural Language Generation (NLG) techniques. This creates an opportunity for constructing stories from sensor data, akin to existing work in life-logging. This paper examines the potential of using NLG to merge the AAC and life-logging domains. It proposes a four stage hierarchy that categorises levels of complexity of output text. It formulates a key subproblem of clustering sensor data into narrative events and describes three potential approaches for resolving this subproblem.

This paper presents a field study of a framework for personalized mobile recommendations in the tourism domain, of sight-seeing Points of Interest (POI). We evaluate the effectiveness, satisfaction and divergence from popularity of a knowledge-based personalization strategy comparing it to recommending most popular sites. We found that participants visited more of the recommended POIs for lists with popular but non-personalized recommendations. In contrast, the personalized recommendations led participants to visit more POIs overall and visit places "off the beaten track". The level of satisfaction between the two conditions was comparable and high, suggesting that our participants were just as happy with the rarer, "off the beaten track" recommendations and their overall experience. We conclude that personalized recommendations set tourists into a discovery mode with an increased chance for serendipitous findings, in particular for returning tourists.

A common approach to designing Recommender Systems (RS) consists of asking users to explicitly rate items in order to collect feedback about their preferences. However, users have been shown to be inconsistent and to introduce a non-negligible amount of natural noise in their ratings that affects the accuracy of the predictions. In this paper, we present a novel approach to improve RS accuracy by reducing the natural noise in the input data via a preprocessing step. In order to quantitatively understand the impact of natural noise, we first analyze the response of common recommendation algorithms to this noise. Next, we propose a novel algorithm to denoise existing datasets by means of re-rating: i.e. by asking users to rate previously rated items again. This denoising step yields very significant accuracy improvements. However, re-rating all items in the original dataset is unpractical. Therefore, we study the accuracy gains obtained when re-rating only some of the ratings.In particular, we propose two partial denoising strategies: data and user-dependent denoising. Finally, we compare the value of adding a rating of an unseen item vs. re-rating an item. We conclude with a proposal for RS to improve the quality of their user data and hence their accuracy: asking users to re-rate items might, in some circumstances, be more beneficial than asking users to rate unseen items.

This thesis focuses on explanations of recommendations. Explanations can have many advantages, from inspiring user trust to helping users make good decisions. We have identified seven different aims of explanations, and in this thesis we will consider how explanations can be optimized for some of these aims. We will consider both an explanation's content and its presentation. As a domain, we are currently investigating explanations for a movie recommender, and developing a prototype system. This paper summarizes the goals of the thesis, the methodology we are using, the work done so far and our intended future work.

This paper characterizes general properties of useful, or Effective, explanations of recommendations. It describes a methodology based on focus groups, in which we elicit what helps moviegoers decide whether or not they would like a movie. Our results highlight the importance of personalizing explanations to the individual user, as well as considering the source of recommendations, user mood, the effects of group viewing, and the effect of explanations on user expectations.