We present a case study of productive flyby users (PFB users) on a recommendation website.These users exhibit counterintuitive behavior: they input a large amount of data during their first visit but never return.This phenomenon can have both positive and negative impacts on the system.On the positive side, their high productivity contributes a substantial amount of data.On the negative side, they may input inappropriate ratings that violate the assumptions of recommendation algorithms, potentially undermining system performance.To better understand the nature and causes of this behavior, we investigated their motivations, expectations, reasons for leaving, and the potential risks associated with their ratings using a mixed-methods approach.Specifically, we conducted interviews with 11 users, surveyed 41 users, and analyzed the impact of 1,000 PFB users on the performance of recommendation algorithms for regular users.Our findings revealed diverse motivations among PFB users.Some engaged with the system merely to pass the time, while others had unrealistic expectations of the recommender system.Regarding rating quality, 27% of surveyed users admitted to rating movies they had not seen, citing reasons such as browsing too quickly or attempting to manipulate the algorithm.Notably, users who reported leaving because they were "just killing time and forgot about the website" were the most likely to rate unseen movies.Overall, PFB users significantly influence recommendation algorithms and their performance for regular users.While some subgroups negatively affect prediction accuracy, others provide

An increasingly important aspect of designing recommender systems involves considering how recommendations will influence consumer choices. This paper addresses this issue by introducing a method for collecting user beliefs about un-experienced goods – a critical predictor of choice behavior. We implemented this method on the MovieLens platform, resulting in a rich dataset that combines user ratings, beliefs, and observed recommendations. We document challenges to such data collection, including selection bias in response and limited coverage of the product space. This unique resource empowers researchers to delve deeper into user behavior and analyze user choices absent recommendations, measure the effectiveness of recommendations, and prototype algorithms that leverage user belief data, ultimately leading to more impactful recommender systems. The dataset can be found at https://grouplens.org/datasets/movielens/ml_belief_2024/.

Recommender systems often struggle to strike a balance between matching users’ tastes and providing unexpected recommendations. When recommendations are too narrow and fail to cover the full range of users’ preferences, the system is perceived as useless. Conversely, when the system suggests too many items that users don’t like, it is considered impersonal or ineffective. To better understand user sentiment about the breadth of recommendations given by a movie recommender, we conducted interviews and surveys and found out that many users considered narrow recommendations to be useful, while a smaller number explicitly wanted greater breadth. Additionally, we designed and ran an online field experiment with a larger user group, evaluating two new interfaces designed to provide users with greater access to broader recommendations. We looked at user preferences and behavior for two groups of users: those with higher initial movie diversity and those with lower diversity. Among our findings, we discovered that different levels of exploration control and users’ subjective preferences on interfaces are more predictive of their satisfaction with the recommender.

Organizations often send bulk emails to employees to make them aware of policy changes, organization plans, and events. Many of these emails, however, are long digests with many separate messages that waste employees' time and reduce their awareness. This study introduces CommTool--a prototype tool to help organizational communicators better understand their emails' performance and cost. We first interviewed 5 communicators and identified the need to measure the performance of each message within bulk email. Then we iteratively designed and deployed an organizational bulk email evaluation platform (CommTool), which enables communicators to get diverse message-level metrics such as reading time, relevance rate, comments, etc. We evaluated these designs through a 2-month field deployment with 5 communicators and 149 organization employees. We found that 1) the message-level metrics, such as reading time and relevance rate, helped communicators understand their audience and design bulk emails; 2) the cost and reputation metrics did not influence the organization leaders' decisions. We summarize with suggestions on designing organizational bulk email evaluation platforms that provide message-level performance and cost information.

A single digital newsletter usually contains many messages (regions). Users’ reading time spent on, and read level (skip/skim/read-in-detail) of each message is important for platforms to understand their users’ interests, personalize their contents, and make recommendations. Based on accurate but expensive-to-collect eyetracker-recorded data, we built models that predict per-region reading time based on easy-to-collect Javascript browser tracking data.

We conduct a 6 month field experiment on a movie-recommendation platform to identify if and how recommendation systems affect consumption. We use within-consumer randomization at the good level and elicit beliefs about unconsumed goods to disentangle exposure from informational effects. We have three experimental groups: (a) control, (b) exposed, and (c) recommended + exposed goods where only goods in (c) are recommended and we elicit beliefs about goods in (b) and (c). Comparing across these treatment arms we find recommendations increase consumption beyond its role in exposing goods to consumers. We provide support for an informational mechanism: recommendations affect consumers' beliefs, which in turn explain consumption. Recommendations reduce uncertainty about goods consumers are most uncertain about and induce information acquisition. Finally, we find evidence for spatial correlation in beliefs.

Bulk email is often used in organizations to communicate "important-to-organization'' messages such as policy changes, organizational plans, and administrative updates. However, normal employees may prefer messages more relevant to their jobs or interests. Organizations face the challenge of balancing prioritizing the messages they prefer employees to know (tactical goals) while maintaining employees' positive experiences with these bulk emails, then they continue to read these emails in the future (strategic goals). Could personalization help organizations achieve these tactical and strategic goals? In an 8-week field experiment with a university newsletter, we implemented a 4x5x5 factorial design on personalizing subject lines, top news, and message order based on both the employees' and the organization's preferences. We measured these designs' influences on the open/interest/recognition/read-in-detail rate of the whole newsletter and the single messages within it. We found that ''important-to-organization'' messages only got higher recognition rates when being put on subject lines / top news (tactical goal). Mixing them with employee-preferred messages in top news did not bring further improvement to their own recognition rates but could improve the whole newsletter's recognition rate. Only when the top news solely contained the employee-preferred messages were the employees slightly more interested in the newsletter (strategic goal). We further analyze on which topics the employees and the organization's preferences conflicted. Finally, we discuss the design suggestions for organizational bulk email.

Structured data peer production (SDPP) platforms like Wikidata play an important role in knowledge production. Compared to traditional peer production platforms like Wikipedia, Wikidata data is more structured and intended to be used by machines, not (directly) by people; end-user interactions with Wikidata often happen through intermediary "invisible machines." Given this distinction, we wanted to understand Wikidata contributor motivations and how they are affected by usage invisibility caused by the machine intermediaries. Through an inductive thematic analysis of 15 interviews, we find that: (i) Wikidata editors take on two archetypes---Architects who define the ontological infrastructure of Wikidata, and Masons who build the database through data entry and editing; (ii) the structured nature of Wikidata reveals novel editor motivations, such as an innate drive for organizational work; (iii) most Wikidata editors have little understanding of how their contributions are used, which may demotivate some. We synthesize these insights to help guide the future design of SDPP platforms in supporting the engagement of different types of editors.

Bulk email is a primary communication channel within organizations, with all-company emails and regular newsletters serving as a mechanism for making employees aware of policies and events. Ineffective communication could result in wasted employee time and a lack of compliance or awareness. Previous studies on organizational emails focused mostly on recipients. However, organizational bulk email system is a multi-stakeholder problem including recipients, communicators, and the organization itself. We studied the effectiveness, practice, and assessments of the organizational bulk email system of a large university from multi-stakeholders' perspectives. We conducted a qualitative study with the university's communicators, recipients, and managers. We delved into the organizational bulk email's distributing mechanisms of the communicators, the reading behaviors of recipients, and the perspectives on emails' values of communicators, managers, and recipients. We found that the organizational bulk email system as a whole was strained, and communicators are caught in the middle of this multi-stakeholder problem. First, though the communicators had an interest in preserving the effectiveness of channels in reaching employees, they had high-level clients whose interests might outweigh judgment about whether a message deserves widespread circulation. Second, though communicators thought they were sending important information, recipients viewed most of the organizational bulk emails as not relevant to them. Third, this disagreement was amplified by the success metric used by communicators. They viewed their bulk emails as successful if they had a high open rate. But recipients often opened and then rapidly discarded emails without reading the details. Last, while the communicators in general understood the challenge, they had a limited set of targeting and feedback tools to support their task.

Group-oriented services such as group recommendations aim to provide services for a group of users. For these applications, how to aggregate the preferences of different group members is the toughest yet most important problem. Inspired by game theory, in this paper, we propose to explore the idea of Nash equilibrium to simulate the selections of members in a group by a game process. Along this line, we first compute the preferences (group-dependent optimal selections) of each individual member in a given group scene, i.e., an equilibrium solution of this group, with the help of two pruning approaches. Then, to get the aggregated unitary preference of each group from all group members, we design a matrix factorization-based method which aggregates the preferences in latent space and estimates the final group preference in rating space. After obtaining the group preference, group-oriented services (e.g., group recommendation) can be directly provided. Finally, we construct extensive experiments on two real-world data sets from multiple aspects. The results clearly demonstrate the effectiveness of our method.