Open User Models (OUM) leverage natural language to help users scrutinize their interests, moving beyond opaque behavioral logs to establish a ground truth of user preference. Building on this foundation, we introduce an LLM-generated open profile for a live movie recommender system that presents editable, personalized interest summaries. Unlike static profiles, this design invites users to directly inspect, audit, and reflect on the system’s inferences. In an eight-week field deployment with 1,775 active users, we found that participants actively engaged in preference construction, specifically pruning inaccurate dislikes and refining nuanced tastes. Our results show that this profile auditing process encourages high-efficiency maintenance behavior, leading to significantly increased reflection activity and engagement. We advocate for a Productively Wrong design strategy, where exposing mild AI inaccuracies serves as a catalyst to stimulate user intervention, fostering transparent systems where users actively curate their digital identity.

We present a case study of productive flyby users (PFB users) on a recommendation website.These users exhibit counterintuitive behavior: they input a large amount of data during their first visit but never return.This phenomenon can have both positive and negative impacts on the system.On the positive side, their high productivity contributes a substantial amount of data.On the negative side, they may input inappropriate ratings that violate the assumptions of recommendation algorithms, potentially undermining system performance.To better understand the nature and causes of this behavior, we investigated their motivations, expectations, reasons for leaving, and the potential risks associated with their ratings using a mixed-methods approach.Specifically, we conducted interviews with 11 users, surveyed 41 users, and analyzed the impact of 1,000 PFB users on the performance of recommendation algorithms for regular users.Our findings revealed diverse motivations among PFB users.Some engaged with the system merely to pass the time, while others had unrealistic expectations of the recommender system.Regarding rating quality, 27% of surveyed users admitted to rating movies they had not seen, citing reasons such as browsing too quickly or attempting to manipulate the algorithm.Notably, users who reported leaving because they were "just killing time and forgot about the website" were the most likely to rate unseen movies.Overall, PFB users significantly influence recommendation algorithms and their performance for regular users.While some subgroups negatively affect prediction accuracy, others provide

This is a dissertation plan built around human-centered empirical experiments evaluating recommender systems (RecSys). We see this as an important research theme since many AI-based RecSys algorithmic studies lack real human assessment. Therefore, we do not know how they work in the wild that only human experiments can tell us. We split this extended abstract into two parts – 1) A series of individual studies focusing on open questions about different human values or recommendation algorithms. Our completed works include user control over content diversity, user appreciation on DL-RecSys algorithms, and human-LLMRec interaction study. We also propose three future works to understand news recommendation depolarization, personalized news podcast, and interactive user representation; 2) An experimentation infrastructure named POPROX. As a personalized news recommendation platform, it aims to support the longitudinal study needs from the general AI and RecSys research community.

Recommender systems often struggle to strike a balance between matching users’ tastes and providing unexpected recommendations. When recommendations are too narrow and fail to cover the full range of users’ preferences, the system is perceived as useless. Conversely, when the system suggests too many items that users don’t like, it is considered impersonal or ineffective. To better understand user sentiment about the breadth of recommendations given by a movie recommender, we conducted interviews and surveys and found out that many users considered narrow recommendations to be useful, while a smaller number explicitly wanted greater breadth. Additionally, we designed and ran an online field experiment with a larger user group, evaluating two new interfaces designed to provide users with greater access to broader recommendations. We looked at user preferences and behavior for two groups of users: those with higher initial movie diversity and those with lower diversity. Among our findings, we discovered that different levels of exploration control and users’ subjective preferences on interfaces are more predictive of their satisfaction with the recommender.

Organizations often send bulk emails to employees to make them aware of policy changes, organization plans, and events. Many of these emails, however, are long digests with many separate messages that waste employees' time and reduce their awareness. This study introduces CommTool--a prototype tool to help organizational communicators better understand their emails' performance and cost. We first interviewed 5 communicators and identified the need to measure the performance of each message within bulk email. Then we iteratively designed and deployed an organizational bulk email evaluation platform (CommTool), which enables communicators to get diverse message-level metrics such as reading time, relevance rate, comments, etc. We evaluated these designs through a 2-month field deployment with 5 communicators and 149 organization employees. We found that 1) the message-level metrics, such as reading time and relevance rate, helped communicators understand their audience and design bulk emails; 2) the cost and reputation metrics did not influence the organization leaders' decisions. We summarize with suggestions on designing organizational bulk email evaluation platforms that provide message-level performance and cost information.

Virtual Reality (VR) has a noteworthy educational potential by providing immersive and collaborative environments. As an alternative but cost-effective way of delivering realistic environments in VR, using 360-degree videos in immersive VR (VR videos) received more attention. Although many studies reported positive learning experiences with VR videos, little is known about how collaborative learning performs on VR video viewing systems. In this study, we implemented two collaborative VR video viewing modes based on the way of group video control, synchronized or shared (Sync mode) and non-synchronized or individual (Non-sync mode) video control, against a conventional VR video viewing setting (Basic mode). We conducted a within-subject study (N = 54) in a lab-simulated remote learning environment. Our results show that collaborative VR video modes (Sync and Non-sync mode) improve users’ learning experiences and collaboration quality, especially with shared video control. Our findings provide directions for designing and employing collaborative VR video tools in online learning environments.

A single digital newsletter usually contains many messages (regions). Users’ reading time spent on, and read level (skip/skim/read-in-detail) of each message is important for platforms to understand their users’ interests, personalize their contents, and make recommendations. Based on accurate but expensive-to-collect eyetracker-recorded data, we built models that predict per-region reading time based on easy-to-collect Javascript browser tracking data.

The COVID-19 pandemic disrupted everyone's life across the world. In this work, we characterize the subjective wellbeing patterns of 112 cities across the United States during the pandemic prior to vaccine availability, as exhibited in subreddits corresponding to the cities. We quantify subjective wellbeing using positive and negative affect. We then measure the pandemic's impact by comparing a community's observed wellbeing with its expected wellbeing, as forecasted by time series models derived from prior to the pandemic. We show that general community traits reflected in language can be predictive of community resilience. We predict how the pandemic would impact the wellbeing of each community based on linguistic and interaction features from normal times before the pandemic. We find that communities with interaction characteristics corresponding to more closely connected users and higher engagement were less likely to be significantly impacted. Notably, we find that communities that talked more about social ties normally experienced in-person, such as friends, family, and affiliations, were actually more likely to be impacted. Additionally, we use the same features to also predict how quickly each community would recover after the initial onset of the pandemic. We similarly find that communities that talked more about family, affiliations, and identifying as part of a group had a slower recovery.

Reranking is a critical task used to refine the initial collaborative filtering (CF) recommendation by incorporating information from different viewpoints, such as the extra item side-information and user profile. In this article, a neural reranking-based CF (NRCF) model is proposed to leverage composite viewpoints from the basic CF model and user preference. More precisely, the predictive implicit user preference is first constructed from the initial top-$k$items. The implicit user preference is then aggregated with the explicit user embedding to enrich the user intent representation. Moreover, the traditional listwise loss functions for reranking optimization are suboptimal, due to the fact that they neglect the relative ranking information (ReinRank) between the unobserved and positive items. To address this issue, a novel listwise loss function that leverages relative ranking information, referred to as ReinRank, is proposed for reranking optimization. ReinRank assigns different values to the unobserved items, according to their relative ranking distances between the positive items. Extensive experiments are performed on three public benchmarks and different CF models, in order to demonstrate the effectiveness of NRCF and ReinRank.

Bulk email is often used in organizations to communicate "important-to-organization'' messages such as policy changes, organizational plans, and administrative updates. However, normal employees may prefer messages more relevant to their jobs or interests. Organizations face the challenge of balancing prioritizing the messages they prefer employees to know (tactical goals) while maintaining employees' positive experiences with these bulk emails, then they continue to read these emails in the future (strategic goals). Could personalization help organizations achieve these tactical and strategic goals? In an 8-week field experiment with a university newsletter, we implemented a 4x5x5 factorial design on personalizing subject lines, top news, and message order based on both the employees' and the organization's preferences. We measured these designs' influences on the open/interest/recognition/read-in-detail rate of the whole newsletter and the single messages within it. We found that ''important-to-organization'' messages only got higher recognition rates when being put on subject lines / top news (tactical goal). Mixing them with employee-preferred messages in top news did not bring further improvement to their own recognition rates but could improve the whole newsletter's recognition rate. Only when the top news solely contained the employee-preferred messages were the employees slightly more interested in the newsletter (strategic goal). We further analyze on which topics the employees and the organization's preferences conflicted. Finally, we discuss the design suggestions for organizational bulk email.