Open User Models (OUM) leverage natural language to help users scrutinize their interests, moving beyond opaque behavioral logs to establish a ground truth of user preference. Building on this foundation, we introduce an LLM-generated open profile for a live movie recommender system that presents editable, personalized interest summaries. Unlike static profiles, this design invites users to directly inspect, audit, and reflect on the system’s inferences. In an eight-week field deployment with 1,775 active users, we found that participants actively engaged in preference construction, specifically pruning inaccurate dislikes and refining nuanced tastes. Our results show that this profile auditing process encourages high-efficiency maintenance behavior, leading to significantly increased reflection activity and engagement. We advocate for a Productively Wrong design strategy, where exposing mild AI inaccuracies serves as a catalyst to stimulate user intervention, fostering transparent systems where users actively curate their digital identity.

We present a case study of productive flyby users (PFB users) on a recommendation website.These users exhibit counterintuitive behavior: they input a large amount of data during their first visit but never return.This phenomenon can have both positive and negative impacts on the system.On the positive side, their high productivity contributes a substantial amount of data.On the negative side, they may input inappropriate ratings that violate the assumptions of recommendation algorithms, potentially undermining system performance.To better understand the nature and causes of this behavior, we investigated their motivations, expectations, reasons for leaving, and the potential risks associated with their ratings using a mixed-methods approach.Specifically, we conducted interviews with 11 users, surveyed 41 users, and analyzed the impact of 1,000 PFB users on the performance of recommendation algorithms for regular users.Our findings revealed diverse motivations among PFB users.Some engaged with the system merely to pass the time, while others had unrealistic expectations of the recommender system.Regarding rating quality, 27% of surveyed users admitted to rating movies they had not seen, citing reasons such as browsing too quickly or attempting to manipulate the algorithm.Notably, users who reported leaving because they were "just killing time and forgot about the website" were the most likely to rate unseen movies.Overall, PFB users significantly influence recommendation algorithms and their performance for regular users.While some subgroups negatively affect prediction accuracy, others provide

Online community research routinely poses minimal risk to individuals, but does the same hold true for online communities? In response to high-profile breaches of online community trust and increased debate in the social computing research community on the ethics of online community research, this paper investigates community-level harms and benefits of research. Through 9 participatory-inspired workshops with four critical online communities (Wikipedia, InTheRooms, CaringBridge, and r/AskHistorians), we found researchers should engage more directly with communities' primary purpose by rationalizing their methods and contributions in the context of community goals to equalize the beneficiaries of community research. To facilitate deeper alignment of these expectations, we present the FACTORS (Functions for Action with Communities: Teaching, Overseeing, Reciprocating, and Sustaining) framework for ethical online community research. Finally, we reflect on our findings by providing implications for researchers and online communities to identify and implement functions for navigating community-level harms and benefits.

An increasingly important aspect of designing recommender systems involves considering how recommendations will influence consumer choices. This paper addresses this issue by introducing a method for collecting user beliefs about un-experienced goods – a critical predictor of choice behavior. We implemented this method on the MovieLens platform, resulting in a rich dataset that combines user ratings, beliefs, and observed recommendations. We document challenges to such data collection, including selection bias in response and limited coverage of the product space. This unique resource empowers researchers to delve deeper into user behavior and analyze user choices absent recommendations, measure the effectiveness of recommendations, and prototype algorithms that leverage user belief data, ultimately leading to more impactful recommender systems. The dataset can be found at https://grouplens.org/datasets/movielens/ml_belief_2024/.

The Platform for OPen Recommendation and Online eXperimentation (POPROX) is a new resource to allow RecSys researchers to conduct online user research without having to develop all of the necessary infrastructure and recruit users. Our first domain is personalized news recommendations – POPROX 1.0 provides a daily newsletter (with content from the Associated Press) to users who have already consented to participate in research, along with interfaces and protocols to support researchers in conducting studies that assign subsets of users to various experimental algorithms and/or interfaces.

Recommender systems often struggle to strike a balance between matching users’ tastes and providing unexpected recommendations. When recommendations are too narrow and fail to cover the full range of users’ preferences, the system is perceived as useless. Conversely, when the system suggests too many items that users don’t like, it is considered impersonal or ineffective. To better understand user sentiment about the breadth of recommendations given by a movie recommender, we conducted interviews and surveys and found out that many users considered narrow recommendations to be useful, while a smaller number explicitly wanted greater breadth. Additionally, we designed and ran an online field experiment with a larger user group, evaluating two new interfaces designed to provide users with greater access to broader recommendations. We looked at user preferences and behavior for two groups of users: those with higher initial movie diversity and those with lower diversity. Among our findings, we discovered that different levels of exploration control and users’ subjective preferences on interfaces are more predictive of their satisfaction with the recommender.

Organizations often send bulk emails to employees to make them aware of policy changes, organization plans, and events. Many of these emails, however, are long digests with many separate messages that waste employees' time and reduce their awareness. This study introduces CommTool--a prototype tool to help organizational communicators better understand their emails' performance and cost. We first interviewed 5 communicators and identified the need to measure the performance of each message within bulk email. Then we iteratively designed and deployed an organizational bulk email evaluation platform (CommTool), which enables communicators to get diverse message-level metrics such as reading time, relevance rate, comments, etc. We evaluated these designs through a 2-month field deployment with 5 communicators and 149 organization employees. We found that 1) the message-level metrics, such as reading time and relevance rate, helped communicators understand their audience and design bulk emails; 2) the cost and reputation metrics did not influence the organization leaders' decisions. We summarize with suggestions on designing organizational bulk email evaluation platforms that provide message-level performance and cost information.

A single digital newsletter usually contains many messages (regions). Users’ reading time spent on, and read level (skip/skim/read-in-detail) of each message is important for platforms to understand their users’ interests, personalize their contents, and make recommendations. Based on accurate but expensive-to-collect eyetracker-recorded data, we built models that predict per-region reading time based on easy-to-collect Javascript browser tracking data.

We conduct a 6 month field experiment on a movie-recommendation platform to identify if and how recommendation systems affect consumption. We use within-consumer randomization at the good level and elicit beliefs about unconsumed goods to disentangle exposure from informational effects. We have three experimental groups: (a) control, (b) exposed, and (c) recommended + exposed goods where only goods in (c) are recommended and we elicit beliefs about goods in (b) and (c). Comparing across these treatment arms we find recommendations increase consumption beyond its role in exposing goods to consumers. We provide support for an informational mechanism: recommendations affect consumers' beliefs, which in turn explain consumption. Recommendations reduce uncertainty about goods consumers are most uncertain about and induce information acquisition. Finally, we find evidence for spatial correlation in beliefs.

Artificial intelligence (AI) algorithms are increasingly making decisions and supporting human decisions. Both autonomous systems and algorithm-in-the-loop decision-support systems use AI algorithm...

Widespread adoption of artificial intelligence (AI) technologies is substantially affecting the human condition in ways that are not yet well understood. Negative unintended consequences abound including the perpetuation and exacerbation of societal inequalities and divisions via algorithmic decision making. We present six grand challenges for the scientific community to create AI technologies that are human-centered, that is, ethical, fair, and enhance the human condition. These grand challenges are the result of an international collaboration across academia, industry and government and represent the consensus views of a group of 26 experts in the field of human-centered artificial intelligence (HCAI). In essence, these challenges advocate for a human-centered approach to AI that (1) is centered in human well-being, (2) is designed responsibly, (3) respects privacy, (4) follows human-centered design principles, (5) is subject to appropriate governance and oversight, and (6) interacts with individuals while respecting human’s cognitive capacities. We hope that these challenges and their associated research directions serve as a call for action to conduct research and development in AI that serves as a force multiplier towards more fair, equitable and sustainable societies.

The Every Student Succeeds Act (ESSA) prescribes holistic measures of schools for student success and well-being. However, many early warning systems rely exclusively on the "Attendance, Behavior, Course" (ABC) taxonomy, which misses potentially crucial determinants such as school climate and students' socioemotional learning. We report early findings from a larger project that aims to apply machine learning methods to improve an early warning system by incorporating factors related to school climate and socioemotional learning. These preliminary analyses suggest that a culturally inclusive, socially supportive, and emotionally and physically safe school climate is related to academic success and fewer engagement/behavior problems at the school level. They suggest the promise of integrating these features into early warning systems to help schools change their practices to better support student well-being.

Bulk email is often used in organizations to communicate "important-to-organization'' messages such as policy changes, organizational plans, and administrative updates. However, normal employees may prefer messages more relevant to their jobs or interests. Organizations face the challenge of balancing prioritizing the messages they prefer employees to know (tactical goals) while maintaining employees' positive experiences with these bulk emails, then they continue to read these emails in the future (strategic goals). Could personalization help organizations achieve these tactical and strategic goals? In an 8-week field experiment with a university newsletter, we implemented a 4x5x5 factorial design on personalizing subject lines, top news, and message order based on both the employees' and the organization's preferences. We measured these designs' influences on the open/interest/recognition/read-in-detail rate of the whole newsletter and the single messages within it. We found that ''important-to-organization'' messages only got higher recognition rates when being put on subject lines / top news (tactical goal). Mixing them with employee-preferred messages in top news did not bring further improvement to their own recognition rates but could improve the whole newsletter's recognition rate. Only when the top news solely contained the employee-preferred messages were the employees slightly more interested in the newsletter (strategic goal). We further analyze on which topics the employees and the organization's preferences conflicted. Finally, we discuss the design suggestions for organizational bulk email.

Bulk email is a primary communication channel within organizations, with all-company emails and regular newsletters serving as a mechanism for making employees aware of policies and events. Ineffective communication could result in wasted employee time and a lack of compliance or awareness. Previous studies on organizational emails focused mostly on recipients. However, organizational bulk email system is a multi-stakeholder problem including recipients, communicators, and the organization itself. We studied the effectiveness, practice, and assessments of the organizational bulk email system of a large university from multi-stakeholders' perspectives. We conducted a qualitative study with the university's communicators, recipients, and managers. We delved into the organizational bulk email's distributing mechanisms of the communicators, the reading behaviors of recipients, and the perspectives on emails' values of communicators, managers, and recipients. We found that the organizational bulk email system as a whole was strained, and communicators are caught in the middle of this multi-stakeholder problem. First, though the communicators had an interest in preserving the effectiveness of channels in reaching employees, they had high-level clients whose interests might outweigh judgment about whether a message deserves widespread circulation. Second, though communicators thought they were sending important information, recipients viewed most of the organizational bulk emails as not relevant to them. Third, this disagreement was amplified by the success metric used by communicators. They viewed their bulk emails as successful if they had a high open rate. But recipients often opened and then rapidly discarded emails without reading the details. Last, while the communicators in general understood the challenge, they had a limited set of targeting and feedback tools to support their task.

Recommender systems research is largely focused on the value such systems can create for users, e.g., by helping them finding items of interest in situations of information overload. However, there are various other ways in which recommender systems can create value and have an impact on individuals and organizations. The goal of the workshop is to serve as a platform where researchers discuss recent insights on how recommender systems affect individuals, user communities, or organizations. The workshop also aims at raising awareness regarding the importance of impact-oriented research.

Group projects are an essential component of teaching user interface (UI) design. We identified six challenges in transferring traditional group projects into the context of Massive Open Online Courses: managing dropout, avoiding free-riding, appropriate scaffolding, cultural and time zone differences, and establishing common ground. We present a case study of the design of a group project for a UI Design MOOC, in which we implemented technical tools and social structures to cope with the above challenges. Based on survey analysis, interviews, and team chat data from the students over a six-month period, we found that our socio-technical design addressed many of the obstacles that MOOC learners encountered during remote collaboration. We translate our findings into design implications for better group learning experiences at scale.

User-system interaction in recommender systems involves three aspects: temporal browsing (viewing recommendation lists and/or searching/filtering), action (performing actions on recommended items, e.g., clicking, consuming) and inaction (neglecting or skipping recommended items). Modern recommenders build machine learning models from recordings of such user interaction with the system, and in doing so they commonly make certain assumptions (e.g., pairwise preference orders, independent or competitive probabilistic choices, etc.). In this paper, we set out to study the effects of these assumptions along three dimensions in eight different single models and three associated hybrid models on a user browsing data set collected from a real-world recommender system application. We further design a novel model based on recurrent neural networks and multi-task learning, inspired by Decision Field Theory, a model of human decision making. We report on precision, recall, and MAP, finding that this new model outperforms the others.

Organizers of online groups often struggle to recruit members who can most effectively carry out the group's activities and remain part of the group over time. In a study of a sample of 30,000 new editors belonging to 1,054 English WikiProjects, we empirically examine the effects of generalized prior work-productivity experience (measured by overall prior article edits), prior leadership experience (measured by overall prior project edits), and localized prior work-productivity experience (measured by pre-joining article edits on a project) on early retention and productivity. We find that (1)generalized prior work-productivity experience is positively associated with retention, but negatively associated with productivity (2) prior leadership experience is negatively associated with both retention and productivity, and (3) localized prior work-productivity experience is positively associated with both retention and productivity within that focal project. We then discuss implications to inform the designs of early interventions aimed at group success.

Temporally, users browse and interact with items in recommender systems. However, for most systems, the majority of the displayed items do not elicit any action from users. In other words, the user-system interaction process includes three aspects: browsing, action, and inaction. Prior recommender systems literature has focused more on actions than on browsing or inaction. In this work, we deployed a field survey in a live movie recommender system to interpret what inaction means from both the user's and the system's perspective, guided by psychological theories of human decision making. We further systematically study factors to infer the reasons of user inaction and demonstrate with offline data sets that this descriptive and predictive inaction model can provide benefits for recommender systems in terms of both action prediction and recommendation timing.

Current recommender systems often show the same most-highly recommended items again and again ignoring the feedback that users neither rate nor click on those items. We conduct an online field experiment to test two ways of manipulating top-N recommendations with the goal of improving user experience: cycling the top-N recommendation based on their past presentation and serpentining the top-N list mixing the best items into later recommendation requests. We find interesting tensions between opt-outs and activities, user perceived accuracy and freshness. Cycling within the same session might be a "love it or hate it" recommender property because users in it have a higher opt-out rate but engage in more activities. Cycling across sessions and serpentining increase user activities without significantly affecting opt-out rates. Users perceive more change and freshness but less accuracy and familiarity. Combining cycling and serpentining does not work as well as each individual manipulation separately. These two ways of manipulations on top-N list demonstrate some attractive properties but also call for innovative approaches to overcome their potential costs.

In MOOCs, both instructors and students invest substantial effort into discussion forums. However, those discussions are abandoned when instructors start a new session in session-based courses. In an observational field study through a popular online Coursera course, we evaluate an approach that directly embeds high-value past discussion threads into future lecture videos to reuse them. Survey feedback shows that this approach can be useful to a large proportion of learners. We find that instructor involvement increases learners' chance of reading the threads, reduces learners' negative reactions, but is not associated with more perceived usefulness. Learners perceive enhancing threads embedded in the middle of videos less enhancing but more explanatory compared with at the end. Embedding explanatory threads at the end is rated less distracting and more helpful to understand the video compared with in the middle, right after the related content is lectured.

The technical barriers for conversing with recommender systems using natural language are vanishing. Already, there are commercial systems that facilitate interactions with an AI agent. For instance, it is possible to say "what should I watch" to an Apple TV remote to get recommendations. In this research, we investigate how users initially interact with a new natural language recommender to deepen our understanding of the range of inputs that these technologies can expect. We deploy a natural language interface to a recommender system, we observe users' first interactions and follow-up queries, and we measure the differences between speaking- and typing-based interfaces. We employ qualitative methods to derive a categorization of users' first queries (objective, subjective, and navigation) and follow-up queries (refine, reformulate, start over). We employ quantitative methods to determine the differences between speech and text, finding that speech inputs are typically longer and more conversational.

Exploiting evidence that sporting results affect fans' mood, we analyze whether National Football League game outcomes can affect the contributions of Wikipedia editors who identify as fans of a specific team. We find that the day after a team loses, their fans decrease their contributions towards football-related pages (relative to after a win). Relative decreases are bigger if losses are unexpected, or if losing margins are big. In contrast, unexpected wins do not cause more contributions relative to wins that were not unexpected. Neither do big wins result in more contributions relative to small wins. Additionally, contributions to non-football-related pages are not affected by NFL game results. Our findings add to the literatures on (i) the determinants of individual contributions to peer production communities, (ii) how community dynamics affect user contributions, (iii) the importance of emotions, (iv) the effect of offline events on online behavior, and (v) the applicability of behavioral economics concepts to the HCI literature.

As the gig economy continues to grow and freelance work moves online, five-star reputation systems are becoming more and more common. At the same time, there are increasing accounts of race and gender bias in evaluations of gig workers, with negative impacts for those workers. We report on a series of four Mechanical Turk-based studies in which participants who rated simulated gig work did not show race- or gender bias, while manipulation checks showed they reliably distinguished between low- and high-quality work. Given prior research, this was a striking result. To explore further, we used a Bayesian approach to verify absence of ratings bias (as opposed to merely not detecting bias). This Bayesian test let us identify an upper- bound: if any bias did exist in our studies, it was below an average of 0.2 stars on a five-star scale. We discuss possible interpretations of our results and outline future work to better understand the results.

Online communities suffer serious newcomer attrition. This paper explores whether and how early activity diversity - the degree to which a newcomer engages in a wide range of a site's activities in the first session - is associated with their longevity. We introduce a metric (DSCORE) to characterize early activity diversity in online sites and run our analyses on an online community ‘MovieLens’. We find that DSCORE is significant both by itself and in conjunction with a measure of quantity of activity in predicting longevity. This finding is robust to different measures of longevity (aggregate number of sessions and attritions after sessions 1, 5, and 10). The immediate implication is an effective classifier for identifying users with higher (or lower) expected longevity from the first-session activity. We also find DSCORE is more useful than a traditional measure of measuring diversity such as the Gini-Simpson index. We conclude by discussing how early activity diversity may be more broadly effective in supporting design and management of online communities.

We introduce a theoretical framework called precision crowdsourcing whose goal is to help turn online information consumers into information contributors. The framework looks at the timing and nature of the requests made of users and the feedback provided to users with the goal of increasing long-term contribution and engagement in the site or system. We present the results of a field experiment in which almost 3000 users were asked to tag movies (plus a null control group) as we varied the selection of task (popular/obscure), timing of requests (immediate or varying delays), and relational rhetoric (neutral, system reciprocal, other users reciprocal) of the requests. We found that asking increases tags provided overall, though asking generally decreases the provision of unprompted tags. Users were more likely to comply with our request when we asked them to tag obscure movies and when we used reciprocal request rhetoric.

Prior work relevant to incorporating personality into recommender systems falls into two categories: social science studies and algorithmic ones. Social science studies of preference have found only small relationships between personality and category preferences, whereas, algorithmic approaches found a little improvement when incorporating personality into recommendations. As a result, despite good reasons to believe personality assessments should be useful in recommenders, we are left with no substantial demonstrated impact. In this work, we start with user data from a live recommender system, but study category-by-category variations in preference (both rating levels and distribution) across different personality types. By doing this, we hope to isolate specific areas where personality is most likely to provide value in recommender systems, while also modeling an analytic process that can be used in other domains. After controlling for the family-wise error rate, we find that High Agreeableness users rate at least 0.5 stars higher on a 5-star scale compared to low Agreeableness users. We also find differences in consumption in four different personality types between people who manifested high and low levels of that personality.

As users browse a recommender system, they systematically consider or skip over much of the displayed content. It seems obvious that these eye gaze patterns contain a rich signal concerning these users' preferences. However, because eye tracking data is not available to most recommender systems, these signals are not widely incorporated into personalization models. In this work, we show that it is possible to predict gaze by combining easily-collected user browsing data with eye tracking data from a small number of users in a grid-based recommender interface. Our technique is able to leverage a small amount of eye tracking data to infer gaze patterns for other users. We evaluate our prediction models in MovieLens -- an online movie recommender system. Our results show that incorporating eye tracking data from a small number of users significantly boosts accuracy as compared with only using browsing data, even though the eye-tracked users are different from the testing users (e.g. AUC=0.823 vs. 0.693 in predicting whether a user will fixate on an item). We also demonstrate that Hidden Markov Models (HMMs) can be applied in this setting; they are better than linear models in predicting fixation probability and capturing the interface regularity through Bayesian inference (AUC=0.823 vs. 0.757).

In this work, we explore the degree to which personality information can be used to model newcomer retention, investment, intensity of engagement, and distribution of activity in a recommender community. Prior work shows that Big-Five Personality traits can explain variation in user behavior in other contexts. Building on this, we carry out and report on an analysis of 1008 MovieLens users with identified personality profiles. We find that Introverts and low Agreeableness users are more likely to survive into the second and subsequent sessions compared to their respective counterparts; Introverts and low Conscientiousness users are a significantly more active population compared to their respective counterparts; High Openness and High Neuroticism users contribute (tag) significantly more compared to their counterparts, but their counterparts consume (browse and bookmark) more; and low Agreeableness users are more likely to rate whereas high Agreeableness users are more likely to tag. These results show how modeling newcomer behavior from user personality can be useful for recommender systems designers as they customize the system to guide people towards tasks that need to be done or tasks the users will find rewarding and also decide which users to invest retention efforts in.

The MovieLens datasets are widely used in education, research, and industry. They are downloaded hundreds of thousands of times each year, reflecting their use in popular press programming books, traditional and online courses, and software. These datasets are a product of member activity in the MovieLens movie recommendation system, an active research platform that has hosted many experiments since its launch in 1997. This article documents the history of MovieLens and the MovieLens datasets. We include a discussion of lessons learned from running a long-standing, live research platform from the perspective of a research organization. We document best practices and limitations of using the MovieLens datasets in new research.

Recommender systems are not one-size-fits-all; different algorithms and data sources have different strengths, making them a better or worse fit for different users and use cases. As one way of taking advantage of the relative merits of different algorithms, we gave users the ability to change the algorithm providing their movie recommendations and studied how they make use of this power. We conducted our study with the launch of a new version of the MovieLens movie recommender that supports multiple recommender algorithms and allows users to choose the algorithm they want to provide their recommendations. We examine log data from user interactions with this new feature to under-stand whether and how users switch among recommender algorithms, and select a final algorithm to use. We also look at the properties of the algorithms as they were experienced by users and examine their relationships to user behavior. We found that a substantial portion of our user base (25%) used the recommender-switching feature. The majority of users who used the control only switched algorithms a few times, trying a few out and settling down on an algorithm that they would leave alone. The largest number of users prefer a matrix factorization algorithm, followed closely by item-item collaborative filtering; users selected both of these algorithms much more often than they chose a non-personalized mean recommender. The algorithms did produce measurably different recommender lists for the users in the study, but these differences were not directly predictive of user choice.

Studies have shown that the recommendation of unseen, novel or serendipitous items is crucial for a satisfying and engaging user experience. As a result, recent developments in recommendation research have increasingly focused towards introducing novelty in user recommendation lists. While, existing solutions aim to find the right balance between the similarity and novelty of the recommended items, they largely ignore the user needs for novelty. In this paper, we show that there are large individual and temporal differences in the users' novelty preferences. We develop a regression model to predict these dynamic novelty preferences of users using features derived from their past interactions. Finally, we describe an adaptive recommender,~\emph{adaNov-R}, that adapts to the user needs for novel items and show that the model achieves better recommendation performance on a metric that considers both novel and familiar items.

In the fall of 2013, we offered an open online Introduction to Recommender Systems through Coursera, while simultaneously offering a for-credit version of the course on-campus using the Coursera platform and a flipped classroom instruction model. As the goal of offering this course was to experiment with this type of instruction, we performed extensive evaluation including surveys of demographics, self-assessed skills, and learning intent; we also designed a knowledge-assessment tool specifically for the subject matter in this course, administering it before and after the course to measure learning, and again 5 months later to measure retention. We also tracked students through the course, including separating out students enrolled for credit from those enrolled only for the free, open course. Students had significant knowledge gains across all levels of prior knowledge and across all demographic categories. The main predictor of knowledge gain was effort expended in the course. Students also had significant knowledge retention after the course. Both of these results are limited to the sample of students who chose to complete our knowledge tests. Student completion of the course was hard to predict, with few factors contributing predictive power; the main predictor of completion was intent to complete. Students who chose a concepts-only track with hand exercises achieved the same level of knowledge of recommender systems concepts as those who chose a programming track and its added assignments, though the programming students gained additional programming knowledge. Based on the limited data we were able to gather, face-to-face students performed as well as the online-only students or better; they preferred this format to traditional lecture for reasons ranging from pure convenience to the desire to watch videos at a different pace (slower for English language learners; faster for some native English speakers). This article also includes our qualitative observations, lessons learned, and future directions.

In Fall 2013 we offered an open online Introduction to Recommender Systems through Coursera, while simultaneously offering a for-credit version of the course on-campus using the Coursera platform and a flipped classroom instruction model. As the goal of offering this course was to experiment with this type of instruction, we performed extensive evaluation including surveys of demographics, self-assessed skills, and learning intent; we also designed a knowledge-assessment tool specifically for the subject matter in this course, administering it before and after the course to measure learning. We also tracked students through the course, including separating out students enrolled for credit from those enrolled only for the free, open course. This article reports on our findings.

Recent developments in user evaluation of recommender systems have brought forth powerful new tools for understanding what makes recommendations effective and useful. We apply these methods to understand how users evaluate recommendation lists for the purpose of selecting an algorithm for finding movies. This paper reports on an experiment in which we asked users to compare lists produced by three common collaborative filtering algorithms on the dimensions of novelty, diversity, accuracy, satisfaction, and degree of personalization, and to select a recommender that they would like to use in the future. We find that satisfaction is negatively dependent on novelty and positively dependent on diversity in this setting, and that satisfaction predicts the user's final selection. We also compare users' subjective perceptions of recommendation properties with objective measures of those same characteristics. To our knowledge, this is the first study that applies modern survey design and analysis techniques to a within-subjects, direct comparison study of recommender algorithms.

The new user experience is one of the important problems in recommender systems. Past work on recommending for new users has focused on the process of gathering information from the user. Our work focuses on how different algorithms behave for new users. We describe a methodology that we use to compare representatives of three common families of algorithms along eleven different metrics. We find that for the first few ratings a baseline algorithm performs better than three common collaborative filtering algorithms. Once we have a few ratings, we find that Funk's SVD algorithm has the best overall performance. We also find that ItemItem, a very commonly deployed algorithm, performs very poorly for new users. Our results can inform the design of interfaces and algorithms for new users.

Eli Pariser coined the term 'filter bubble' to describe the potential for online personalization to effectively isolate people from a diversity of viewpoints or content. Online recommender systems - built on algorithms that attempt to predict which items users will most enjoy consuming - are one family of technologies that potentially suffers from this effect. Because recommender systems have become so prevalent, it is important to investigate their impact on users in these terms. This paper examines the longitudinal impacts of a collaborative filtering-based recommender system on users. To the best of our knowledge, it is the first paper to measure the filter bubble effect in terms of content diversity at the individual level. We contribute a novel metric to measure content diversity based on information encoded in user-generated tags, and we present a new set of methods to examine the temporal effect of recommender systems on the user experience. We do find that recommender systems expose users to a slightly narrowing set of items over time. However, we also see evidence that users who actually consume the items recommended to them experience lessened narrowing effects and rate items more positively.

This paper explores a method for incorporating into a recommender system explicit representations of user's preferences over non-content attributes such as popularity, recency, and similarity of recommended items. We show how such attributes can be modeled as a preference vector that can be used in a vector-space content-based recommender, and how that content-based recommender can be integrated with various collaborative filtering techniques through re-weighting of Top-M recommendations. We evaluate this approach on several recommender systems datasets and collaborative filtering methods, and find that incorporating the three preference attributes can lead to a substantial increase in Top-50 precision while also enhancing diversity and novelty.

Although many off-line organizations give their employees training, mentorship, a cohort and other socialization experiences that improve their retention and productivity, online production communities rarely do this. This paper describes the planning, execution and evaluation of a socialization regime for an online technical support community. In a two-phase project, we first automatically identified from participants' early behavior, those with high potential to become core members. We then designed, delivered and experimentally evaluated socialization experiences intended to build commitment and competence among these potential core members. We were able to identify potential core members with high accuracy from only two weeks of behavior. A year later, those classified as potential core members participated in the community ten times more actively than those not identified. In an evaluation experiment, some potential core members were randomly assigned to receive socialization experiences, while others were not. A year later, those who had participated in the socialization regime contributed more answers in the community compared to those in the control condition. The socialization experiences, however, undercut their sense of connection to the community and the quality of their contributions. We discuss what was effective and what could be improved in designing socialization experiences for online groups.

In this paper, we introduce the concept of question temporality as a measure of the usefulness of the answers provided on the questions asked in the Question Answering sites (QA). We define question temporality based on when the answers provided on the questions would expire. We use classification methods to show that the question temporality can be assessed automatically. Our regression analysis highlights features that predict temporality of the questions. Our research can be instructive for interface designers to design temporality-aware interfaces and influence selection of questions and answers for display.

Community Question Answering (CQA) services enable their users to exchange knowledge in the form of questions and answers. These communities thrive as a result of a small number of highly active users, typically calledexperts, who provide a large number of high-quality useful answers. Expert identification techniques enable community managers to take measures to retain the experts in the community. There is further value in identifying the experts during the first few weeks of their participation as it would allow measures to nurture and retain them. In this article we address two problems: (a) How to identify current experts in CQA? and (b) How to identify users who have potential of becoming experts in future (potential experts)? In particular, we propose a probabilistic model that captures the selection preferences of users based on the questions they choose for answering. The probabilistic model allows us to run machine learning methods for identifying experts and potential experts. Our results over several popular CQA datasets indicate that experts differ considerably from ordinary users in their selection preferences; enabling us to predict experts with higher accuracy over several baseline models. We show that selection preferences can be combined with baseline measures to improve the predictive performance even further.

The ability to embed links to other resources in user generated content can help authors create more useful and usable content. A variety of interfaces have emerged for entity-linking at popular online sites; such interfaces vary in the way that entity linking is initiated (in-band or out-of-band with respect to the message creation), the timing of entity resolution (interrupting or deferred), and the method of resolving the entity (auto-completion or search). Four interfaces mimicking popular entity linking websites were developed and tested. Results showed that out-of-band initiation (e.g., a link button) was faster to learn, but that in-band initiation performance improved with familiarity. Deferred search was disliked and led to worse performance. And auto-completion was generally preferred to search interfaces.

This tutorial is an introduction to the concepts and techniques from human-computer interaction, focused on designing usable interfaces. Topics covered include user and task analysis, prototyping, design techniques, interface evaluation, and various processes for designing interfaces. While the overall content mirrors general user interface design courses, a section will focus specifically on challenges in designing intelligent systems (including adaptive systems, recommender systems, agent-based interfaces, etc.), including an exploration of the metaphors and paradigms of interaction with intelligent systems.

Recommender systems research is being slowed by the difficulty of replicating and comparing research results. Published research uses various experimental methodologies and metrics that are difficult to compare. It also often fails to sufficiently document the details of proposed algorithms or the evaluations employed. Researchers waste time reimplementing well-known algorithms, and the new implementations may miss key details from the original algorithm or its subsequent refinements. When proposing new algorithms, researchers should compare them against finely-tuned implementations of the leading prior algorithms using state-of-the-art evaluation methodologies. With few exceptions, published algorithmic improvements in our field should be accompanied by working code in a standard framework, including test harnesses to reproduce the described results. To that end, we present the design and freely distributable source code of LensKit, a flexible platform for reproducible recommender systems research. LensKit provides carefully tuned implementations of the leading collaborative filtering algorithms, APIs for common recommender system use cases, and an evaluation framework for performing reproducible offline evaluations of algorithms. We demonstrate the utility of LensKit by replicating and extending a set of prior comparative studies of recommender algorithms --- showing limitations in some of the original results --- and by investigating a question recently raised by a leader in the recommender systems community on problems with error-based prediction evaluation.

We examine the use of modern recommender system technology to aid command awareness in complex software applications. We first describe our adaptation of traditional recommender system algorithms to meet the unique requirements presented by the domain of software commands. A user study showed that our item-based collaborative filtering algorithm generates 2.1 times as many good suggestions as existing techniques. Motivated by these positive results, we propose a design space framework and its associated algorithms to support both global and contextual recommendations. To evaluate the algorithms, we developed the CommunityCommands plug-in for AutoCAD. This plug-in enabled us to perform a 6-week user study of real-time, within-application command recommendations in actual working environments. We report and visualize command usage behaviors during the study, and discuss how the recommendations affected users behaviors. In particular, we found that the plug-in successfully exposed users to new commands, as unique commands issued significantly increased.

Community Question Answering (CQA) services enables users to ask and answer questions. In these communities, there are typically a small number of experts amongst the large population of users. We study which questions a user select for answering and show that experts prefer answering questions where they have a higher chance of making a valuable contribution. We term this preferential selection as question selection bias and propose a mathematical model to estimate it. Our results show that using Gaussian classification models we can effectively distinguish experts from ordinary users over their selection biases. In order to estimate these biases, only a small amount of data per user is required, which makes an early identification of expertise a possibility. Further, our study of bias evolution reveals that they do not show significant changes over time indicating that they emanates from the intrinsic characteristics of users.

All new researchers face the daunting task of familiarizing themselves with the existing body of research literature in their respective fields. Recommender algorithms could aid in preparing these lists, but most current algorithms do not understand how to rate the importance of a paper within the literature, which might limit their effectiveness in this domain. We explore several methods for augmenting existing collaborative and content-based filtering algorithms with measures of the influence of a paper within the web of citations. We measure influence using well-known algorithms, such as HITS and PageRank, for measuring a node's importance in a graph. Among these augmentation methods is a novel method for using importance scores to influence collaborative filtering. We present a task-centered evaluation, including both an offline analysis and a user study, of the performance of the algorithms. Results from these studies indicate that collaborative filtering outperforms content-based approaches for generating introductory reading lists.

Contests and challenges have energized researchers and focused attention in many fields recently, including recommender systems. At the 2008 RecSys conference, winners were announced for a contest proposing new startup companies. The 2009 conference featured a panel reflecting on the then recently completed Netflix challenge.

Tens of thousands of questions are asked and answered every day on social question and answer (Q&A) Web sites such as Yahoo Answers. While these sites generate an enormous volume of searchable data, the problem of determining which questions and answers are archival quality has grown. One major component of this problem is the prevalence of conversational questions, identified both by Q&A sites and academic literature as questions that are intended simply to start discussion. For example, a conversational question such as "do you believe in evolution?" might successfully engage users in discussion, but probably will not yield a useful web page for users searching for information about evolution. Using data from three popular Q&A sites, we confirm that humans can reliably distinguish between these conversational questions and other informational questions, and present evidence that conversational questions typically have much lower potential archival value than informational questions. Further, we explore the use of machine learning techniques to automatically classify questions as conversational or informational, learning in the process about categorical, linguistic, and social differences between different question types. Our algorithms approach human performance, attaining 89.7% classification accuracy in our experiments.

Question and answer (Q&A) sites such as Yahoo! Answers are places where users ask questions and others answer them. In this paper, we investigate predictors of answer quality through a comparative, controlled field study of responses provided across several online Q&A sites. Along with several quantitative results concerning the effects of factors such as question topic and rhetorical strategy, we present two high-level messages. First, you get what you pay for in Q&A sites. Answer quality was typically higher in Google Answers (a fee-based site) than in the free sites we studied, and paying more money for an answer led to better outcomes. Second, we find that a Q&A site's community of users contributes to its success. Yahoo! Answers, a Q&A site where anybody can answer questions, outperformed sites that depend on specific individuals to answer questions, such as library reference services.

Algorithmic recommender systems attempt to predict which items a target user will like based on information about the user's prior preferences and the preferences of a larger community. After more than a decade of widespread use, researchers and system users still debate whether such "impersonal" recommender systems actually perform as well as human recommenders. We compare the performance of MovieLens algorithmic predictions with the recommendations made, based on the same user profiles, by active MovieLens users. We found that algorithmic collaborative filtering outperformed humans on average, though some individuals outperformed the system substantially and humans on average outperformed the system on certain prediction tasks.

Recommender systems help users find the information, products, and other people they most want to find. This tutorial provides participants with a hands-on learning experience about using recommender system technologies. After completing this tutorial, participants will understand the range of technologies being used for recommender systems, including collaborative filtering, rules-based systems, and information filtering.

Spyware is an increasing problem. Interestingly, many programs carrying spyware honestly disclose the activities of the software, but users install the software anyway. We report on a study of software installation to assess the effectiveness of different notices for helping people make better decisions on which software to install. Our study of 222 users showed that providing a short summary notice, in addition to the End User License Agreement (EULA), before the installation reduced the number of software installations significantly. We also found that providing the short summary notice after installation led to a significant number of uninstalls. However, even with the short notices, many users installed the program and later expressed regret for doing so. These results, along with a detailed analysis of installation, regret, and survey data about user behaviors informs our recommendations to policymakers and designers for assessing the "adequacy" of consent in the context of software that exhibits behaviors associated with spyware.

Watching video online is becoming increasingly popular, and new video streaming technologies have the potential to transform video watching from a passive, isolating experience into an active, socially engaging experience. However, the viability of an active social experience is unclear: both chatting and watching video require attention, and may interfere with one another and detract from the experience. In this paper, we empirically examine the activity of chatting while watching video online. We examine how groups of friends and strangers interact, and find that chat has a positive influence on social relationships, and people chat despite being distracted. We discuss the benefits and opportunities provided by mixing chat and video, uncover some of the attentional and social challenges inherent in this combination of media, and provide guidance for structuring the viewing experience.

Rapid and continuous growth of digital libraries, coupled with brisk advancements in technology, has driven users to seek tools and services that are not only customized to their specific needs, but are also helpful in keeping them stay abreast with the latest developments in their field. TechLens is a recommender system that learns about its users through implicit feedback, builds correlations among them, and uses that information to generate recommendations that match the user's profile. It gives users control over which parts of their profile of known citations are used in forming recommendations for new articles. This demonstration is a prototype that showcases some of the tools and services that TechLens offers to the users of digital libraries.

If recommenders are to help people be more productive, they need to support a wide variety of real-world information seeking tasks, such as those found when seeking research papers in a digital library. There are many potential pitfalls, including not knowing what tasks to support, generating recommendations for the wrong task, or even failing to generate any meaningful recommendations whatsoever. We posit that different recommender algorithms are better suited to certain information seeking tasks. In this work, we perform a detailed user study with over 130 users to understand these differences between recommender algorithms through an online survey of paper recommendations from the ACM Digital Library. We found that pitfalls are hard to avoid. Two of our algorithms generated 'atypical' recommendations recommendations that were unrelated to their input baskets. Users reacted accordingly, providing strong negative results for these algorithms. Results from our 'typical' algorithms show some qualitative differences, but since users were exposed to two algorithms, the results may be biased. We present a wide variety of results, teasing out differences between algorithms. Finally, we succinctly summarize our most striking results as "Don't Look Stupid" in front of users.

This workshop intends to bring recommender systems researchers and practitioners together in order to discuss the current state of recommender systems research, both on existing and emerging research topics, and to determine how research in this area should proceed. We are at a pivotal point in recommender systems research where researchers are both looking inward at what recommender systems are and looking outward at where recommender systems can be applied, and the implications of applying them out 'in the wild.' This creates a unique opportunity to both reassess the current state of research and directions research is taking in the near and long term.

Spyware is a significant problem for most computer users. The term "spyware" loosely describes a new class of computer software. This type of software may track user activities online and offline, provide targeted advertising and/or engage in other types of activities that users describe as invasive or undesirable.While the magnitude of the spyware problem is well documented, recent studies have had only limited success in explaining the broad range of user behaviors that contribute to the proliferation of spyware. As opposed to viruses and other malicious code, users themselves often have a choice whether they want to install these programs.In this paper, we discuss an ecological study of users installing five real world applications. In particular, we seek to understand the influence of the form and content of notices (e.g., EULAs) on user's installation decisions.Our study indicates that while notice is important, notice alone may not be enough to affect users' decisions to install an application. We found that users have limited understanding of EULA content and little desire to read lengthy notices. Users found short, concise notices more useful, and noticed them more often, yet they did not have a significant effect on installation for our population. When users were informed of the actual contents of the EULAs to which they agreed, we found that users often regret their installation decisions.We discovered that regardless of the bundled content, users will often install an application if they believe the utility is high enough. However, we discovered that privacy and security become important factors when choosing between two applications with similar functionality. Given two similar programs (e.g. KaZaA and Edonkey), consumers will choose the one they believe to be less invasive and more stable. We also found that providing vague information in EULAs and short notices can create an unwarranted impression of increased security. In these cases, it may be helpful to have a standardized format for assessing the possible options and trade-offs between applications.

In this work we present topic diversification, a novel method designed to balance and diversify personalized recommendation lists in order to reflect the user's complete spectrum of interests. Though being detrimental to average accuracy, we show that our method improves user satisfaction with recommendation lists, in particular for lists generated using the common item-based collaborative filtering algorithm.Our work builds upon prior research on recommender systems, looking at properties of recommendation lists as entities in their own right rather than specifically focusing on the accuracy of individual recommendations. We introduce the intra-list similarity metric to assess the topical diversity of recommendation lists and the topic diversification approach for decreasing the intra-list similarity. We evaluate our method using book recommendation data, including offline analysis on 361, !, 349 ratings and an online study involving more than 2, !, 100 subjects.

Recommender systems have been evaluated in many, often incomparable, ways. In this article, we review the key decisions in evaluating collaborative filtering recommender systems: the user tasks being evaluated, the types of analysis and datasets being used, the ways in which prediction quality is measured, the evaluation of prediction attributes other than quality, and the user-based evaluation of the system as a whole. In addition to reviewing the evaluation strategies used by prior researchers, we present empirical results from the analysis of various accuracy metrics on one content domain where all the tested metrics collapsed roughly into three equivalence classes. Metrics within each equivalency class were strongly correlated, while metrics from different equivalency classes were uncorrelated.

and founder of the eCommerce Competence Center in Vienna. His research activities cover decision support systems, simulation, artificial intelligence, and Internet-based information systems, especially in the field of tourism. He earned an MS and PhD in computer science from the Technical University Vienna. He is a member of the strategic advisory board for the European research program IST, acts as the editor in chief of the journal Information

Recommender systems using collaborative filtering are a popular technique for reducing information overload and finding products to purchase. One limitation of current recommenders is that they are not portable. They can only run on large computers connected to the Internet. A second limitation is that they require the user to trust the owner of the recommender with personal preference data. Personal recommenders hold the promise of delivering high quality recommendations on palmtop computers, even when disconnected from the Internet. Further, they can protect the user's privacy by storing personal information locally, or by sharing it in encrypted form. In this article we present the new PocketLens collaborative filtering algorithm along with five peer-to-peer architectures for finding neighbors. We evaluate the architectures and algorithms in a series of offline experiments. These experiments show that Pocketlens can run on connected servers, on usually connected workstations, or on occasionally connected portable devices, and produce recommendations that are as good as the best published algorithms to date.

DEMAIS is an informal design tool that we claim helps a multimedia designer explore and communicate temporal and interactive (behavioral) design ideas better than existing tools. This paper seeks to empirically validate our claim. We report on an evaluation comparing DEMAIS to pencil and paper and Authorware for the exploration and communication of behavior in early multimedia design. The main results are that (i) DEMAIS was better than Authorware for both exploring and communicating behavior, (ii) DEMAIS was better than pencil and paper for communicating behavior, and (iii) DEMAIS was able to capture most of a designer's behavioral design ideas. Our results show that DEMAIS bridges the early investment/communication gap that exists among current multimedia design tools.

Recommender systems use people's opinions about items in an information domain to help people choose other items. These systems have succeeded in domains as diverse as movies, news articles, Web pages, and wines. The psychological literature on conformity suggests that in the course of helping people make choices, these systems probably affect users' opinions of the items. If opinions are influenced by recommendations, they might be less valuable for making recommendations for other users. Further, manipulators who seek to make the system generate artificially high or low recommendations might benefit if their efforts influence users to change the opinions they contribute to the recommender. We study two aspects of recommender system interfaces that may affect users' opinions: the rating scale and the display of predictions at the time users rate items. We find that users rate fairly consistently across rating scales. Users can be manipulated, though, tending to rate toward the prediction the system shows, whether the prediction is accurate or not. However, users can detect systems that manipulate predictions. We discuss how designers of recommender systems might react to these findings.

Recommender systems have changed the way people shop online. Recommender systems on wireless mobile devices may have the same impact on the way people shop in stores. We present our experience with implementing a recommender system on a PDA that is occasionally connected to the network. This interface helps users of the MovieLens movie recommendation service select movies to rent, buy, or see while away from their computer. The results of a nine month field study show that although there are several challenges to overcome, mobile recommender systems have the potential to provide value to their users today

In a world where the number of choices can be overwhelming, recommender systems help users find and evaluate items of interest. They do so by connecting users with information regarding the content of recommended items or the opinions of other individuals. Such systems have become powerful tools in domains such as electronic commerce, digital libraries, and knowledge management. In this paper, we address such systems and introduce a new class of recommender system called meta-recommenders. Meta-recommenders provide users with personalized control over the generation of a single recommendation list formed from a combination of rich data using multiple information sources and recommendation techniques. We discuss experiments conducted to aid in the design of interfaces for a meta-recommender in the domain of movies. We demonstrate that meta-recommendations fill a gap in the current design of recommender systems. Finally, we consider the challenges of building real-world, usable meta-recommenders across a variety of domains.

Collaborative filtering has proven to be valuable for recommending items in many different domains. In this paper, we explore the use of collaborative filtering to recommend research papers, using the citation web between papers to create the ratings matrix. Specifically, we tested the ability of collaborative filtering to recommend citations that would be suitable additional references for a target research paper. We investigated six algorithms for selecting citations, evaluating them through offline experiments against a database of over 186,000 research papers contained in ResearchIndex. We also performed an online experiment with over 120 users to gauge user opinion of the effectiveness of the algorithms and of the utility of such recommendations for common research tasks. We found large differences in the accuracy of the algorithms in the offline experiment, especially when balanced for coverage. In the online experiment, users felt they received quality recommendations, and were enthusiastic about the idea of receiving recommendations in this domain.

Recommender systems have become valuable resources for users seeking intelligent ways to search through the enormous volume of information available to them. One crucial unsolved problem for recommender systems is how best to learn about a new user. In this paper we study six techniques that collaborative filtering recommender systems can use to learn about new users. These techniques select a sequence of items for the collaborative filtering system to present to each new user for rating. The techniques include the use of information theory to select the items that will give the most value to the recommender system, aggregate statistics to select the items the user is most likely to have an opinion about, balanced techniques that seek to maximize the expected number of bits learned per presented item, and personalized techniques that predict which items a user will have an opinion about. We study the techniques thru offline experiments with a large pre-existing user data set, and thru a live experiment with over 300 users. We show that the choice of learning technique significantly affects the user experience, in both the user effort and the accuracy of the resulting predictions.

We present PolyLens, a new collaborative filtering recommender system designed to recommend items for groups of users, rather than for individuals. A group recommender is more appropriate and useful for domains in which several people participate in a single activity, as is often the case with movies and restaurants. We present an analysis of the primary design issues for group recommenders, including questions about the nature of groups, the rights of group members, social value functions for groups, and interfaces for displaying group recommendations. We then report on our PolyLens prototype and the lessons we learned from usage logs and surveys from a nine-month trial that included 819 users. We found that users not only valued group recommendations, but were willing to yield some privacy to get the benefits of group recommendations. Users valued an extension to the group recommender system that enabled them to invite non-members to participate, via email.

To create an innovative interactive multimedia application, a multimedia designer needs to rapidly explore numerous behavioral design ideas early in the design process, as creating innovative behavior is the cornerstone of creating innovative multimedia. Current tools and techniques do not support a designer's need for early behavior exploration, limiting her ability to rapidly explore and effectively communicate behavioral design ideas. To address this need, we have developed a sketch-based, interactive multimedia storyboard tool that uses a designer's ink strokes and textual annotations as an input design vocabulary. By operationalizing this vocabulary, the tool transforms an otherwise static sketch into a working example. The behavioral sketch can be quickly edited using gestures and an expressive visual language. By enabling a designer to explore and communicate behavioral design ideas using working examples early in the design process, our tool facilitates the creation of a more effective, compelling, and entertaining multimedia application.

Recommender systems apply knowledge discovery techniques to the problem of making personalized recommendations for information, products or services during a liveinteraction. These systems, especially the k-nearest neighbor collaborative ltering based ones, are achieving widespread success on the Web. The tremendous growth in the amountofavailable information and the number of visitors to Web sites in recentyears poses some key challenges for recommender systems. These are: producing high quality recommendations, performing many recommendations per second for millions of users and items and achieving high coverage in the face of data sparsity. In traditional collaborative ltering systems the amountofwork increases with the number of participants in the system. New recommender system technologies are needed that can quickly produce high quality recommendations, even for very large-scale problems. To address these issues we have explored item-based collaborative ltering techniques. Item-based techniques rst analyze the user-item matrix to identify relationships between dierent items, and then use these relationships to indirectly compute recommendations for users. In this paper we analyze dierent item-based recommendation generation algorithms. Welookinto dierenttechniques for computing item-item similarities (e.g., item-item correlation vs. cosine similarities between item vectors) and dierenttechniques for obtaining recommendations from them (e.g., weighted sum vs. regression model). Finally, weexperimentally evaluate our results and compare them to the basic k-nearest neighbor approach. Our experiments suggest that item-based algorithms provide dramatically better performance than user-based algorithms, while at the same time providing better quality than th...

Automated collaborative filtering (ACF) systems predict a person's affinity for items or information by connecting that person's recorded interests with the recorded interests of a community of people and sharing ratings between like-minded persons. However, current recommender systems are black boxes, providing no transparency into the working of the recommendation. Explanations provide that transparency, exposing the reasoning and data behind a recommendation. In this paper, we address explanation interfaces for ACF systems - how they should be implemented and why they should be implemented. To explore how, we present a model for explanations based on the user's conceptual model of the recommendation process. We then present experimental results demonstrating what components of an explanation are the most compelling. To address why, we present experimental evidence that shows that providing explanations can improve the acceptance of ACF systems. We also describe some initial explorations into measuring how explanations can improve the filtering performance of users.

Recommender systems apply statistical and knowledge discovery techniques to the problem of making product recommendations during a live customer interaction and they are achieving widespread success in E-Commerce nowadays. In this paper, we investigate several techniques for analyzing large-scale purchase and preference data for the purpose of producing useful recommendations to customers. In particular, we apply a collection of algorithms such as traditional data mining, nearest-neighbor collaborative filtering, and dimensionality reduction on two different data sets. The first data set was derived from the web-purchasing transaction of a large E-commerce company whereas the second data set was collected from MovieLens movie recommendation site. For the experimental purpose, we divide the recommendation generation process into three sub processes - representation of input data, neighborhood formation, and recommendation generation. We devise different techniques for different sub processes and apply their combinations on our data sets to compare for recommendation quality and performance.

As users continue offloading more control and responsibility to the computer, coordinating the asynchronous interactions between the user and computer is becoming increasingly important. Without proper coordination, an application attempting to gain the user's attention risks interrupting the user in the midst of performing another task. To justify why an application should avoid interrupting the user whenever possible, we designed an experiment measuring the disruptive effect of an interruption on a user's task performance. The experiment utilized six Web-based task categories and two categories of interruption tasks. The results of the experiment demonstrate that: (i) a user performs slower on an interrupted task than a non-interrupted task, (ii) the disruptive effect of an interruption differs as a function of the task category, and (iii) different interruption tasks cause similar disruptive effects on task performance. These results empirically validate the need to better coordinate user interactions among applications that are competing for the user's attention.

Recommender systems are changing from novelties used by a few E-commerce sites, to serious business tools that are re-shaping the world of E-commerce.Many of the largest commerce Web sites are already using recommender systems to help their customers find products to purchase.A recommender system learns from a customer and recommends products that she will find most valuable from among the available products.In this paper we present an explanation of how recommender systems help Ecommerce sites increase sales, and analyze six sites that use recommender systems including several sites that use more than one recommender system.Based on the examples, we create a taxonomy of recommender systems, including the interfaces they present to customers, the technologies used to create the recommendations, and the inputs they need from customers.We conclude with ideas for new applications of recommender systems to E-commerce.

% & ' ( ) * !+ , -) ./ 0 1 32 -4 5 6 1 & ) * 7 98 ": <; '= ?> <; ": A@ B DC &E )F HG I8 KJ L 9E )F NM O7 "8 QP RE 6S T> AU B <V W8 ":

Collaborative filtering systems help address information overload by usingthe opinions of users in a community to make personal recommendations fordocuments to each user. Many collaborative filtering systems have few useropinions relative to the large number of documents available. This sparsityproblem can reduce the utility of the filtering system by reducing thenumber of documents for which the system can make recommendations andadversely affecting the quality of recommendations. This paper defines andimplements a model for integrating content-based ratings into acollaborative filtering system. The filterbot model allows collaborativefiltering systems to address sparsity by tapping the strength of contentfiltering techniques. We identify and evaluate metrics for assessing theeffectiveness of filterbots specifically, and filtering system enhancementsin general. Finally, we experimentally validate the filterbot approach byshowing that even simple filterbots such as spell checking can increase theutility for users of sparsely populated collaborative filtering systems.Keywords Collaborative filtering, information filtering, content analysis,recommendation systems, social filtering, GroupLens Research, informationfiltering agents.

This paper presents the results of introducing an agent into a real-world work situation -production of the online edition of a daily newspaper.Quantitative results show that agents helped users accomplish the task more rapidly without increasing user error and that users consistently underestimated the quality of their own performance.Qualitative results show that users accepted agents rapidly and that they unknowingly altered their working styles to adapt to the agent.

Creating innovative interactive multimedia presentations requires a great deal of time, skill, and effort. We have developed a multimedia synchronization toolkit, called Nsync (pronounced 'in-sync'), to address the complicated issues inherent in designing flexible, interactive multimedia presentations. The toolkit consists of two major components, a declarative synchronization definition language and a run-time presentation management system. The synchronization definition language supports the specification of synchronous interaction, asynchronous interaction, and fine-grained relationships among multiple media objects. The language also supports the combination of interactive behavior and fine-grained relationships through the use of conjunctive and disjunctive operators. We illustrate the use of the language through several examples. We argue that with the inclusion of asynchronous interaction, pre-computed playout schedules are too inflexible, thus we have developed a new presentat...

Serial periodic data exhibit both serial and periodic properties. For example, time continues forward serially, but weeks, months, and years are periods that recur. While there are extensive visualization techniques for exploring serial data, and a few for exploring periodic data, no existing technique simultaneously displays serial and periodic attributes of a data set. We introduce a spiral visualization technique, which displays data along a spiral to highlight serial attributes along the spiral axis and periodic ones along the radii. We show several applications of the spiral visualization to data exploration tasks, present our implementation, discuss the capacity for data analysis, and present findings of our informal study with users in data-rich scientific domains. Keywords Information Visualization, Spiral, Data Visualization, Interactive Data Exploration, Serial Periodic Data. INTRODUCTION Information visualization is the practice of mapping data sets into visual media for...

In information visualization, as the volume and complexity of the data increases, researchers require more powerful visualization tools that enable them to more effectively explore multidimensional datasets. In this paper, we discuss the general utility of a novel visualization spreadsheet framework. Just as a numerical spreadsheetenables exploration of numbers, a visualization spreadsheet enables exploration of visual forms of information. We show that the spreadsheet approach facilitates certain information visualization tasks that are more difficult using other approaches. Unlike traditional spreadsheets, which store only simple data elements and formulas in each cell, a visualization spreadsheetcell can hold anentire complex data set, selection criteria, viewing specifications, and other information needed for a full-fledged information visualization. Similarly, inter-cell operations are far more complex, stretching beyond simple arithmetic and string operations to encompass a range of domain-specific operators. We have built two prototype systems that illustrate some of these research issues. The underlying approach in our work allows domain experts to define new data types and data operations, and enables visualization experts to incorporate new visualizations, viewing parameters, and view operations. 1

This full-day workshop will bring together researchers and practitioners to explore techniques for integrating personal and community recommendations into CSCW systems. Personal recommendations are tailored to an individual user, while community recommendations reflect the values or tastes of a broader community of users.

Multimedia presentation technology has enormous potential for a myriad of applications including academic classrooms, industrial training, and business presentations. As presentation technology advances, it is possible to incorporate a wider range of media including variable duration media such as simulations and animations. At the same time, users are able to take more control over presentations by controlling the rate and selection of media being played. To make full use of these advances, multimedia systems must support flexible presentations that incorporate many variations in the way they are played. This paper identifies three requirements for flexible presentations and derives four requirements for synchronization of flexible presentations. The paper presents flexible interactive presentation synchronization (FLIPS), a model for specifying coarse synchronization for flexible presentations. FLIPS supports a wide range of temporal synchronization specifications. It also provides algorithms for attaining a consistent and coherent presentation state in response to user interaction (e.g. skipping to a different slide or selection) and other state-changing events. Applications of the FLIPS model are discussed.

No abstract available.

PICASSO is a graphical user interface development en-vironment built using the Common Lisp Object System (CLOS). This paper describes how CLOS features in-cluding multiple inheritance, instance methods, multi-methods, and method combinations were used to imple-ment the system. In addition, the benefits and draw-backs of CLOS development are discussed including code quality, maintainability and performance. 1.