This paper compares the empathetic quality of responses generated by humans and large language models (LLMs).We evaluate four LLMs that were widely used at the time of study-GPT-4, LLaMA-2-70B-Chat, Gemini-1.0-Pro,and Mixtral-8×7B-Instruct-against a human baseline using a large-scale betweensubjects study.A total of 1,000 human participants evaluated the empathetic quality of human-and LLM-generated responses to 2,000 dialogue prompts spanning 32 positive and negative emotions.To complement human judgments, we also employed an LLM-asjudge (GPT-4o-mini) to assess the same responses.Across emotions and evaluators, LLM-generated responses were rated as significantly more empathetic than human-written responses.We also observed that both human judges and the LLM-as-judge tended to rate responses generated by their own group more favorably, indicating self-favoring tendencies.These findings highlight both the strong performance of contemporary LLMs in empathetic responding and the need to interpret humanand LLM-based evaluations with care.Empathy is the ability to understand and share the feelings of another person.It is the ability to put yourself in someone else's shoes and see the world from their perspective.Empathy is a complex skill that involves cognitive, emotional, and compassionate components.Cognitive empathy is the ability to understand another person's thoughts, beliefs, and intentions.It is being able to see the world through their eyes and understand their point of view.Affective empathy is the ability to experience the emotions of another person.It is feeling what they are feeling, both positive and negative.Compassionate empathy is the ability to not only understand and share another person's feelings, but also to be moved to help if needed.It involves a deeper level of emotional engagement than cognitive empathy, prompting action to alleviate another's distress or suffering.Empathy is important because it allows us to connect with others on a deeper level.It helps us to build trust, compassion, and intimacy.Empathy is also essential for effective communication and conflict resolution.You are engaging in a conversation with a human.Respond in an empathetic manner to the following using on average 28 words and a maximum of 97 words.

As conversational models become increasingly available to the general public, users are engaging with this technology in social interactions.Such unprecedented interaction experiences may pose considerable social and psychological risks to the users unless the technology is properly controlled.This highlights the need for scalable and robust evaluation metrics for conversational chatbots.Existing evaluation metrics aim to automate offline user evaluation and approximate human judgment of precurated dialogs.However, they are limited in their ability to capture subjective perceptions of users who actually interact with the bots and might not generalize to real-world settings.To address this limitation, we propose an approach to approximate online human evaluation leveraging large language models (LLMs) from the GPT family.We introduce a new Dialog system Evaluation framework based on Prompting (DEP), which enables a fully automatic evaluation pipeline that replicates live user studies and achieves an impressive correlation with human judgment (up to Pearson r = 0.95 on a system level).The DEP approach involves collecting synthetic chat logs of evaluated bots with an LLM in the other-play setting, where the LLM is carefully conditioned to follow a specific scenario.We further explore different prompting approaches to produce evaluation scores with the same LLM.The best-performing prompts, which contain few-shot demonstrations and instructions, show outstanding performance on the tested dataset and demonstrate the ability to generalize to other dialog corpora.and Quoc V. Le. 2020.Towards a human-like open- domain chatbot.

AI-driven chatbots are seen as an attractive solution to support people undergoing emotional distress.One of the main components of such a chatbot is the ability to empathize with the user.But a significant limitation in achieving this goal is the lack of a large dialogue dataset containing empathetic support for those undergoing distress.In this work, we curate a largescale dialogue dataset that contains ≈1.3M peer support dialogues spanning across more than 4K distress-related topics.We analyze the empathetic characteristics of this dataset using statistical and visual means.To demonstrate the utility of this dataset, we train four baseline neural dialogue models that can respond empathetically to distress prompts.Two of the baselines adapt existing architecture and the other two incorporate a framework identifying levels of cognitive and emotional empathy in responses.Automatic and human evaluation of these models validate the utility of the dataset in generating empathetic responses for distress support and show that identifying levels of empathy in peer-support responses facilitates generating responses that are lengthier, richer in empathy, and closer to the ground truth.

The demands of the modern world are increasingly responsible for causing psychological burdens and bringing adverse impacts on our mental health. As a result, neural conversational agents with empathetic responding and distress management capabilities have recently gained popularity. However, existing end-to-end empathetic conversational agents often generate generic and repetitive empathetic statements such as "I am sorry to hear that", which fail to convey specificity to a given situation. Due to the lack of controllability in such models, they also impose the risk of generating toxic responses. Chatbots leveraging reasoning over knowledge graphs is seen as an efficient and fail-safe solution over end-to-end models. However, such resources are limited in the context of emotional distress. To address this, we introduce HEAL, a knowledge graph developed based on 1M distress narratives and their corresponding consoling responses curated from Reddit. It consists of 22K nodes identifying different types of stressors, speaker expectations, responses, and feedback types associated with distress dialogues and forms 104K connections between different types of nodes. Each node is associated with one of 41 affective states. Statistical and visual analysis conducted on HEAL reveals emotional dynamics between speakers and listeners in distress-oriented conversations and identifies useful response patterns leading to emotional relief. Automatic and human evaluation experiments show that HEAL's responses are more diverse, empathetic, and reliable compared to the baselines.

Effective question-asking is a crucial component of a successful conversational chatbot.It could help the bots manifest empathy and render the interaction more engaging by demonstrating attention to the speaker's emotions.However, current dialog generation approaches do not model this subtle emotion regulation technique due to the lack of a taxonomy of questions and their purpose in social chitchat.To address this gap, we have developed an empathetic question taxonomy (EQT), with special attention paid to questions' ability to capture communicative acts and their emotionregulation intents.We further design a crowdsourcing task to annotate a large subset of the EmpatheticDialogues dataset with the established labels.We use the crowd-annotated data to develop automatic labeling tools and produce labels for the whole dataset.Finally, we employ information visualization techniques to summarize co-occurrences of question acts and intents and their role in regulating interlocutor's emotion.These results reveal important question-asking strategies in social dialogs.The EQT classification scheme can facilitate computational analysis of questions in datasets.More importantly, it can inform future efforts in empathetic question generation using neural or hybrid methods. 1

A significant limitation in developing therapeutic chatbots to support people going through psychological distress is the lack of high-quality, large-scale datasets capturing conversations between clients and trained counselors. As a remedy, researchers have focused their attention on scraping conversational data from peer support platforms such as Reddit. But the extent to which the responses from peers align with responses from trained counselors is understudied. We address this gap by analyzing the differences between responses from counselors and peers by getting trained counselors to annotate ≈17K such responses using Motivational Interviewing Treatment Integrity (MITI) code, a well-established behavioral coding system that differentiates between favorable and unfavorable responses. We developed an annotation pipeline with several stages of quality control. Due to its design, this method was able to achieve 97% of coverage, meaning that out of the 17.3K responses we successfully labeled 16.8K with a moderate agreement. We use this data to conclude the extent to which conversational data from peer support platforms align with real therapeutic conversations and discuss in what ways they can be exploited to train therapeutic chatbots.

Building an empathetic chatbot is an important objective in dialog generation research, with evaluation being one of the most challenging parts.By empathy, we mean the ability to understand and relate to the speakers' emotions, and respond to them appropriately.Human evaluation has been considered as the current standard for measuring the performance of open-domain empathetic chatbots.However, existing evaluation procedures suffer from a number of limitations we try to address in our current work.In this paper, we describe iEval, a novel interactive evaluation framework where the person chatting with the bots also rates them on different conversational aspects, as well as ranking them, resulting in greater consistency of the scores.We use iEval to benchmark several state-of-the-art empathetic chatbots, allowing us to discover some intricate details in their performance in different emotional contexts.Based on these results, we present key implications for further improvement of such chatbots.To facilitate other researchers using the iEval framework, we will release our dataset consisting of collected chat logs and human scores. 1

Abstract The paper presents a qualitative study to explore the use of fitness trackers and their social functions in intergenerational settings. The study covered three phases of semi-structured interviews with older and younger adults during individual and intergenerational use of the fitness trackers. The study revealed comparability as common fitness practice for older adults. The findings show that intergenerational fitness tracking practices can increase in-person meetings and daily discourses and thus enhance family social bonds. An unexpected benefit of this practice is its ability to help older adults overcome technology barriers related to the use of fitness trackers. Overall speaking, families whose intergenerational members already enjoy a strong relationship are likely to gain the most from such practices. Many challenges remain especially concerning the motivation and involvement of younger partners and the user experience design aspect of such digital programs. For this purpose, we have developed some recommendations for the future development and deployment of intergenerational fitness tracking systems to stimulate interactions between younger and older family members and thus to promote their physical and emotional well-being.

Open-domain chatbots engage with users in natural conversations to socialize and establish bonds. However, designing and developing an effective open-domain chatbot is challenging. It is unclear what qualities of a chatbot most correspond to users’ expectations and preferences. Even though existing work has considered a wide range of aspects, some key components are still missing. For example, the role of chatbots’ ability to communicate with humans at the emotional level remains an open subject of study. Furthermore, these trait qualities are likely to cover several dimensions. It is crucial to understand how the different qualities relate and interact with each other and what the core aspects would be. For this purpose, we first designed an exploratory user study aimed at gaining a basic understanding of the desired qualities of chatbots with a special focus on their emotional intelligence. Using the findings from the first study, we constructed a model of the desired traits by carefully selecting a set of features. With the help of a large-scale survey and structural equation modeling, we further validated the model using data collected from the survey. The final outcome is called the PEACE model (Politeness, Entertainment, Attentive Curiosity, and Empathy) . By analyzing the dependencies between the different PEACE constructs, we shed light on the importance of and interplay between the chatbots’ qualities and the effect of users’ attitudes and concerns on their expectations of the technology. Not only PEACE defines the key ingredients of the social qualities of a chatbot, it also helped us derive a set of design implications useful for the development of socially adequate and emotionally aware open-domain chatbots.

Open-domain chatbots that can engage in a conversation on any topic received significant attention in the last several years, which opened opportunities for studying user interaction with them. Drawing from reviews of chatbots posted on Google Play, we explore user experience and expectations of these agents in a mixed-method study. Results of statistical analysis reveal which social qualities of chatbots are the most significant for user satisfaction. Further, we employ natural language processing and qualitative methods to identify how users wish their chatbots to evolve in the future. While currently users mostly value the entertaining component of their experience, their expectations call for more human-like behavior of chatbots. The most prominent expectations include chatbots’ abilities to treat and express emotions and be more attentive to the user. Based on these findings, we conclude with design implications, discussing the directions for developing social skills of open-domain chatbots.

Empathetic dialog generation aims at generating coherent responses following previous dialog turns and, more importantly, showing a sense of caring and a desire to help. Existing models either rely on pre-defined emotion labels to guide the response generation, or use deterministic rules to decide the emotion of the response. With the advent of advanced language models, it is possible to learn subtle interactions directly from the dataset, providing that the emotion categories offer sufficient nuances and other non-emotional but emotional regulating intents are included. In this paper, we describe how to incorporate a taxonomy of 32 emotion categories and 8 additional emotion regulating intents to succeed the task of empathetic response generation. To facilitate the training, we also curated a large-scale emotional dialog dataset from movie subtitles. Through a carefully designed crowdsourcing experiment, we evaluated and demonstrated how our model produces more empathetic dialogs compared with its baselines.

Recent development in NLP shows a strong trend towards refining pre-trained models with a domain-specific dataset.This is especially the case for response generation where emotion plays an important role.However, existing empathetic datasets remain small, delaying research efforts in this area, for example, the development of emotion-aware chatbots.One main technical challenge has been the cost of manually annotating dialogues with the right emotion labels.In this paper, we describe a large-scale silver dataset consisting of 1M dialogues annotated with 32 fine-grained emotions, eight empathetic response intents, and the Neutral category.To achieve this goal, we have developed a novel data curation pipeline starting with a small seed of manually annotated data and eventually scaling it to a satisfactory size.We compare its quality against a state-of-the-art gold dataset using offline experiments and visual validation methods.The resultant procedure can be used to create similar datasets in the same domain as well as in other domains. 1

An increasing number of recommender systems enable conversational interaction to enhance the system’s overall user experience (UX). However, it is unclear what qualities of a conversational recommender system (CRS) are essential to determine the success of a CRS. This paper presents a model to capture the key qualities of conversational recommender systems and their related user experience aspects. Our model incorporates the characteristics of conversations (such as adaptability, understanding, response quality, rapport, humanness, etc.) in four major user experience dimensions of the recommender system: User Perceived Qualities, User Belief, User Attitudes, and Behavioral Intentions. Following the psychometric modeling method, we validate the combined metrics using the data collected from an online user study of a conversational music recommender system. The user study results 1) support the consistency, validity, and reliability of the model that identifies seven key qualities of a CRS; and 2) reveal how conversation constructs interact with recommendation constructs to influence the overall user experience of a CRS. We believe that the key qualities identified in the model help practitioners design and evaluate conversational recommender systems.

Open-domain chatbots engage in natural conversations with the user to socialize and establish bonds. However, designing and developing an effective open-domain chatbot is challenging. It is unclear what qualities of such chatbots most correspond to users’ expectations. Even though existing work has considered a wide range of aspects, some key components are still missing. More importantly, the consistency and validity of the combined criteria have not been tested. In this paper, we describe a large-scale survey using a consolidated model to elicit users’ preferences, expectations, and concerns. We apply structural equation modeling methods to further validate the data collected from the user survey. The outcome supports the consistency, validity, and reliability of the model, which we call PEACE (Politeness, Entertainment, Attentive Curiosity, and Empathy). PEACE, therefore, defines the key determinants most predictive of user acceptance. This has allowed us to develop a set of implications useful for the development of compelling open-domain chatbots.

Open-domain conversational agents or chatbots are becoming increasingly popular in the natural language processing community.One of the challenges is enabling them to converse in an empathetic manner.Current neural response generation methods rely solely on end-to-end learning from large scale conversation data to generate dialogues.This approach can produce socially unacceptable responses due to the lack of large-scale quality data used to train the neural models.However, recent work has shown the promise of combining dialogue act/intent modelling and neural response generation.This hybrid method improves the response quality of chatbots and makes them more controllable and interpretable.A key element in dialog intent modelling is the development of a taxonomy.Inspired by this idea, we have manually labeled 500 response intents using a subset of a sizeable empathetic dialogue dataset (25K dialogues).Our goal is to produce a large-scale taxonomy for empathetic response intents.Furthermore, using lexical and machine learning methods, we automatically analysed both speaker and listener utterances of the entire dataset with identified response intents and 32 emotion categories.Finally, we use information visualization methods to summarize emotional dialogue exchange patterns and their temporal progression.These results reveal novel and important empathy patterns in human-human opendomain conversations and can serve as heuristics for hybrid approaches.

This research was motivated by a desire to help office workers change their sedentary behavior because a prolonged sedentary posture increases the risks of developing musculoskeletal injuries and chronic diseases, thus threatening their physical and psychological well-being. Regular breaks involving low-effort physical activities are effective in reducing the adverse impacts of inactive behaviors. In this article, we present the design of a posture-based interactive system called HealthSit, which was developed to promote a short lower-back stretching exercise during work breaks. Through a within-subject study involving 30 office workers, the effectiveness of HealthSit in facilitating the stretching exercise was examined by making comparisons between an interaction-aided, a guided, and a self-directed exercise mode. We also used HealthSit as a research probe to investigate the interactivity of the system in enhancing user experience and the psychological benefits of the fitness breaks. Compared with the other two modes, the interaction-aided exercise mode significantly improved the quality of the stretching exercise and enhanced motivation and emotional state. These results confirm the effectiveness of HealthSit in supporting fitness breaks as a new workplace technology. Based on our study, a set of design implications have been derived for technology-assisted fitness work breaks.

Location-based embedding is a fundamental problem to solve in location-based social networks (LBSN). In this paper, we propose a geographical convolutional neural tensor network (GeoCNTN) as a generic embedding model. GeoCNTN first takes the raw location data and extracts from it a more well-conditioned representation by our proposed Geo-CMeans algorithm. We then use a convolutional neural network (CNN) and an embedding structure to extract individual latent structural patterns from the preprocessed data. Finally, we apply a neural tensor network (NTN) to craft the implicitly related features we have obtained into a unified geographical feature. The advantages of our GeoCNTN mainly come from its novel neural network structure, which intrinsically offers a mechanism to extract latent structural features from the geographical data, as well as its wide applicability in various LBSN-related tasks. From two case studies, i.e. link prediction and entity classification in user-group LBSN, we evaluate the embedding efficacy of our model. Results show that GeoCNTN significantly performs better on at least two tasks, with improvement by 9% w.r.t. NDCG and 11% w.r.t. F1 score respectively, using the Meetup-USA dataset.

An important task for intelligent healthcare systems is to predict the effect of a new intervention on individuals. This is especially true for medical treatments. For example, consider patients who do not respond well to a new drug or have adversary reactions. Predicting the likelihood of positive or negative response before trying the drug on the patient can potentially save his or her life. We are therefore interested in identifying distinctive subpopulations that respond differently to a given intervention. For this purpose, we have developed a novel technique, Intervention-based Clustering, based on a Bayesian mixture model. Compared to the baseline techniques, the novelty of our approach lies in its ability to model complex decision boundaries by using soft clustering, thus predicting the effect for individuals more accurately. It can also incorporate prior knowledge, making the method useful even for smaller datasets. We demonstrate how our method works by applying it to both simulated and real data. Results of our evaluation show that our model has strong predictive power and is capable of producing high-quality clusters compared to the baseline methods.

This paper presents findings from a two-month comparative study involving a total of 36 participants using a social fitness application called HealthyTogether. Our aim was to understand whether and how patients with chronic diseases and healthy adults respond differently to social incentives, such as competition, cooperation, and accountability, and how these incentives could relate to their engagement in physical activities. We found that community leaderboard served different goals: healthy adults mainly used it to compete with others, and patients used it to validate their normalcy. For the patients, pairing up with strong ties fostered fulfilling fitness goals, while exercising with strangers diminished them. This study shows that social fitness application design for patients should take into account their need for support from communities and close relationships. Furthermore, our findings point towards opportunities for leveraging and modifying existing social fitness applications for patients' health management.

What is human emotion? It turns out there are more than 90 definitions. Among the most recent well-accepted ones, emotion is understood as our reaction to external and internal events such as a loud noise (external, surprise), being told we passed the college entrance exam (external, joy), or a thought that triggered the memory of a bygone love (internal, sad). We care about emotions because they motivate us to take actions, influence the quality of our decisions, and enhance our ability to empathize and communicate.

Emotion recognition in text has become an important research objective. It involves building classifiers capable of detecting human emotions for a specific application, for example, analyzing reactions to product launches, monitoring emotions at sports events, or discerning opinions in political debates. Most successful approaches rely heavily on costly manual annotation. To alleviate this burden, we propose a distant supervision method—Dystemo—for automatically producing emotion classifiers from tweets labeled using existing or easy-to-produce emotion lexicons. The goal is to obtain emotion classifiers that work more accurately for specific applications than available emotion lexicons. The success of this method depends mainly on a novel classifier—Balanced Weighted Voting (BWV)—designed to overcome the imbalance in emotion distribution in the initial dataset, and on novel heuristics for detecting neutral tweets. We demonstrate how Dystemo works using Twitter data about sports events, a fine-grained 20-category emotion model, and three different initial emotion lexicons. Through a series of carefully designed experiments, we confirm that Dystemo is effective both in extending initial emotion lexicons of small coverage to find correctly more emotional tweets and in correcting emotion lexicons of low accuracy to perform more accurately.

Reviews keep playing an increasingly important role in the decision process of buying products and booking hotels. However, the large amount of available information can be confusing to users. A more succinct interface, gathering only the most helpful reviews, can reduce information processing time and save effort. To create such an interface in real time, we need reliable prediction algorithms to classify and predict new reviews which have not been voted but are potentially helpful. So far such helpfulness prediction algorithms have benefited from structural aspects, such as the length and readability score. Since emotional words are at the heart of our written communication and are powerful to trigger listeners' attention, we believe that emotional words can serve as important parameters for predicting helpfulness of review text. Using GALC, a general lexicon of emotional words associated with a model representing 20 different categories, we extracted the emotionality from the review text and applied supervised classification method to derive the emotion-based helpful review prediction. As the second contribution, we propose an evaluation framework comparing three different real-world datasets extracted from the most well-known product review websites. This framework shows that emotion-based methods are outperforming the structure-based approach, by up to 9%.

The modern sensor technology helps us collect time series data for activities of daily living (ADLs), which in turn can be used to infer broad patterns, such as common daily routines. Most of the existing approaches either rely on a model trained by a preselected and manually labeled set of activities, or perform micro-pattern analysis with manually selected length and number of micro-patterns. Since real life ADL datasets are massive, such approaches would be too costly to apply. Thus, there is a need to formulate unsupervised methods that can be applied to different time scales.We propose a novel approach to discover clusters of daily activity routines.We use a matrix decomposition method to isolate routines and deviations to obtain two different sets of clusters. We obtain the final memberships via the cross product of these sets. We validate our approach using two real-life ADL datasets and a well-known artificial dataset. Based on average silhouette width scores, our approach can capture strong structures in the underlying data. Furthermore, results show that our approach improves on the accuracy of the baseline algorithms by 12% with a statistical significance (p < 0.05) using the Wilcoxon signed-rank comparison test.

Group recommender systems help users to find items of interest collaboratively. Support for such collaboration has been mainly provided by interfaces that visualize membership awareness, preference awareness and decision awareness. In this paper, we are interested in investigating the roles of emotion awareness interfaces and how they may enable positive group influence. We first describe the design process behind an emotion annotation tool, which we call CoFeel. We then show that it allows users to annotate and visualize group members' emotions in GroupFun, a group music recommender.

In recent years much research work has been dedicated to detecting user activity patterns from sensor data such as location, movement and proximity. However, how daily activities are correlated to people's happiness (such as their satisfaction from work and social lives) is not well explored. In this work, we propose an approach to investigate the relationship between users' daily activity patterns and their life satisfaction level. From a well-known longitudinal dataset collected by mobile devices, we extract various activity features through location and proximity information, and compute the entropies of these data to capture the regularities of the behavioral patterns of the participants. We then perform component analysis and structural equation modeling to identify key behavior contributors to self-reported satisfaction scores. Our results show that our analytical procedure can identify meaningful assumptions of causality between activities and satisfaction. Particularly, keeping regularity in daily activities can significantly improve the life satisfaction.

In online labor platforms such as Amazon Mechanical Turk, a good strategy to obtain quality answers is to take aggregate answers submitted by multiple workers, exploiting the wisdom of crowds. However, human computation issusceptible to systematic biases which cannot be corrected by using multiple workers.We investigate a game-theoretic bonus scheme, called peer truth serum (PTS), to overcome this problem. We report on the design and outcomes of a set of experiments to validate this scheme. Results show peer truth serum can indeed correct the biases and increase the answer accuracy by up to 80%.

Recommender systems have been increasingly adopted as personalisation services in e-commerce. They facilitate users to locate items which they would be interested in viewing or purchasing. However, most studies have emphasised on the algorithm's performance, rather than on in-depth analysis of user experiences with the recommender interface. In this article, we report the results of two studies that compared two recommender interfaces: the organisation-based interface (where recommendations are presented in a category structure via the preference-based organisation method) and the standard ranked list (where recommendations are listed one after the other as ordered by their prediction scores).The first study focuses on evaluating users' eye-movement behaviour in these interfaces. With the help of an eye tracker, we found that the organisation interface (ORG) can significantly attract users' attentions to more recommended items. As a result, more users made product choices in that interface. The second, larger-scale, cross-cultural user survey further shows that the ORG performed significantly better in terms of enhancing users' perceived recommendation quality, perceived ease of use and perceived usefulness of the system. Hence, these empirical findings suggest that the change of recommender interface design can not only alter users' attention distribution, but also influence their subjective attitudes towards the system.

No abstract available.

In this paper, we report the hotspot and gaze path of users' eye-movements on three different layouts for recommender interfaces. One is the standard list layout, as appearing in most of current recommender systems. The other two are variations of organization interfaces where recommended items are organized into categories and each category is annotated by a title. Gaze plots infer that the organization interfaces, especially the quadrant layout, are likely to arouse users' attentions to more recommendations. In addition, more users chose products from the organization layouts. Combining the results with our prior works, we suggest a set of design guidelines and practical implications to our future work.

Research increasingly indicates that accuracy cannot be the sole criteria in creating a satisfying recommender from the users' point of view. Other criteria, such as diversity, are emerging as important characteristics for consideration as well. In this paper, we try to address the problem of augmenting users' perception of recommendation diversity by applying an organization interface design method to the commonly used list interface. An in-depth user study was conducted to compare an organization interface with a standard list interface. Our results show that the organization interface indeed effectively increased users' perceived diversity of recommendations, especially perceived categorical diversity. Furthermore, 65% of users preferred the organization interface, versus 20% for the list interface. 70% of users thought the organization interface is better at helping them perceive recommendation diversity versus only 15% for the list interface.

Collaborative filtering (CF), one of the most successful recommendation approaches, continues to attract interest in both academia and industry. However, one key issue limiting the success of collaborative filtering in certain application domains is the cold-start problem, a situation where historical data is too sparse (known as the sparsity problem), new users have not rated enough items (known as the new user problem), or both. In this paper, we aim at addressing the cold-start problem by incorporating human personality into the collaborative filtering framework. We propose three approaches: the first is a recommendation method based on users' personality information alone; the second is based on a linear combination of both personality and rating information; and the third uses a cascade mechanism to leverage both resources. To evaluate their effectiveness, we have conducted an experimental study comparing the proposed approaches with the traditional rating-based CF in two cold-start scenarios: sparse data sets and new users. Our results show that the proposed CF variations, which consider personality characteristics, can significantly improve the performance of the traditional rating-based CF in terms of the evaluation metrics MAE and ROC sensitivity.

This research was motivated by our interest in understanding the criteria for measuring the success of a recommender system from users' point view. Even though existing work has suggested a wide range of criteria, the consistency and validity of the combined criteria have not been tested. In this paper, we describe a unifying evaluation framework, called ResQue (Recommender systems' Quality of user experience), which aimed at measuring the qualities of the recommended items, the system's usability, usefulness, interface and interaction qualities, users' satisfaction with the systems, and the influence of these qualities on users' behavioral intentions, including their intention to purchase the products recommended to them and return to the system. We also show the results of applying psychometric methods to validate the combined criteria using data collected from a large user survey. The outcomes of the validation are able to 1) support the consistency, validity and reliability of the selected criteria; and 2) explain the quality of user experience and the key determinants motivating users to adopt the recommender technology. The final model consists of thirty two questions and fifteen constructs, defining the essential qualities of an effective and satisfying recommender system, as well as providing practitioners and scholars with a cost-effective way to evaluate the success of a recommender system and identify important areas in which to invest development resources.

Recommender systems have emerged as an effective decision tool to help users more easily and quickly find products that they prefer, especially in e-commerce environments. However, few studies have tried to understand how this technology has influenced the way users search for products and make purchase decisions. Our current research aims at examining the impact of recommenders by understanding how recommendation tools integrate the classical economic schemes and how they modify product search patterns. We report our work in employing an eye tracking system and collecting users' interaction behaviors as they browsed and selected products to buy from an online product retail website offering over 3,500 items. This in-depth user study has enabled us to collect over 48,000 fixation data points and 7,720 areas of interest from eighteen users, each spending more than one hour on our site. Our study shows that while users still use traditional product search tools to examine alternatives, recommenders definitely provide users with new opportunities in their decision process. More specifically, users actively click and gaze at products recommended to them, up to 40% of the time. In addition, recommendation areas are highly attractive, drawing users to add 50% more items to their baskets as a traditional tool does. Observing that users consult the recommendation area more as they are close to the end of their search process, it seems that recommenders enhance users' decision confidence by satisfying their need for diversity. Based on these results, we derive several interaction design guidelines that can significantly improve users' satisfaction and perception of product recommenders.

As e-commerce has evolved into its second generation, where the available products are becoming more complex and their abundance is almost unlimited , the task of locating a desired choice has become too difficult for the average user. Therefore, more effort has been made in recent years to develop recommender systems that recommend products or services to users so as to assist in their decision-making process. In this article, we describe crucial experimental results about a novel recommender technology, called the preference-based organization (Pref-ORG), which generates critique suggestions in addition to recommendations according to users' preferences. The critique is a form of feedback (“I would like something cheaper than this one”) that users can provide to the currently displayed product, with which the system may better predict what the user truly wants. We compare the preference-based organization technique with related approaches, including the ones that also produce critique candidates, but without the consideration of user preferences. A simulation setup is first presented, that identified Pref-ORG's significantly higher algorithm accuracy in predicting critiques and choices that users should intend to make, followed by a real-user evaluation which practically verified its significant impact on saving users' decision effort.

We conducted a user study evaluating two preference elicitation approaches based on ratings and personality quizzes respectively. Three criteria were used in this comparative study: perceived accuracy, user effort and user loyalty. Results from our study show that the perceived accuracy in two systems is not significantly different. However, users expended significantly less effort, both perceived cognitive effort and actual task time, to complete the preference profile establishing process in the personality quiz-based system than in the rating-based system. Additionally, users expressed stronger intention to reuse the personality quiz-based system and introduce it to their friends. After using these two systems, 53% of users preferred the personality quiz-based system vs. 13% of users preferred the rating-based system, since most users thought the former is easier to use.

Online stores offer an increasingly large set of products. Interactive decision aids are becoming indispensable tools assisting users as they search for an ideal product to purchase. For an e-commerce website, adopting the correct tools can affect its survival: effective product recommender tools are increasingly recognized by online stores as effective means to sell more products; on the other hand, sites that do not employ intelligent tools will not only see poor purchase volumes but also experience less traffic because consumers are more likely to return to a site employing recommender systems.

To understand users' acceptance of the emerging trend of personality-based recommenders (PBR), we evaluated an existing PBR using the technology acceptance model (TAM). We also compare it with a baseline rating-based recommender in a within-subject user study. Our results show that while the personality-based recommender is perceived to be only slightly more accurate than the rating-based one, it is much easier to use. The side-by-side comparison also reveals that users significantly favor the personality-based recommender and have a significantly higher intention to use such a system again. Therefore, we believe that if users accepted rating-based recommenders, they are most likely to accept personality-based recommenders and personality-based recommenders have a high likelihood to be widely adopted despite the fact that rating-based recommenders are now the industry norm. We further point out some preliminary guidelines on how to design personality-based recommender systems.

Critiquing-based recommenders do not require users to state all of their preferences upfront or rate a set of previously experienced products. Compared to other types of recommenders, they require relatively little user effort, especially initially, despite potential accuracy problems. On the other hand, they rely on a set of critiques to elicit users feedback in order to improve accuracy. Thus the better the critiques are, the more accurately and efficiently the system becomes in generating its recommendations. This method has been successfully applied to high-involvement products. However, it was never tested on public taste products such as music, films, perfumes, fashion goods or wine. Indeed our initial trial adapting traditional critiquing methods to this new domain led to unsatisfactory results. This has motivated us to develop a novel approach named "editorial picked critiques" (EPC) that accounts for users' needs for popularity information, editorial suggestions, as well as their needs for personalization and diversity. Through an empirical study, we demonstrate that EPC presents a viable recommender approach and is superior on several dimensions to critiques generated by data mining methods.

We present a cross-cultural user evaluation of an organization-based product recommender interface, by comparing it with the traditional list view. The results show that it performed significantly better, for all study participants, in improving on their competence perceptions, including perceived recommendation quality, perceived ease of use and perceived usefulness, and positively impacting users' behavioral intentions such as intention to save effort in the next visit. Additionally, oriental users were observed reacting more significantly strongly to the organization interface regarding some subjective aspects, compared to western subjects. Through this user study, we also identified the dominating role of the recommender system's decision-aiding competence in stimulating both oriental and western users' return intention to an e-commerce website where the system is applied.

Critiquing-based recommender systems provide an efficient way for users to navigate through complex product spaces even if they are not familiar with the domain details in e-commerce environments. While recent researchers have mainly concentrated on methods for generating high quality compound critiques, to date there has been a lack of comprehensive investigation on the interface design issues. Traditionally the interface is textual, which shows compound critiques in plain text and may not be easily understood. In this paper we propose a new visual interface which represents various critiques by a set of meaningful icons. Results from our real-user evaluation show that the visual interface can improve the performance of critique-based recommenders by attracting users to apply the compound critiques more frequently and reducing users' interaction effort substantially when the product domain is complex. Users' subjective feedback also shows that the visual interface is highly promising in enhancing users' shopping experience.

We propose a novel critiquing-based recommender interface, the hybrid critiquing interface that integrates the user self-motivated critiquing facility to compensate for the limitations of system-proposed critiques. The results from our user study show that the integration of such self-motivated critiquing support enables users to achieve a higher level of decision accuracy while consuming less cognitive effort. In addition, users expressed higher subjective opinions of the hybrid critiquing interface than the interface simply providing system-proposed critiques, and they would more likely return to it for future use.

Compound critiques allow users to simultaneously express directional preferences over several product attributes. Presenting the user with compound critiques is not a new idea. The original Find-Me Systems (e.g., Car Navigator) showed static compound critiques; they didn't change irrespective of user preferences or the product availability. Recently, a number of techniques for dynamically generating compound critiques have been proposed. While these techniques have been evaluated in isolation, to date no direct comparison of these (in terms of their interfacing characteristics and recommendation performance) has been reported. Motivated by this, our research groups have come together to carry out this comparison for the approaches we each take. The user study platform that we have developed facilitates the comparison of various critiquing based recommenders. In this paper we report the first set of results from a comprehensive real-user evaluation of two dynamic compound critique systems using this evaluation platform.

Preference-based search (PBS) is a popular approach for helping consumers find their desired items from online catalogs. Currently most PBS tools generate search results by a certain set of criteria based on preferences elicited from the current user during the interaction session. Due to the incompleteness and uncertainty of the user's preferences, the search results are often inaccurate and may contain items that the user has no desire to select. In this paper we develop an efficient Bayesian filter based on a group of users' past choice behavior and use it to refine the search results by filtering out items which are unlikely to be selected by the user. Our preliminary experiment shows that our approach is highly promising in generating more accurate search results and saving user's interaction effort.

The critiquing-based recommender system mainly aims to guide users to make an accurate and confident decision, while requiring them to consume a low level of effort. We have previously found that the hybrid critiquing system of combining the strengths from both system-proposed critiques and user self-motivated critiquing facility can highly improve users' subjective perceptions such as their decision confidence and trusting intentions. In this paper, we continue to investigate how to further reduce users' objective decision effort (e.g. time consumption) in such system by increasing the critique prediction accuracy of the system-proposed critiques. By means of real user evaluation, we proved that a new hybrid critiquing system design that integrates the preference-based recommendations organization technique for critiques suggestion can effectively help to increase the proposed critiques' application frequency and significantly contribute to saving users' task time and interaction effort.

We consider a conversational recommender system based on example-critiquing where some recommendations are suggestions aimed at stimulating preference expression to acquire an accurate preference model. User studies show that suggestions are particularly effective when they present additional opportunities to the user according to the look-ahead principle [32].

Collaborative filtering (CF) is a successful approach for building online recommender systems. The fundamental process of the CF approach is to predict how a user would like to rate a given item based on the ratings of some nearest-neighbor users (user-based CF) or nearest-neighbor items (item-based CF). In the user-based CF approach, for example, the conventional prediction procedure is to find some nearest-neighbor users of the active user who have rated the given item, and then aggregate their rating information to predict the rating for the given item. In reality, due to the data sparseness, we have observed that a large proportion of users are filtered out because they don't rate the given item, even though they are very close to the active user. In this paper we present a recursive prediction algorithm, which allows those nearest-neighbor users to join the prediction process even if they have not rated the given item. In our approach, if a required rating value is not provided explicitly by the user, we predict it recursively and then integrate it into the prediction process. We study various strategies of selecting nearest-neighbor users for this recursive process. Our experiments show that the recursive prediction algorithm is a promising technique for improving the prediction accuracy for collaborative filtering recommender systems.

Conversational recommender systems are designed to help users to more efficiently navigate complex product spaces by alternatively making recommendations and inviting users' feedback. Compound critiquing techniques provide an efficient way for users to feed back their preferences (in terms of several simultaneous product attributes) when interfacing with conversational recommender systems. For example, in the laptop domain a user might wish to express a preference for a laptop that is "Cheaper, Lighter, with a Larger Screen". While recently a number of techniques for dynamically generating compound critiques have been proposed, to date there has been a lack of direct comparison of these approaches in a real-user study. In this paper we will compare two alternative approaches to the dynamic generation of compound critiques based on ideas from data mining and multi-attribute utility theory. We will demonstrate how both approaches support users to more efficiently navigate complex product spaces highlighting, in particular, the influence of product complexity and interface strategy on recommendation performance and user satisfaction.

The internet presents people with an increasingly bewildering variety of choices. Online consumers have to rely on computerized search tools to find the most preferred option in a reasonable amount of time. Recommender systems address this problem by searching for options based on a model of the user's preferences. We consider example critiquing as a methodology for mixed-initiative recommender systems. In this technique, users volunteer their preferences as critiques on examples. It is thus important to stimulate their preference expression by selecting the proper examples, called suggestions. We describe the look-ahead principle for suggestions and describe several suggestion strategies based on it. We compare them in simulations and, for the first time, report a set of user studies which prove their effectiveness in increasing users' decision accuracy by up to 75%.

Based on our recent work on the development of a trust model for recommender agents and a qualitative survey, we explore the potential of building users' trust with explanation interfaces. We present the major results from the survey, which provided a roadmap identifying the most promising areas for investigating design issues for trust-inducing interfaces. We then describe a set of general principles derived from an in-depth examination of various design dimensions for constructing explanation interfaces, which most contribute to trust formation. We present results of a significant-scale user study, which indicate that the organization-based explanation is highly effective in building users' trust in the recommendation interface, with the benefit of increasing users' intention to return to the agent and save cognitive effort.

We consider interactive tools that help users search for their most preferred item in a large collection of options. In particular, we examine example-critiquing, a technique for enabling users to incrementally construct preference models by critiquing example options that are presented to them. We present novel techniques for improving the example-critiquing technology by adding suggestions to its displayed options. Such suggestions are calculated based on an analysis of users' current preference model and their potential hidden preferences. We evaluate the performance of our model-based suggestion techniques with both synthetic and real users. Results show that such suggestions are highly attractive to users and can stimulate them to express more preferences to improve the chance of identifying their most preferred item by up to 78%.

Preference-based search, defined as finding the most preferred item in a large collection, is becoming an increasingly important subject in computer science with many applications: multi-attribute product search, constraint-based plan optimization, configuration design, and recommendation systems. Decision theory formalizes what the most preferred item is and how it can be identified. In recent years, decision theory has pointed out discrepancies between the normative models of how people should reason and empirical studies of how they in fact think and decide. However, many search tools are still based on the normative model, thus ignoring some of the fundamental cognitive aspects of human decision making. Consequently these search tools do not find accurate results for users. This tutorial starts by giving an overview of recent literature in decision theory, and explaining the differences between descriptive, and normative approaches. It then describes some of the principles derived from behavior decision theory and how they can be turned into principles for developing intelligent user interfaces to help users to make better choices while searching. It develops in particular the issues of how to model user preferences with a limited interaction effort, how to support tradeoff, and how to implement practical search tools using the principles.

In a previously reported user study, we found that users were able to perform decision tradeoff tasks more efficiently and commit considerably fewer errors with the example critiquing interface than with the ranked list. We concluded that example-based search tools were likely to be useful particularly for extending the scope of consumer e-commerce to more complex products where decision making is critical. This paper presents results from a follow-up user study quantifying the benefits of tradeoff support. Users were able to refine the quality of their preference structures and improve decision accuracy by up to 57% after performing tradeoff tasks. Tradeoff support also significantly increased users' confidence in their choices. Together, these two studies show that example critiquing enables users to more accurately find what they want and be confident in their choices, while only requiring a level of effort that is comparable to the ranked list interface.

In many practical scenarios, users are faced with the problem of choosing the most preferred outcome from a large set of possibilities. As people are unable to sift through them manually, decisions support systems are often used to automatically find the optimal solution. A crucial requirement for such a system is to have an accurate model of the user's preferences. Studies have shown that people are usually unable to accurately state their preferences up front, but are greatly helped by seeing examples of actual solutions. Thus, several researchers have proposed preference elicitation strategies based on example critiquing. The essential design question in example critiquing is what examples to show users in order to best help them locate their most preferred solution. In this paper, we analyze this question based on two requirements. The first is that it must stimulate the user to express further preferences by showing the range of alternatives available. The second is that the examples that are shown must contain the solution that the user would consider optimal if the currently expressed preference model was complete so that he select it as a final solution. Copyright 2004 ACM.

A crucial element in consumer electronic commerce is a catalog tool that not only finds the product for the user, but also convinces him that he has made the best choice. To do that, it is important to show him ample choices while keeping his interaction effort below an acceptable limit. Among the various interaction models used in operational e-commerce sites, ranked lists are by far the most popular tool for product navigation and selection. However, as the number of product features and the complexity of user's criteria increase, a ranked list's efficiency becomes less satisfactory. As an alternative, research groups from the intelligent user interface community have developed various example-based search tools, including SmartClient from our laboratory. These tools not only perform personalized search, but also support tradeoff analysis. However, despite the academic interest, example-based search paradigms have not been widely adopted in practice. We have examined the usability of such tools on a variety of tasks involving selection and tradeoff. The studies clearly show that example-based search is comparable to ranked lists on simple tasks, but significantly reduces the error rate and search time when complex tradeoffs are involved. This shows that such tools are likely to be useful particularly for extending the scope of consumer e-commerce to more complex products.

To find products in online environments, people increasingly rely on computerized search tools. The performance of such tools depends crucially on an accurate model of their users' preferences. Obtaining such models requires an adequate interaction model and system guidance.

We consider automated decision aids that help users select the best solution from a large set of options. For such tools to successfully accomplish their task, eliciting and representing users' decision preferences is a crucial task. It is usually too complex to get a complete and accurate model of their preferences, especially regarding the trade‐offs between different criteria.We consider decision aid tools where users specify their preferences qualitatively: they are only able to state the criteria they consider, but not the precise numerical utility functions. For each criterion, the tool provides a standardized numerical function that is fixed and identical for all users and used to compare solutions. To compensate for the imprecision of this qualitative model, we let the user choose among a displayed set of possibilities rather than a single optimal solution. We consider the probability of finding the most preferred solution as a function of the number of displayed possibilities and the number of preferences. We present a probabilistic analysis, empirical validation on randomly generated configuration problems and a commercial application. We provide mathematical principles for the design of the selection mechanism, guaranteeing that users are able to find the target solution.

The social navigation types that include recommendations and guides were discussed. The systems goal was to increase visual awareness of social cues that exist in the information space, without levying the final decision on the user. Recommendation systems include computed information that help in guiding the users.

In this paper we discuss the design and use of fisheye view techniques to explore semantic relationships in information. Traditional fisheye and "focus + context" techniques dynamically modify the visual rendering of data in response to the changing interest of the user. "Interesting" information is shown in more detail or visually emphasized, while less relevant information is shown in less detail, de-emphasized, or filtered. These techniques are effective for navigating through large sets of information in a constrained display, and for discovering hidden relationships in a particular representation. An open area of research with these techniques, however, is how to redefine interest as a user's tasks and information needs change.

Personalization in e-commerce has so far been server-centric, requiring users to create a separate individual profile on each server that they like to access. As product information is increasingly coming from multiple and heterogeneous sources, the number of profiles becomes unmanageably large. We present SmartClient, a technology based on constraint programming where a thin but intelligent client provides personalized information access for its user. As the process can run on the user's side, it allows much stronger filtering and visualization support with a wider range of personalization options than existing tools. It also eliminates the need to personalize many sites individually with different parameters, and supports product configuration and integration of different information sources in the same framework. We illustrate the technology using an application in travel e-commerce, which is currently under commercial deployment.

Visual thinking tools are visualization-enabled mixed initiative systems that empower people in solving complex problems by engaging them in the entire resolution process, suggesting appropriate actions with visual cues, and reducing their cognitive load with visual representations of their tasks. At the same time, the visual interaction style provides an alternative to the dialog-based model employed in most mixed-initiative (MI) systems. Visual thinking tools avoid complex analyses of turn taking, and put users in control all the time. We are especially interested in implementing visual "affordances" in such systems and present three examples used in COMIND, a visual MI system that we have developed. We show how humans can more effectively concentrate on synthesizing problems, selecting resolution paths that were unseen by the machine, and reformulating problems if solutions cannot be found or are unsatisfactory. We further discuss our evaluation of the techniques at the end of the paper.

In electronic commerce, a satisfying buyer experience is a key competitive element. We show new techniques for better adapting interaction with an electronic catalog system to actual buying behavior. Our model replaces the sequential separation of needs identification and product brokering with a conversation in which both processes occur simultaneously. This conversation supports the buyer in formulating his or her needs, and in deciding which criteria to apply in selecting a product to buy. We have experimented with this approach in the area of travel planning and developed a system called SmartClient Travel which supports this process. It includes tools for need identification, visualization of alternatives, and choosing the most suitable one. We describe the system and its implementation, and report on user studies showing its advantages for electronic catalogs.

Communication networks are expected to offer a wide range of services to an increasingly large number of users, with a diverse range of quality of service. This calls for efficient control and management of these networks. In this paper, we address the problem of quality-of-service routing, more specifically the planning of bandwidth allocation to communication demands. Shortest path routing is the traditional technique applied to this problem. However, this can lead to poor network utilization and even congestion. We show how an abstraction technique combined with systematic search algorithms and heuristics derived from artificial intelligence make it possible to solve this problem more efficiently and in much tighter networks, in terms of bandwidth usage.

In our previous paper, we have shown that case-based reasoning (CBR) techniques can be used as a viable formulation for solving assembly sequence generation problems. The issues covered in that paper were case base organization, case selection and matching, and case indexing. The part on case adaptation was not addressed in a formal way to allow satisfactory generalization of the method to a large class of assembly planning problems. We present in this paper a methodology which formalizes the adaptation process of CBR using constraint satisfaction techniques. Combining CBR with constraint satisfaction provides a generalized formalism for assembly planning problem solving.

In a job shop where machining time takes a comparable amount as material transportation time, it is no longer realistic to ignore the scheduling of material transportation systems such as automated guided vehicles (AGVs). The authors' algorithm discussed in this paper integrates AGV schedules into a heuristic-based scheduling algorithm to achieve the maximum amount of flexibility and optimization for a flexible manufacturing environment. Furthermore in order to allow their system to achieve different optimization goals such as shortest schedules or schedules that require the minimum computation time, the authors' system architecture consists of individual modules of heuristics so that one or any combination of these modules can be used. The authors then discuss how they control the use of these heuristic modules to achieve the best results with some experimental data.>

The author explores the possibility of using case-based reasoning (CBR) techniques to handle search in assembly sequence generation (ASG). CBR solves a new problem by retrieving from its case library a solution which has solved a similar problem in the past and then adapting the solution to the new problem. To illustrate how the system works, two experiments are described to show how a case-based search derived from a time-consuming spatial problem in ASG, the receptacle device, can be efficiently applied to two similar problems: a ball-point pen assembly and a complicated industrial assembly.>

Existing methods for robot planning fall far behind human capabilities: they require approximations of shapes, and they cannot generate plans which involve moving obstacles to clear a path for the moving object. The authors explore the hypothesis that means-ends analysis based on a world model involving mental imagery allows more human-like solutions. The method is based on a way or representing planning constraints which makes it possible to generate incrementally the symbolic representations for means-ends planning using only imagery operations.>

A synergism has begun to surface from the artificial intelligence (AI) and engineering communities: an effort to apply AI techniques to engineering problem‐solving activities, and to study problems arisen from various engineering fields as a way to develop AI theories and methodologies. This paper first discusses the needs of such a synergical approach and identifies in a broad perspective some AI techniques currently being applied to engineering. It then describes a system, called KREATOR, which applies qualitative reasoning, a subfield of AI, to computer‐aided design (CAD). The key observation is that an engineer designer's qualitative knowledge can offer a good basis for the reasoning of device behaviors. Such knowledge, however, is not captured by conventional CAD systems for lack of good representations. KREATOR is a knowledge capturing scheme that allows the designers to record their qualitative knowledge of how mechanical devices behave, KREATOR then automatically generates qualitative simulations.

A new method of calibration is proposed for active visual sensing for use in 3D analysis. The method, based on an adaptation of the two camera plane m model, permits the calibration of a mobile camera as a function of the position and orientation of the camera.