An increasing number of recommendation systems try to enhance the overall user experience by incorporating conversational interaction. However, evaluating conversational recommender systems (CRSs) from the user’s perspective remains elusive. The GUI-based system evaluation criteria may be inadequate for their conversational counterparts. This article presents our proposed unifying framework, CRS-Que , to evaluate the user experience of CRSs. This new evaluation framework is developed based on ResQue , a popular user-centric evaluation framework for recommender systems. Additionally, it includes user experience metrics of conversation (e.g., understanding, response quality, humanness) under two dimensions of ResQue (i.e., Perceived Qualities and User Beliefs). Following the psychometric modeling method, we validate our framework by evaluating two conversational recommender systems in different scenarios: music exploration and mobile phone purchase . The results of the two studies support the validity and reliability of the constructs in our framework and reveal how conversation constructs and recommendation constructs interact and influence the overall user experience of the CRS. We believe this framework could help researchers conduct standardized user-centric research for conversational recommender systems and provide practitioners with insights into designing and evaluating a CRS from users’ perspectives.

Music can affect the human brain and cognition. Melodies and lyrics that resonate with us can awaken our inner feelings and thoughts; being in touch with these feelings and expressing them allow us to understand ourselves better and increase our self-awareness. To support self-awareness elicited by music, we designed a novel conversational agent (CA) that guides users to become self-aware and express their thoughts when they listen to music. Moreover, we investigated two prominent design factors in the CA, proactive guidance and social information. We then conducted a 2x2 between-subjects experiment (N = 90) to investigate how the two design factors affect self-awareness, user acceptance, and mental well-being. The results of a five-day user study reveal that high proactive guidance and social information increased self-awareness, but high proactive guidance tended to influence perceived autonomy and usefulness negatively. Further, users’ subjective feedback revealed the CA’s potential to support mental well-being.