In this paper, we model aspects of communication beyond the words that are said. Specifically, we aim to detect interruptions and active listening events, which are important elements in any dialogue. We build a dataset with fine-grained annotations for each category and train multimodal models that take into account all channels in a digital conversation, that is, the video, the audio, and the text. Our experiments show that multimodality is a necessary component in modeling the complexity of the non-textual components of the conversation as different artifacts require different modalities to capture effectively.

Critiquing is a method for conversational recommendation that incrementally adapts recommendations in response to user preference feedback. Recent advances in critiquing have leveraged the power of VAE-CF recommendation in a critiquable-explainable (CE-VAE) framework that updates latent user preference embeddings based on their critiques of keyphrase-based explanations. However, the CE-VAE has two key drawbacks: (i) it uses a second VAE head to facilitate explanations and critiquing, which can sacrifice recommendation performance of the first VAE head due to multiobjective training, and (ii) it requires iterating an inverse decoding-encoding loop for multi-step critiquing that yields poor performance. To address these deficiencies, we propose a novel Bayesian Keyphrase critiquing VAE (BK-VAE) framework that builds on the strengths of VAE-CF, but avoids the problematic second head of CE-VAE. Instead, the BK-VAE uses a Concept Activation Vector (CAV) inspired approach to determine the alignment of item keyphrase properties with latent user preferences in VAE-CF. BK-VAE leverages this alignment in a Bayesian framework to model uncertainty in a user's latent preferences and to perform posterior updates to these preference beliefs after each critique --- essentially achieving CE-VAE's explanation and critique inversion through a simple application of Bayes rule. Our empirical evaluation on two datasets demonstrates that BK-VAE matches or dominates CE-VAE in both recommendation and multi-step critiquing performance.

Interactive (a.k.a. conversational) recommendation systems provide the potential capability to personalize interactions with increasingly prevalent dialog-based AI assistants. In the conversational recommendation setting, a user often has long-term preferences inferred from previous interactions along with ephemeral session-based preferences that need to be efficiently elicited through minimal interaction. Historically, Bayesian preference elicitation methods have proved effective for (i) leveraging prior information to incrementally estimate uncertainty in user preferences as new information is observed, and for (ii) supporting active elicitation of preference feedback to quickly zero in on the best recommendations in a session. Previous work typically focused on eliciting preferences in the space of items or a small set of attributes; in the dialog-based setting, however, we are faced with the task of eliciting preferences in the space of natural language while using this feedback to determine a user’s preferences in item space. To address this task in the era of modern, latent embedding-based recommender systems, we propose a method for coembedding user-item preferences with keyphrase descriptions (i.e., not explicitly known attributes, but rather subjective judgments mined from user reviews or tags) along with a closed-form Bayesian methodology for incrementally estimating uncertainty in user preferences based on elicited keyphrase feedback. We then combine this framework with well-known preference elicitation techniques that can leverage Bayesian posteriors such as Upper Confidence Bounds, Thompson Sampling, and a variety of other methods. Our empirical evaluation on real-world datasets shows that the proposed query selection strategies effectively update user beliefs, leading to high-quality recommendations with a minimal number of keyphrase queries.

Providing explanations for recommended items not only allows users to understand the reason for receiving recommendations but also provides users with an opportunity to refine recommendations by critiquing undesired parts of the explanation. While much research focuses on improving the explanation of recommendations, less effort has focused on interactive recommendation by allowing a user to critique explanations. Aside from traditional constraint- and utility-based critiquing systems, the only end-to-end deep learning based critiquing approach in the literature so far, CE-VNCF, suffers from unstable and inefficient training performance. In this paper, we propose a Variational Autoencoder (VAE) based critiquing system to mitigate these issues and improve overall performance. The proposed model generates keyphrase-based explanations of recommendations and allows users to critique the generated explanations to refine their personalized recommendations. Our experiments show promising results: (1) The proposed model is competitive in terms of general performance in comparison to state-of-the-art recommenders, despite having an augmented loss function to support explanation and critiquing. (2) The proposed model can generate high-quality explanations compared to user or item keyphrase popularity baselines. (3) The proposed model is more effective in refining recommendations based on critiquing than CE-VNCF, where the rank of critiquing-affected items drops while general recommendation performance remains stable. In summary, this paper presents a significantly improved method for multi-step deep critiquing based recommender systems based on the VAE framework.

Critiquing is a method for conversational recommendation that iteratively adapts recommendations in response to user preference feedback. In this setting, a user is iteratively provided with an item recommendation and attribute description for that item; a user may either accept the recommendation, or critique the attributes in the item description to generate a new recommendation. Historical critiquing methods were largely based on explicit constraint- and utility-based methods for modifying recommendations w.r.t. critiqued item attributes. In this paper, we revisit the critiquing approach in the era of recommendation methods based on latent embeddings with subjective item descriptions (i.e., keyphrases from user reviews). Two critical research problems arise: (1) how to co-embed keyphrase critiques with user preference embeddings to update recommendations, and (2) how to modulate the strength of multi-step critiquing feedback, where critiques are not necessarily independent, nor of equal importance. To address (1), we build on an existing state-of-the-art linear embedding recommendation algorithm to align review-based keyphrase attributes with user preference embeddings. To address (2), we exploit the linear structure of the embeddings and recommendation prediction to formulate a linear program (LP) based optimization problem to determine optimal weights for incorporating critique feedback. We evaluate the proposed framework on two recommendation datasets containing user reviews with simulated users. Empirical results compared to a standard approach of averaging critique feedback show that our approach reduces the number of interactions required to find a satisfactory item and increases the overall success rate.