Bundle generation aims to provide a bundle of items for the user, and has been widely studied and applied on online service platforms. Existing bundle generation methods mainly utilized user’s preference from historical interactions in common recommendation paradigm, and ignored the potential textual query which is user’s current explicit intention. There can be a scenario in which a user proactively queries a bundle with some natural language description, the system should be able to generate a bundle that exactly matches the user’s intention through the user’s query and preferences. In this work, we define this user-friendly scenario as Query-based Bundle Generation task and propose a novel framework Text2Bundle that leverages both the user’s short-term interests from the query and the user’s long-term preferences from the historical interactions. Our framework consists of three modules: (1) an intention extractor based on Large Language Model that mines the user’s fine-grained interests from the query; (2) a unified state encoder that learns the current bundle context state and the user’s preferences based on historical interaction and current query; and (3) a bundle generator that generates personalized and complementary bundles using reinforcement learning with specifically designed rewards. We conduct extensive experiments on three real-world datasets and demonstrate the effectiveness of our framework compared with several state-of-the-art methods.

Despite remarkable advances in image captioning, existing models still lack the ability to generate controllable and diverse captions. As a solution, controllable image captioning (CIC) has recently gained attention, with the goal of generating image captions that satisfy the constraints of the given control signals. Current CIC methods have two main limitations: (1) They can only handle one specific control signal and lack the ability to handle combinations of multiple control signals. (2) They depend on costly supervised learning from task-specific data, which becomes impractical with increasing model size. To this end, we propose an energy-based sampling method for controllable image captioning, named SamCap. Specifically, by combining various constraint functions with the log likelihood of the image captioner into an energy function, we can generate captions that satisfy the specified constraints through gradient-based sampling. SamCap provides a learning-free and plug-and-play solution, that can integrate with any existing image captioner without task-specific fine-tuning. Extensive results demonstrate that SamCap not only matches the performance of SOTA signal-specific CIC models for single control signals, but also shows significant advantages in handling combinations of multiple control signals.

Conversational recommendation system (CRS) is able to obtain fine-grained and dynamic user preferences based on interactive dialogue. Previous CRS assumes that the user has a clear target item, which often deviates from the real scenario. The user may have a clear single preference for some attribute types (e.g., brand) of items, while for other attribute types (e.g., color), the user may have multiple preferences or even no clear preferences, which leads to multiple acceptable items under multiple combinations of attribute instances. Furthermore, previous works assume that users would provide clear responses to any questions asked by the system. And, they also assume that users would be dedicated to the target item, that is, user would answer “yes” to the attribute corresponding to the target item and answer “no” to other attributes. However, users’ responses to attributes are not completely dependent on target items, but also influenced by users’ inherent interests. Besides, for some over-specific or equivocal questions, the feedback of user might not be clear (“yes”/“no”) and user might give some fuzzy response like “I don’t know”. To address the aforementioned issues, we first propose a more realistic conversational recommendation learning setting, namely Multi-Interest Multi-round Conversational Recommendation (MIMCR), where users may have multiple interests in attribute instance combinations and accept multiple items with partially overlapped combinations of attribute instances. To effectively cope with MIMCR, we propose a novel learning framework, namely Multiple Choice questions based on Multi-Interest Policy Learning. Moreover, we further propose a more realistic User-centric User Simulator with Fuzzy Feedback (UUSFF), which naturally calibrates the user response with additional fuzzy feedback based on user’s inherent preference. To better match the new scenario UUSFF, we propose a simple but effective adaption method for different backbones. Extensive experimental results on several datasets demonstrate the superiority of our methods for the proposed settings.

Visual reasoning requires models to construct a reasoning process towards the final decision. Previous studies have used attention maps or textual explanations to illustrate the reasoning process, but both have their limitations. Attention maps can be difficult to read, while textual explanations cannot fully describe the process of reasoning, and both are hard to evaluate quantitatively. This paper proposes a novel pixel-to-explanation reasoning model that employs a user-friendly multimodal rationale to depict the reasoning process. The model dissects the question into subquestions, and constructs reasoning cells to retrieve knowledge from the image and question based on these subquestions. The intermediate outcomes from the reasoning cells are translated into object bounding boxes and classes, with the final output beging classified as a standard VQA answer and translated into a complete answer to summarize the entire reasoning process. All the generated results can be combined to produce a human-readable and informative explanation that can be evaluated quantitatively. Besides the interpretability, we achieved a 4.4% improvement over our baseline model on the GQA dataset and attained new state-of-the-art results on the challenging GQA-OOD dataset.

Social recommendation based on social network has achieved great success in improving the performance of the recommendation system. Since social network (user-user relations) and user-item interactions are both naturally represented as graph-structured data, Graph Neural Networks (GNNs) have thus been widely applied for social recommendation. Despite the superior performance of existing GNNs-based methods, there are still several severe limitations: (i) Few existing GNNs-based methods have considered a single heterogeneous global graph which takes into account user-user relations, user-item interactions, and item-item similarities simultaneously. That may lead to a lack of complex semantic information and rich topological information when encoding users and items based on GNN. (ii) Furthermore, previous methods tend to overlook the reliability of the original user-user relations which may be noisy and incomplete. (iii) More importantly, the item-item connections established by a few existing methods merely using initial rating attributes or extra attributes (such as category) of items, may be inaccurate or sub-optimal with respect to social recommendation. In order to address these issues, we propose an end-to-end heterogeneous global graph learning framework, namely Graph Learning Augmented Heterogeneous Graph Neural Network (GL-HGNN) for social recommendation. GL-HGNN aims to learn a heterogeneous global graph that makes full use of user-user relations, user-item interactions and item-item similarities in a unified perspective. To this end, we design a Graph Learner (GL) method to learn and optimize user-user and item-item connections separately. Moreover, we employ a Heterogeneous Graph Neural Network (HGNN) to capture the high-order complex semantic relations from our learned heterogeneous global graph. To scale up the computation of graph learning, we further present the Anchor-based Graph Learner (AGL) to reduce computational complexity. Extensive experiments on four real-world datasets demonstrate the effectiveness of our model.

Predicting the next interaction of a short-term interaction session is a challenging task in session-based recommendation. Almost all existing works rely on item transition patterns, and neglect user historical sessions while modeling user preference, which often leads to non-personalized recommendation. And existing personalized session-based recommenders are limited to sessions of the current user, and ignore the useful item-transition patterns from other user's historical sessions. To address these issues, we propose a novel Heterogeneous Global Graph Neural Networks (HG-GNN) to exploit the item transitions over all sessions in a subtle manner for better inferring user preference from the current and historical sessions. To effectively exploit the item transitions over all sessions from users, our global graph contains item transitions of sessions, user-item interactions and global co-occurrence items. Moreover, to capture user preference from sessions comprehensively, we propose a graph augmented preference encoder to learn the session representation. Specifically, we design a novel heterogeneous graph neural network (HGNN) on heterogeneous global graph to learn long-term user preference and item representations with rich semantics. Based on the HGNN, we propose the Personalized Session Encoder to combine the general user preference and temporal interest of the current session to generate the personalized session representation for recommendation. Extensive experimental results on three real-world datasets show that our model outperforms other state-of-the-art methods.

Conversational recommendation system (CRS) is able to obtain fine-grained and dynamic user preferences based on interactive dialogue. Previous CRS assumes that the user has a clear target item, which often deviates from the real scenario, that is for many users who resort to CRS, they might not have a clear idea about what they really like. Specifically, the user may have a clear single preference for some attribute types (e.g. brand) of items, while for other attribute types (e.g. color), the user may have multiple preferences or even no clear preferences, which leads to multiple acceptable attribute instances (e.g. black and red) of one attribute type. Therefore, the users could show their preferences over items under multiple combinations of attribute instances rather than a single item with unique combination of all attribute instances. As a result, we first propose a more realistic conversational recommendation learning setting, namely Multi-Interest Multi-round Conversational Recommendation (MIMCR), where users may have multiple interests in attribute instance combinations and accept multiple items with partially overlapped combinations of attribute instances. To effectively cope with the new CRS learning setting, in this paper, we propose a novel learning framework, namely Multiple Choice questions based Multi-Interest Policy Learning (MCMIPL). In order to obtain user preferences more efficiently, the agent generates multiple choice questions rather than binary yes/no ones on specific attribute instance. Furthermore, we propose a union set strategy to select candidate items instead of existing intersection set strategy in order to overcome over-filtering items during the conversation. Finally, we design a Multi-Interest Policy Learning (MIPL) module, which utilizes captured multiple interests of the user to decide next action, either asking attribute instances or recommending items. Extensive experimental results on four datasets demonstrate the superiority of our method for the proposed MIMCR setting.