To help researchers find papers of interest quickly and accurately, paper recommendations are widely deployed. However, they face unique challenges, including the exploitation of rich multi-modal features and the modeling of complex relationships. In terms of multi-modal features, previous work has focused on textual information but ignored visual information, as paper images may be missing or their meanings are highly domain specific. On the other hand, although conventional recommendation methods are suitable for modeling the user-paper relationship, the lack of specific designs for the paper-paper relationship leads to sub-optimal results. To overcome these limitations, we propose a Multi-interest based Multi-modal paper recommendation model named TMRec. TMRec utilizes screenshots of papers as visual impressions to capture paper style features while avoiding the dilemma of missing paper images. Additionally, we have specially designed the multi-interest extraction module and the multi-level interaction module to consider both the multi-interest and the citation relationship between papers in the user behavior sequence. Compared to various strong baselines, TMRec has a relative improvement for up to 20% in terms of recall rate on real-world datasets, which demonstrates the superiority of TMRec and effectiveness of the visual impression feature.

The emergence of large language models (LLMs) has revolutionized the capabilities of text comprehension and generation. Multi-modal generation attracts great attention from both the industry and academia, but there is little work on personalized generation, which has important applications such as recommender systems. This paper proposes the first method for personalized multimodal generation using LLMs, showcases its applications and validates its performance via an extensive experimental study on two datasets. The proposed method, Personalized Multimodal Generation (PMG for short) first converts user behaviors (e.g., clicks in recommender systems or conversations with a virtual assistant) into natural language to facilitate LLM understanding and extract user preference descriptions. Such user preferences are then fed into a generator, such as a multimodal LLM or diffusion model, to produce personalized content. To capture user preferences comprehensively and accurately, we propose to let the LLM output a combination of explicit keywords and implicit embeddings to represent user preferences. Then the combination of keywords and embeddings are used as prompts to condition the generator. We optimize a weighted sum of the accuracy and preference scores so that the generated content has a good balance between them. Compared to a baseline method without personalization, PMG has a significant improvement on personalization for up to 8% in terms of LPIPS while retaining the accuracy of generation.