Narrative-driven recommenders aim to provide personalized suggestions for user requests expressed in free-form text such as ''I want to watch a thriller with a mind-bending story, like Shutter Island.'' Although large language models (LLMs) have been shown to excel in processing general natural language queries, their effectiveness for handling such recommendation requests remains relatively unexplored. To close this gap, we compare the performance of 38 open- and closed-source LLMs of various sizes, such as LLama 3.2 and GPT-4o, in a movie recommendation setting. For this, we utilize a gold-standard, crowdworker-annotated dataset of posts from reddit's movie suggestion community and employ various prompting strategies, including zero-shot, identity, and few-shot prompting. Our findings demonstrate the ability of LLMs to generate contextually relevant movie recommendations, significantly outperforming other state-of-the-art approaches, such as doc2vec. While we find that closed-source and large-parameterized models generally perform best, medium-sized open-source models remain competitive, being only slightly outperformed by their more computationally expensive counterparts. Furthermore, we observe no significant differences across prompting strategies for most models, underscoring the effectiveness of simple approaches such as zero-shot prompting for narrative-driven recommendations. Overall, this work offers valuable insights for recommender system researchers as well as practitioners aiming to integrate LLMs into real-world recommendation tools.

While searching for consumer goods, users frequently ask for suggestions from their peers by writing short free-form textual requests. For example, when searching for movies users may ask for “Drama movies with a mind-bending story and a surprise ending, such as Fight Club” in one of the many online discussion boards. Despite the recent developments in large language models (LLMs) and natural language processing (NLP), modern recommender systems still struggle to process such requests. Therefore, in this paper we evaluate several approaches for annotating structured information from such short, free-form natural language user texts to calculate recommendations. We set up this evaluation as a two phase processes including (a) identification of the best NLP approach to identify key elements of users’ requests, and (b) assessment of the quality of recommendations computed with such elements. For our evaluation, we use a gold-standard reddit movie recommendation dataset consisting of annotations, manually created by crowdworkers who extracted keywords, actor names and movie titles. Using this dataset we evaluate a collection of more than 30 NLP and five recommender approaches. In addition, we perform an ablation study to assess relative annotation importance for movie recommendations. We find that domain-specific deep learning models, trained on a subset of data as well as embedding-based recommendation approaches are able to match the recommendation performance of recommendations computed from manual annotations. These promising results warrant further investigation in automatic processing of short free-form texts for computation of recommendations. Specifically, we provide insights into which NLP models and configurations work best for automatically annotating free text to compute (movie) recommendations, hence substantially reducing the search space for combinations of NLP and recommendation algorithms in the movie and potentially other domains.

Recommender systems have become omni-present tools that are used by a wide variety of users in everyday life tasks, such as finding products in Web stores or online movie streaming portals. However, in situations where users already have an idea of what they are looking for (e.g., 'The Lord of the Rings', but in space with a dark vibe), most traditional recommender algorithms struggle to adequately address such a priori defined requirements. Therefore, users have built dedicated discussion boards to ask peers for suggestions, which ideally fulfill the stated requirements. In this paper, we set out to determine the utility of well-established recommender algorithms for calculating recommendations when provided with such a narrative. To that end, we first crowdsource a reference evaluation dataset from human movie suggestions. We use this dataset to evaluate the potential of five recommendation algorithms for incorporating such a narrative into their recommendations. Further, we make the dataset available for other researchers to advance the state of research in the field of narrative-driven recommendations. Finally, we use our evaluation dataset to improve not only our algorithmic recommendations, but also existing empirical recommendations of IMDb. Our findings suggest that the implemented recommender algorithms yield vastly different suggestions than humans when presented with the same a priori requirements. However, with carefully configured post-filtering techniques, we can outperform the baseline by up to 100%. This represents an important first step towards more refined algorithmic narrative-driven recommendations.