In this study, we investigate the role of visual content in accurately predicting movie genres. By extracting keyframes from Hindi, Bengali, Malayalam, and Telugu language-based Indian movie trailers in the Flickscore dataset, we analyse visual elements using visual language model (VLM) and Large Language Models (LLMs). Our approach focuses on the FAMOS aspects (focus, action, mood, object, setting) to understand the movie’s theme, summary, and genre. This method eliminates the need to manually prepare textual metadata, thus reducing the time and effort required for genre identification. The visual features captured from the keyframes are leveraged to predict genres more efficiently, offering a reliable way to automate this process. Additionally, we integrate the visual information into a content-based filtering (CBF) system to predict user preferences. Our study highlights the effectiveness of using visual features, which significantly enhances the performance of recommendation systems (RS) by improving accuracy in predicting user preferences based on movie genres. We demonstrate that by analysing the implicit content in movie frames, we can achieve better insights than traditional metadata-driven approaches. Overall, our findings emphasise that visual content holds valuable information for understanding movie genres and can play a critical role in user preference understanding.

Recommender systems used in online platforms can drive users to consume content continuously in an attempt to maximize satisfaction. Such engagement is invariably broken due to more pressing work, alternate pursuits, distractions or fatigue. Recommender systems need to ensure the continuity of experience when the user joins back. Session-based recommender systems typically create different sessions based on a fixed time interval (θ), often resulting in creation of a separate session when the user gets off the platform temporarily. When the user joins back, session-based recommender systems are likely to recommend content different than what they would have in case the earlier session had continued. This may cause dissatisfaction given that there is a difference in the predicted world model of the user, i.e. the expectation from the last session, and the observed one, i.e. the recommendations. To handle this problem, we propose the creation of content-driven sessions instead of time-driven sessions. In our setting, a session continues while a single item category dominates in the user-item interactions. A new session is created when a different item category begins to dominate. The proposed content-driven method also solves the long-standing problem of deciding the optimal value of time threshold (θ) for defining the time-based session. We report that the proposed method outperforms existing SOTA methodologies set by time-based sessions by a large margin in terms of recommendation performance on multiple datasets.

Learning the preferences of new users and items with limited interaction history is a major challenge in recommender systems, known as the cold-start recommendation problem. Meta-learning methods, particularly those based on MAML, show strong potential by learning generalized initializations across randomly sampled tasks. In the context of cold-start recommendation, a task represents a new user or item and their associated interactions. These initializations enable rapid adaptation to new users or items for effective preference learning. However, these methods often struggle with personalization due to two key limitations: 1) the generalized initialization is learned under the assumption of a uniform task distribution, and 2) the inability to capture user or item-specific rating tendencies. To address these limitations, we propose Dynamic Task-Adaptive Meta Optimization (DTAMO). DTAMO learns initializations on dynamic tasks, which are clusters of similar users based on its profile attributes and interaction patterns. This helps to capture both personalized and shared user preferences. We design a Cross-Attention Transformer-based autoencoder to discover these user clusters and introduce a Task-Adaptive Meta Optimizer (TAMO) that performs attention-weighted gradient aggregation within each dynamic task. This enables TAMO to efficiently capture both shared and personalized preferences in a single optimization step. To better model user and item-specific rating behaviors, DTAMO incorporates lightweight rating-distribution-aware representations, which capture rating tendencies of user or items with minimal computational cost. We integrate ordinal regression into the meta-learning framework to further enhance the personalization by aligning the model’s learning objective with the ordinal structure of rating data. Extensive experiments on three public datasets demonstrate that DTAMO consistently outperforms state-of-the-art methods in both accuracy and scalability.

With the increasing availability of multimodal movie data, there is a growing interest in leveraging these data to improve movie recommendations. In the recent era, due to the increase in the number of users and movies on OTT platforms such as Amazon Prime, its services, including personalized movie recommendations, become challenging. This article proposes a novel approach$M^{2}RM^{2}UL$, which stands for multimodal movie recommendation with multitasking and movie–user learning. The initial phase involves preprocessing multimodal movie data to extract features encompassing visual and textual information. Subsequently, a multitasking architecture is employed, simultaneously undertaking classification and regression tasks to acquire user and movie representations. The learned representations are used to make personalized and accurate movie recommendations. Additionally, the Netflix Prize dataset has been augmented to include textual and visual features, rendering it multimodal. We conducted extensive experiments on three real-world multimodal movie datasets (Movielens-100K, MMTF-14K, and Netflix Prize) and compared our approach with several state-of-the-art movie recommendation algorithms. The experimental results illustrate that our approach outperforms the baseline methods in terms of recommendation accuracy and diversity. Furthermore, we demonstrate the effectiveness of our approach in different scenarios, such as cold-start and sparse data. Our empirical study provides strong evidence for the effectiveness of the proposed approach in multimodal movie recommendation.

Saiteja Kosgi, Sarath Sivaprasad, Niranjan Pedanekar, Anil Nelakanti, Vineet Gandhi. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2022.

Predicting emotions that movies are designed to evoke, can be useful in entertainment applications such as content personalization, video summarization and ad placement. Multimodal input, primarily audio and video, helps in building the emotional content of a movie. Since the emotion is built over time by audio and video, the temporal context of these modalities is an important aspect in modeling it. In this paper, we use Long Short-Term Memory networks (LSTMs) to model the temporal context in audio-video features of movies. We present continuous emotion prediction results using a multimodal fusion scheme on an annotated dataset of Academy Award winning movies. We report a significant improvement over the state-of-the-art results, wherein the correlation between predicted and annotated values is improved from 0.62 vs 0.84 for arousal, and from 0.29 to 0.50 for valence.

Several hundred Massively Open Online Courses (MOOCs) are available for students of Computer Science (CS) across the Internet. Yet, it has been observed that students exhibit a short attention span while watching MOOC videos. At the same time, as a viewer, they are likely to watch much longer movies and even educational films. In this poster, we propose that production of MOOCs needs to borrow certain affective features from more professionally produced educational films. To support this argument, we first present the results of a limited survey indicating an affective preference of users towards educational films over video lectures from MOOCs. Taking a cue from films in general, we present an analysis of certain affective audio-visual features of educational films vis-à-vis MOOC videos. These features include visual features related to the variety of color, number of scenes and movement, and audio features related to liveliness of speech. We then use these features to classify two groups of educational videos, and conclude that MOOC-like videos often tend to lack such affective audio-visual features. We also indicate possible directions of research in educational videos based on our initial findings.

Relevance of a concept being taught to the real world is believed to contribute to an increase in the intrinsic motivation and engagement of a learner. Such relevance is often found lacking in learning material such as textbooks. Practical issues and problems one could face while learning or implementing new concepts are a means of establishing such relevance. In this paper, we propose a method to automatically augment learning material with practical questions about the concept being learnt. We use questions and answers from StackOverflow, a leading social Questions and Answers (Q&A) website to augment an electronic textbook interface, thus connecting the concepts being taught to the real world. For achieving this automatically, we first mine the textbook content to locate words and phrases which are likely to be the most important concepts on each page of the textbook. We then select only those words and phrases which appear as ‘tags’ in StackOverflow, typically defined by users while asking and answering questions. Using permutations of these tags as queries, we query the StackOverflow database to obtain relevant questions and answers to augment any given page. We present an interface to augment textbooks with such questions using the aforementioned method. We also present the results of a student survey examining the effectiveness of the augmentation in establishing relevance to their learning.

Sticky learning refers to learning that is retained by a learner over a long period of time. Explanatory analogies are often used by good teachers to explain complex concepts in a sticky manner. Such analogies explain an unfamiliar target concept by mapping it onto a more familiar source concept. However, the use of analogies in teaching and learning often relies on the imagination of individual teachers or the initiative taken by students in finding them. In this paper, we present Stickipedia, an analogy search engine that automatically retrieves analogies populated on the internet for a searched target concept. Based on a student survey, we also suggest attributes of analogies which could aid students in choosing the analogies they prefer. We populate some of these attributes in Stickipedia for the retrieved analogies.

Computer science has a wide variety of applications in a wide variety of fields. Yet computer science education focuses primarily on the theoretician's perspective. We believe that if a variety of perspectives are brought in during learning, learners' intrinsic motivation can be increased, and learning computer science can be made more engaging and personalized. In this work-in-progress paper, we propose the concept of `hats' for augmenting computer science learning material with different perspectives. Hats are different perspectives which can be donned by the learner while learning computer science. We propose examples of hats such as Programmer, Historian, Job Seeker, Troubleshooter and Visualizer. To enable the use of hats while learning, we propose a web-based interface to augment base learning content such as textbooks with additional learning content classified under different hats. To provide scale in populating hats, we propose algorithms to automatically find content from the internet that can be categorized under different hats. We also present the results from a pilot study conducted using the hats interface.