In this study, we investigate the role of visual content in accurately predicting movie genres. By extracting keyframes from Hindi, Bengali, Malayalam, and Telugu language-based Indian movie trailers in the Flickscore dataset, we analyse visual elements using visual language model (VLM) and Large Language Models (LLMs). Our approach focuses on the FAMOS aspects (focus, action, mood, object, setting) to understand the movie’s theme, summary, and genre. This method eliminates the need to manually prepare textual metadata, thus reducing the time and effort required for genre identification. The visual features captured from the keyframes are leveraged to predict genres more efficiently, offering a reliable way to automate this process. Additionally, we integrate the visual information into a content-based filtering (CBF) system to predict user preferences. Our study highlights the effectiveness of using visual features, which significantly enhances the performance of recommendation systems (RS) by improving accuracy in predicting user preferences based on movie genres. We demonstrate that by analysing the implicit content in movie frames, we can achieve better insights than traditional metadata-driven approaches. Overall, our findings emphasise that visual content holds valuable information for understanding movie genres and can play a critical role in user preference understanding.

Recommender systems used in online platforms can drive users to consume content continuously in an attempt to maximize satisfaction. Such engagement is invariably broken due to more pressing work, alternate pursuits, distractions or fatigue. Recommender systems need to ensure the continuity of experience when the user joins back. Session-based recommender systems typically create different sessions based on a fixed time interval (θ), often resulting in creation of a separate session when the user gets off the platform temporarily. When the user joins back, session-based recommender systems are likely to recommend content different than what they would have in case the earlier session had continued. This may cause dissatisfaction given that there is a difference in the predicted world model of the user, i.e. the expectation from the last session, and the observed one, i.e. the recommendations. To handle this problem, we propose the creation of content-driven sessions instead of time-driven sessions. In our setting, a session continues while a single item category dominates in the user-item interactions. A new session is created when a different item category begins to dominate. The proposed content-driven method also solves the long-standing problem of deciding the optimal value of time threshold (θ) for defining the time-based session. We report that the proposed method outperforms existing SOTA methodologies set by time-based sessions by a large margin in terms of recommendation performance on multiple datasets.

Learning the preferences of new users and items with limited interaction history is a major challenge in recommender systems, known as the cold-start recommendation problem. Meta-learning methods, particularly those based on MAML, show strong potential by learning generalized initializations across randomly sampled tasks. In the context of cold-start recommendation, a task represents a new user or item and their associated interactions. These initializations enable rapid adaptation to new users or items for effective preference learning. However, these methods often struggle with personalization due to two key limitations: 1) the generalized initialization is learned under the assumption of a uniform task distribution, and 2) the inability to capture user or item-specific rating tendencies. To address these limitations, we propose Dynamic Task-Adaptive Meta Optimization (DTAMO). DTAMO learns initializations on dynamic tasks, which are clusters of similar users based on its profile attributes and interaction patterns. This helps to capture both personalized and shared user preferences. We design a Cross-Attention Transformer-based autoencoder to discover these user clusters and introduce a Task-Adaptive Meta Optimizer (TAMO) that performs attention-weighted gradient aggregation within each dynamic task. This enables TAMO to efficiently capture both shared and personalized preferences in a single optimization step. To better model user and item-specific rating behaviors, DTAMO incorporates lightweight rating-distribution-aware representations, which capture rating tendencies of user or items with minimal computational cost. We integrate ordinal regression into the meta-learning framework to further enhance the personalization by aligning the model’s learning objective with the ordinal structure of rating data. Extensive experiments on three public datasets demonstrate that DTAMO consistently outperforms state-of-the-art methods in both accuracy and scalability.

With the increasing availability of multimodal movie data, there is a growing interest in leveraging these data to improve movie recommendations. In the recent era, due to the increase in the number of users and movies on OTT platforms such as Amazon Prime, its services, including personalized movie recommendations, become challenging. This article proposes a novel approach$M^{2}RM^{2}UL$, which stands for multimodal movie recommendation with multitasking and movie–user learning. The initial phase involves preprocessing multimodal movie data to extract features encompassing visual and textual information. Subsequently, a multitasking architecture is employed, simultaneously undertaking classification and regression tasks to acquire user and movie representations. The learned representations are used to make personalized and accurate movie recommendations. Additionally, the Netflix Prize dataset has been augmented to include textual and visual features, rendering it multimodal. We conducted extensive experiments on three real-world multimodal movie datasets (Movielens-100K, MMTF-14K, and Netflix Prize) and compared our approach with several state-of-the-art movie recommendation algorithms. The experimental results illustrate that our approach outperforms the baseline methods in terms of recommendation accuracy and diversity. Furthermore, we demonstrate the effectiveness of our approach in different scenarios, such as cold-start and sparse data. Our empirical study provides strong evidence for the effectiveness of the proposed approach in multimodal movie recommendation.

We investigate the problem of smart operational efficiency, at scale, in Machine Learning models for Big Data streams, in context of embedded AI applications, by learning optimal early exits. Embedded AI applications that employ deep neural models depend on efficient model inference at scale, especially on resource-constrained hardware. Recent vision/text/audio models are computationally complex with huge parameter spaces and input samples typically pass through multiple layers, each with large tensor computations, to produce valid outputs. Generally, in most real scenarios, AI applications deal with big data streams, such as streams of audio signals, static images and/or high resolution video frames. Deep ML models powering such applications have to continuously perform inference on such big data streams for varied tasks such as noise suppression, face detection, gait estimation and so on. Ensuring efficiency is challenging, even with model compression techniques since they reduce model size but often fail to achieve scalable inference efficiency over continuous streams. Early exits enable adaptive inference by extracting valid outputs from any pre-final layer of a deep model which significantly boosts efficiency at scale since many of the input instances need not be processed at all the layers of a deep model, especially for big streams. Suitable early exit structure design (number + positions) is a difficult but crucial aspect in improving efficiency without any loss in predictive performance, especially in context of big streams. Naive manual early exit design that does not consider the hardware capacity or data stream characteristics is counterproductive. We propose NASEREX framework that leverages Neural architecture Search (NAS) with a novel saliency-constrained search space and exit decision metric to learn suitable early exit structure to augment Deep Neural models for scalable efficient inference on big image streams. Optimized exit-augmented models perform $\approx 2.5 \times$ faster having $\approx 4 \times$ aggregated lower effective FLOPs, with no significant accuracy loss.

The increasing popularity of digital entertainment systems has made personalization a key factor for success in the industry. Recommendation systems, particularly for videos and movies, are crucial in this regard. However, many existing systems are implicit feedback recommendation system that uses indirect signals to infer user preferences, such as user actions (e.g. clicks, views, purchases) or interactions with items (e.g. listening to a song, watching a movie). The challenge lies in the limited information and uncertainty present in user behavior, making it difficult to predict their interests and preferences. In Previous research, Recurrent Neural Networks (RNNs) have shown to be efficient in predicting the next item in a session, based on past item click sequences, but their effectiveness is limited when only relying on click sequences as input data. In this paper, we extend the Hierarchical RNN architecture (HRNN) for generating recommendations by combining session clicks and item content information, such as item ids and item description respectively. The Bidirectional Encoder Representations from Transformers (BERT) architecture is applied for generating feature vectors from text descriptions of the items. Our model has been extensively tested on the benchmark dataset MovieLens 1m has demonstrated superiority over state-of-the-art (SOTA) session-based recommendation systems (SBRS) models. Experimental results establish the efficacy of using content information along with item ids for recommendation.

The description of an item plays a pivotal role in providing concise and informative summaries to captivate potential viewers and is essential for recommendation systems. Traditionally, such descriptions were obtained through manual web scraping techniques, which are time-consuming and susceptible to data inconsistencies. In recent years, Large Language Models (LLMs), such as GPT-3.5, and open source LLMs like Alpaca have emerged as powerful tools for natural language processing tasks. In this paper, we have explored how we can use LLMs to generate detailed descriptions of the items. To conduct the study, we have used the MovieLens 1M dataset comprising movie titles and the Goodreads Dataset consisting of names of books and subsequently, an open-sourced LLM, Alpaca, was prompted with few-shot prompting on this dataset to generate detailed movie descriptions considering multiple features like the names of the cast and directors for the ML dataset and the names of the author and publisher for the Goodreads dataset. The generated description was then compared with the scraped descriptions using a combination of Top Hits, MRR, and NDCG as evaluation metrics. The results demonstrated that LLM-based movie description generation exhibits significant promise, with results comparable to the ones obtained by web-scraped descriptions.

In the real world, when we seek our friends’ opinions on various items or events, we request verbal social recommendations. It has been observed that we often turn to our friends for recommendations on a daily basis. The emergence of online social platforms has enabled users to share their opinion with their social connections. Therefore, we should consider users’ social connections to enhance online recommendation performance. The social recommendation aims to fuse social links with user-item interactions to offer more relevant recommendations. Several efforts have been made to develop an effective social recommendation system. However, there are two significant limitations to current methods: First, they haven’t thoroughly explored the intricate relationships between the diverse influences of neighbours on users’ preferences. Second, existing models are vulnerable to overfitting due to the relatively low number of user-item interaction records in the interaction space. For the aforementioned problems, this paper offers a novel framework called CR-SoRec, an effective recommendation model based on BERT and consistency regularization. This model incorporates Bidirectional Encoder Representations from Transformer(BERT) to learn bidirectional context-aware user and item embeddings with neighbourhood sampling. The neighbourhood Sampling technique samples the most influential neighbours for all the users/ items. Further, to effectively use the available user-item interaction data and social ties, we leverage diverse perspectives via consistency regularization to harness the underlying information. The main objective of our model is to predict the next item that a user would interact with based on its interaction behaviour and social connections. Experimental results show that our model defines a new state-of-the-art on various datasets and outperforms previous work by a significant margin. Extensive experiments are also conducted to analyze the proposed method.

Designing suitable deep model architectures for AI-driven on-device apps and features, at par with evolving mobile hardware and complex target scenarios, is difficult. Though Neural Architecture Search (NAS/AutoML) has made this easier by automated architecture learning from data saving substantial manual effort, yet it has major limitations, in context of mobile devices, including model-hardware alignment, prohibitive search times and divergence from primary target objective(s). So, we propose AUTOCOMET that can learn the most suitable DNN architecture optimized for varied types of device hardware and task contexts, ≈ 3× faster. Our novel co-regulated shaping reinforcement controller together with the high fidelity hardware meta-behavior predictor produces a smart, fast NAS framework that adapts to context via a generalized formalism for any kind of multi-criteria optimization.

The modern era of artificial intelligence is mostly driven by Deep Neural Network (DNN). As a result most of the intelligent/smart apps running on edge devices (mobile phones, televisions etc.) use DNN for their predictive ability. DNN based prediction suffers with operational overheads, which is the summation of model loading latency and inference latency. Model loading latency affects the first response of the DNN powered apps, whereas inference latency affects the subsequent responses. As apps switching has become a common practice among edge device users, so it is of utmost interest to make the switching smooth by reducing the model loading latency. In this paper asymmetrical DNN modularization is proposed as a potential solution. The proposed method solves two distinct problems (a). Improves the model loading latency by parallel loading of all the modules (child models) of given DNN model. (b). Helps in on-device training by keeping the live gradients only for last child model. The decision about modularization index and their corresponding positions are taken by reinforcement learning unit (RLU). RLU takes into account the available hardware resources on-device (eg. h/w threads), loading latency of each layer on dedicated compute units. In response, it provides the best modularization index k and their corresponding positions$\vec{p}$specific to the DNN model and device, where$P=(p_{1},p_{2},\ \ldots,p_{k})$and$p_{i}$is the end position of child$i$. The proposed method has shown significant loading time improvement (up to 7X) on popular DNNs, used for camera use-cases. Along with improving the loading latency the proposed modularization method facilitates for On-device personalization by separating the module with trainable layers and loading them particularly while training on-device.