Comprehending long videos remains a significant challenge for Large Multi-modal Models (LMMs). Current LMMs struggle to process even minutes to hours videos due to their lack of explicit memory and retrieval mechanisms. To address this limitation, we propose RAVU (Retrieval Augmented Video Understanding), a novel framework for video understanding enhanced by retrieval with compositional reasoning over a spatio-temporal graph. We construct a graph representation of the video, capturing both spatial and temporal relationships between entities. This graph serves as a long-term memory, allowing us to track objects and their actions across time. To answer complex queries, we decompose the queries into a sequence of reasoning steps and execute these steps on the graph, retrieving relevant key information. Our approach enables more accurate understanding of long videos, particularly for queries that require multi-hop reasoning and tracking objects across frames. Our approach demonstrate superior performances with limited retrieved frames (5-10) compared with other SOTA methods and baselines on two major video QA datasets: NExT-QA and EgoSchema.

Human image synthesis with pose guidance generates images of a specified human in a given pose, a task complicated by dis-occlusions and varying body articulations. While generative model-based approaches are effective, they often require paired training data, limiting generalizability. Recent selfsupervised methods, such as reconstruction from body parts and jigsaw puzzle-solving, face issues like pose leaking and inadequate appearance encoding. We propose a novel approach that learns to reconstruct images from body parts using a body symmetricity loss, leveraging human body symmetries. Our method preserves appearance information and mitigates pose leaking by aligning appearance features of corresponding body parts from symmetric left-right halves. Additionally, we leverage pretrained models, specifically stable-diffusion, to enhance performance and training efficiency. Extensive experiments and ablation studies on the deepfashion dataset demonstrate our method’s effectiveness.

Convolutional neural networks have been successful in restoring images captured under poor illumination conditions. Nevertheless, such approaches require a large number of paired low-light and ground truth images for training. Thus, we study the problem of semi-supervised learning for low-light image restoration when limited low-light images have ground truth labels. Our main contributions in this work are twofold. We first deploy an ensemble of low-light restoration networks to restore the unlabeled images and generate a set of potential pseudo-labels. We model the contrast distortions in the labeled set to generate different sets of training data and create the ensemble of networks. We then design a contrastive self-supervised learning based image quality measure to obtain the pseudo-label among the images restored by the ensemble. We show that training the restoration network with the pseudo-labels allows us to achieve excellent restoration performance even with very few labeled pairs. We conduct extensive experiments on three popular low-light image restoration datasets to show the superior performance of our semi-supervised low-light image restoration compared to other approaches. Project page is available at https://github.com/sameerIISc/SSL-LLR.

We study the problem of low light image restoration through contrast enhancement and denoising. We approach this problem by learning a model that relates a noisy low light and well lit image pair. The low light image is modeled to suffer from contrast distortion and additive noise. In particular, we model the loss of contrast through a global parametric function, which enables the estimation of the underlying noise. We then use a pair of convolutional neural network (CNN) models to learn the noise and the parameters of a function to achieve contrast enhancement. This contrast enhancement function is modeled as a linear combination of multiple gamma enhancers. We show through extensive evaluations that our Low Light Image Model for Enhancement Network (LLIMENet) achieves superior restoration performance when compared to other methods on several publicly available datasets.

We consider the problem of low light image restoration through joint contrast enhancement and denoising. Deep convolutional neural networks (CNNs) based on residual learning have been successful in achieving state of the art performance in image denoising. However, their application to joint contrast enhancement and denoising poses challenges owing to the nature of the distortion process involving both loss of details and noise. Thus, we propose a multiscale learning approach by learning the subbands obtained in a Laplacian pyramid decomposition through a subband CNN (SCNN). The enhanced subbands at multiple scales are then combined to obtain the final restored image using a recomposition CNN (ReCNN). We refer to the overall network involving SCNN and ReCNN as low light restoration network (LLRNet). We show through extensive experiments based on the `See in the Dark' Dataset that our approach produces better quality restored images when compared to other contrast enhancement techniques and CNN based approaches.