This paper tackles the fundamental failure of Large Language Models (LLMs) to solve new tasks when prompted with a sufficient, yet overly complex, set of multi-modal episodes. This failure stems from the model's inability to distill underlying patterns from the noisy experiences. We propose Hypothesis-Driven Reasoning (HDR), a framework that enhances LLM reasoning by building an explicit semantic memory—a set of hypotheses induced from the multi-modal episodes. HDR employs a two-stage pipeline. It first extracts potential factors from the episodes and then iteratively refines hypotheses by generate-verify loop with the factors. We first empirically demonstrates this failure and the potential of sematic memory, showing that oracle hypotheses can boost accuracy from 35.3% to 92.0% on a novel task we designed. We then evaluate our HDR, achieving near-oracle performance and significantly outperforming baselines, especially on smaller models. This paper validates a shift from unstructured in-context recall to explicit knowledge abstraction for robust reasoning.

Comprehending long videos remains a significant challenge for Large Multi-modal Models (LMMs). Current LMMs struggle to process even minutes to hours videos due to their lack of explicit memory and retrieval mechanisms. To address this limitation, we propose RAVU (Retrieval Augmented Video Understanding), a novel framework for video understanding enhanced by retrieval with compositional reasoning over a spatio-temporal graph. We construct a graph representation of the video, capturing both spatial and temporal relationships between entities. This graph serves as a long-term memory, allowing us to track objects and their actions across time. To answer complex queries, we decompose the queries into a sequence of reasoning steps and execute these steps on the graph, retrieving relevant key information. Our approach enables more accurate understanding of long videos, particularly for queries that require multi-hop reasoning and tracking objects across frames. Our approach demonstrate superior performances with limited retrieved frames (5-10) compared with other SOTA methods and baselines on two major video QA datasets: NExT-QA and EgoSchema.

Human image synthesis with pose guidance generates images of a specified human in a given pose, a task complicated by dis-occlusions and varying body articulations. While generative model-based approaches are effective, they often require paired training data, limiting generalizability. Recent selfsupervised methods, such as reconstruction from body parts and jigsaw puzzle-solving, face issues like pose leaking and inadequate appearance encoding. We propose a novel approach that learns to reconstruct images from body parts using a body symmetricity loss, leveraging human body symmetries. Our method preserves appearance information and mitigates pose leaking by aligning appearance features of corresponding body parts from symmetric left-right halves. Additionally, we leverage pretrained models, specifically stable-diffusion, to enhance performance and training efficiency. Extensive experiments and ablation studies on the deepfashion dataset demonstrate our method’s effectiveness.

Human-Object Interaction (HOI) detection aims to localize human-object pairs and recognize their interactions astriplets. Recent advancements in pre-trained vision-language model (VLM) have improved zero-shot HOI detection, enabling identification of unseen triplets. However, existing methods leverage the VLM as an additional encoder only for interaction prediction, not for human/object detection. This limitation hinders their ability to detect unseen objects. Furthermore, the additional encoder increases both model size and computational cost. This paper proposes a novel HOI detection framework ECI-HOI, which unleashes potentials of the pre-trained VLM for the zero-shot HOI detection by leveraging it for both of the sub-tasks. We first employ CLIP as a single image encoder, reducing redundancy in the network architecture. In addition, we propose an instance selector and a HO pair decoder to effectively harmonize the human/object detection and the interaction prediction in zero-shot manner. We evaluate our model under various settings on HICO-DET and our two new testsets: out-of-distribution image testset and novel object testset. Our model outperforms the state-of-the-art models while reducing the model size by more than 50%, especially achieving a + 1 0.01 mAP improvement under the unseen object setting on HICO-DET. The results on the proposed datasets highlight the zero-shot performance of our model on more challenging settings.

Transformers achieve great performance on Visual Question Answering (VQA). However, their systematic generalization capabilities, i.e., handling novel combinations of known concepts, is unclear. We reveal that Neural Module Networks (NMNs), i.e., question-specific compositions of modules that tackle a sub-task, achieve better or similar systematic generalization performance than the conventional Transformers, even though NMNs' modules are CNN-based. In order to address this shortcoming of Transformers with respect to NMNs, in this paper we investigate whether and how modularity can bring benefits to Transformers. Namely, we introduce Transformer Module Network (TMN), a novel NMN based on compositions of Transformer modules. TMNs achieve state-of-the-art systematic generalization performance in three VQA datasets, improving more than 30% over standard Transformers for novel compositions of sub-tasks. We show that not only the module composition but also the module specialization for each sub-task are the key of such performance gain.