Although multi-modal large language models (MLLMs) have impressive cross-modal reasoning and prediction capabilities, a unified and rigorous evaluation standard is still lacking. In this paper, we propose a future event prediction task to evaluate their cross-modal temporal prediction capability. This task requires the model to generate descriptions of events that may occur in future based on the input premise video. We build a dataset on the existing datasets for model evaluation. This task faces many challenges, including the complexity of processing video data, such as understanding changes in objects, actions, and time dimensions within the video and the interference of redundant information. To address these challenges, we propose a novel cross-modal prediction framework that introduces cross-modal supplementary learning and template-based reasoning chains based on MLLMs. Cross-modal supplementary learning aims to promote visual and text information to supplement and mine their respective information, primarily to capture critical information in videos, relying on the adaptive temporal filter and casual Q-Former. The template-based reasoning chain drives GPT-4 to generate a series of template question pairs through design prompts, gradually guiding the model to perform hierarchical reasoning to support the final prediction. Through experimental evaluation, the performance of the current MLLMs may not meet the requirements, and our model outperforms all existing models in predicting future events. It shows that the capabilities of MLLMs can be further explored.

Multimodal Large Language Models (MLLMs) have demonstrated remarkable abilities in image-language reasoning. However, they deal with Video Question Answering (VideoQA) insufficiently, especially for questions demanding causal-temporal reasoning. Typically, they directly concatenate features of uniformly sampled frames as visual inputs for VideoQA. This gives rise to two challenges. For one thing, uniformly sampled frames are discrete and separately distributed across different timestamps, disrupting the coherence of question-critical events or actions. For another, it considers every scene within videos equally and introduces redundant frames that may distract the model from discovering the truth. Towards this, we highlight the importance of identifying continuous frames that are crucial for answering the questions, and propose a lightweight and differentiableCoherenceRecognizer (CoRe) to achieve this. Guided by the semantics of questions,CoRecomputes scores recording the relevance between each frame and the question, and selects a set of continuous frames with the highest scores for answer prediction. Additionally,CoReencodes the unselected frames into a short and coarse-grained representation as a completion of the general context. Equipped withCoRe, we can efficiently fine-tune the current MLLMs for VideoQA in an end-to-end manner, without suffering from the problems of incoherence or distraction. Extensive experiments demonstrate that our method achieves substantial improvements on several VideoQA benchmarks.

Video Question Answering (VideoQA) aims to answer natural language questions based on the information observed in videos. Despite the recent success of Large Multimodal Models (LMMs) in image-language understanding, they deal with VideoQA insufficiently, by simply taking uniformly sampled frames as visual inputs, which ignores question-relevant visual clues. Moreover, there are no human annotations for question-critical timestamps in existing VideoQA datasets. In light of this, we propose a weakly supervised framework to enforce the LMMs to reason out the answers with question-critical moments as visual inputs. Specifically, we fuse the question and answer pairs as event descriptions to find multiple keyframes as target moments and pseudo-labels. With these pseudo-labeled keyframes as additionally weak supervision, we devise a lightweight Gaussian-based Contrastive Grounding (GCG) module. GCG learns multiple Gaussian masks to characterize the temporal structure of the video, and sample question-critical frames as positive moments to be the visual inputs of LMMs. Extensive experiments on several benchmarks verify the effectiveness of our framework, and we achieve substantial improvements compared to previous state-of-the-art methods.

Although multi-modal large language models possess impressive cross-modal reasoning and prediction capabilities, they lack a unified and rigorous evaluation standard. In this paper, we introduce a future event prediction task to assess the cross-modal temporal prediction capabilities of these models. This task requires the model to generate descriptions of events that may occur in the future based on input video. To tackle this new task, we propose an object-centric cross-modal knowledge reasoning framework, which combines a basic information encoder, an adaptive multi-segment filter, a spatial-temporal relation encoder, a vision-text interaction module, and a pre-trained large language model decoder. The adaptive multi-segment filter captures selectively capture critical visual information in videos, enhancing the model’s focus on relevant features. The spatial-temporal relation encoder decomposes and associates the objects and scene information in the video. Additionally, the vision-text interaction module enhances the connection between visual sequences and their corresponding textual narratives, ensuring semantic coherence and consistency. To evaluate our framework, we constructed a dataset containing descriptions, dialogues of future events, and object-centric event reasoning chains. Experimental results indicate that the proposed framework outperforms all previous methods for future event prediction. Ablation studies further demonstrate the effectiveness of the designed modules.

In this paper, we develop a weakly supervised learning algorithm to learn robust semantic correspondences from large-scale datasets with only image-level labels. Following the spirit of multiple instance learning (MIL), we decompose the weakly supervised correspondence learning problem into three stages: image-level matching, region-level matching, and pixel-level matching. We propose a novel cascaded online correspondence refinement algorithm to integrate MIL and the correspondence filtering and refinement procedure into a single deep network and train this network end-to-end with only image-level supervision, i.e., without point-to-point matching information. During the correspondence learning process, pixel-to-pixel matching pairs inferred from weak supervision are propagated, filtered, and enhanced through masked correspondence voting and calibration. Besides, we design a correspondence consistency check algorithm to select images with discriminative key points to generate pseudo-labels for classical matching algorithms. Finally, we filter out about 110,000 images from the ImageNet ILSVRC training set to formulate a new dataset, called SC-ImageNet. Experiments on several popular benchmarks indicate that pre-training on SC-ImageNet can improve the performance of state-ofthe-art algorithms efficiently. Our project is available on https://github.com/21210240056/SC-ImageNet.