Visual Relationship Forecasting (VRF) in video aims to anticipate relations among objects without observing future visual content. The task relies on capturing and modeling the semantic coherence in object interactions, as it underpins the evolution of events and scenes in videos. However, existing VRF datasets provide limited support for learning such coherence. Their noisy annotations fail to reflect distinct action or scene changes, and weak correlations exist between different actions and relational transitions in subject-object pairs. Furthermore, existing methods struggle to distinguish similar relationships and overfit to unchanging relationships in consecutive frames rather than reflecting the semantic coherence of object interactions. To address these challenges, we present SemCoBench, a benchmark that emphasizes semantic coherence for visual relationship forecasting in video. Based on action labels and short-term subject-object pairs, SemCoBench decomposes relationship categories and dynamics by cleaning and reorganizing video datasets to ensure predicting semantic coherence in object interactions. In addition, we propose the Semantic Coherent Transformer (SemCoFormer), which consists of a Relationship Augmented Module (RAM) and a Coherent Reasoning Module (CRM). The RAM bidirectionally enhances cross-modal features to better distinguish similar relationships, while the CRM performs sparse encoding of cross-frame relation transitions to model dynamic relational changes. The experimental results on SemCoBench demonstrate that modeling the semantic coherence is a key step toward reasonable, fine-grained, and diverse visual relationship forecasting, contributing to a more comprehensive understanding of video scenes. Our project page: https://lyao-61.github.io/ SemCo/.

Prompt learning has emerged as an effective strategy for adapting multimodal vision-language models to downstream tasks such as video action recognition, which has been widely applied in human-computer interaction, video surveillance and medical fields. However, existing approaches often suffer from overfitting to task-specific distributions and prompt collapse, especially under limited data or when class semantics are highly similar. In this work, we propose Cross-modal Consistent and Prompt-diverse Regularized CLIP (CPR-CLIP), a flexible framework that enhances the generalization of prompt learning in action recognition. Specifically, we introduce Cross-modal Consistency Regularization (CCR) strategy to preserve the original feature representations encoded by the frozen CLIP encoder, thereby reducing overfitting caused by prompt adaptation. Additionally, we design a Prompt Diversity Regularization (PDR) term to encourage category-wise separation among prompts, alleviating prompt collapse and enhancing discriminability. Experiments on HMDB51, UCF101, and SSv2 under base-to-novel and few-shot settings show that CPR-CLIP consistently outperforms existing methods, achieving strong generalization to novel classes.

Action recognition in dynamic video remains challenging, particularly when distinguishing between visually similar actions. While existing methods often rely on holistic representations, they overlook the fine-grained details that are significant for accurate classification. We propose a novel Fine-grained Teacher-student CLIP (FT-CLIP) that integrates body part analysis with holistic action recognition through a teacher-student architecture, bridging the gap between fine-grained action parsing and overall action understanding. The teacher model processes individual body parts alongside specialized description to generate part-specific features, which are then aggregated and distilled into the student model. Through knowledge distillation with learnable prompts, the student model effectively learns to capture subtle action distinctions while maintaining efficient inference. FT-CLIP achieves a more nuanced understanding of complex actions by progressing from detailed body part analysis to comprehensive action recognition. Experiments on Kinetics-TPS under a fully-supervised setting and on HMDB51 and UCF101 under a zero-shot setting demonstrate the effectiveness of our method.

As a video understanding task, activity parsing aims at encompassing actions into multiple levels of activity components, including activity, sub-activity and atomic action, enabling understanding of complex video scenes within multimedia systems. Existing methods form activity parsing as a multi-task learning problem to predict multi-granular activity labels simultaneously, which ignores modeling the hierarchical structure and the fine-grained transitions of activity components at different levels. In this paper, we propose a Hierarchical Adaptive Reasoning Graph (HARG) to model the hierarchical structure (i.e., object level$\rightarrow$atomic action level$\rightarrow$activity level) dynamically and precisely. To achieve that, an object reasoning graph (ORG) and an atomic action reasoning graph (ARG) are designed to reason fine-grained information transitions between multiple actors at different levels. In addition, an adaptive segmentation module (ASM) is investigated for bridging the gap among different levels, permitting step-by-step reasoning from the object level to the atomic action level. Experimental results show our method outperforms state-of-the-art methods on two activity parsing datasets, achieving hierarchical modeling and fine-grained reasoning for activity understanding. The code is available on GitHub:https://github.com/whuoyj/HARG.

Action recognition is a challenging task since the attributes of objects as well as their relationships change constantly in the video. Existing methods mainly use object-level graphs or scene graphs to represent the dynamics of objects and relationships, but ignore modeling the fine-grained relationship transitions directly. In this paper, we propose an Object-Relation Reasoning Graph (OR2G) for reasoning about action in videos. By combining an object-level graph (OG) and a relation-level graph (RG), the proposed OR2G catches the attribute transitions of objects and reasons about the relationship transitions between objects simultaneously. In addition, a graph aggregating module (GAM) is investigated by applying the multi-head edge-to-node message passing operation. GAM feeds back the information from the relation node to the object node and enhances the coupling between the object-level graph and the relation-level graph. Experiments in video action recognition demonstrate the effectiveness of our approach when compared with the state-of-the-art methods.

With the rapid development of video distribution and broadcasting, affective video content analysis has attracted a lot of research and development activities recently. Predicting emotional responses of movie audiences is a challenging task in affective computing, since the induced emotions can be considered relatively subjective. In this article, we propose a multimodal local-global attention network (MMLGAN) for affective video content analysis. Inspired by the multimodal integration effect, we extend the attention mechanism to multi-level fusion and design a multimodal fusion unit to obtain a global representation of affective video. The multimodal fusion unit selects key parts from multimodal local streams in the local attention stage and captures the information distribution across time in the global attention stage. Experiments on the LIRIS-ACCEDE dataset, the MediaEval 2015 and 2016 datasets, the FilmStim dataset, the DEAP dataset and the VideoEmotion dataset demonstrate the effectiveness of our approach when compared with the state-of-the-art methods.

Pan-sharpening is a fundamental task for remote sensing image processing. It aims at creating a high-resolution multispectral (HRMS) image from a multispectral (MS) image and a panchromatic (PAN) image. In this article, a new band-independent encoder-decoder network is proposed for pan-sharpening. The network takes a single band of the MS (BMS) image, the PAN image, and the low-resolution PAN (LRPAN) image as inputs. The output of the network is the corresponding band of high-resolution MS (HRBMS) image. In this way, the network can process MS images with any number of bands. The overall structure of the network consists of two encoder-decoder modules at low-resolution and high-resolution, respectively. An auxiliary LRPAN image is used to speed up the training and improve the performance. The partly shared network and hierarchical structure for low-resolution and high-resolution enable a better fusion of features extracted from different scales. With a fast fine-tuning strategy, the trained model can be applied to images from different sensors. Experiments performed on different data sets demonstrate that the proposed method outperforms several state-of-the-art pan-sharpening methods in both visual appearance and objective indexes, and the single-band evaluation results further verify the superiority of the proposed method.

Pan-sharpening is an important preprocessing step for remote sensing image processing tasks; it fuses a low-resolution multispectral image and a high-resolution (HR) panchromatic (PAN) image to reconstruct a HR multispectral (MS) image. This paper introduces a new end-to-end bidirectional pyramid network for pan-sharpening. The overall structure of the proposed network is a bidirectional pyramid, which permits the network to process MS and PAN images in two separate branches level by level. At each level of the network, spatial details extracted from the PAN image are injected into the upsampled MS image to reconstruct the pan-sharpened image from coarse resolution to fine resolution. Subpixel convolutional layers and the enhanced residual blocks are used to make the network efficient. Comparison of the results obtained with our proposed method and the results using other widely used state-of-the-art approaches confirms that our proposed method outperforms the others in visual appearance and objective indexes.