Long-term Action Quality Assessment (AQA) is of significant importance in applications such as sports analysis and medical rehabilitation. Compared with short-term AQA, long-term AQA involves longer temporal spans and more complex motion structures, posing greater challenges to spatio-temporal modeling. Existing methods typically rely on a single modality, either video or skeleton data. Video-based approaches are easily affected by background noise, while skeleton-based methods lack visual appearance information, making it difficult to comprehensively represent action quality. To address these limitations, we propose a multimodal framework termed Skeleton-Guided Spatio-Temporal Video Representation (SG-STVR). By introducing skeletal priors, we develop a Spatial Prior Injection (SPI) strategy that utilizes human foreground masks generated from skeleton data to guide the model to focus on action-relevant regions and suppress background interference. In addition, we propose a Temporal Prior Guidance (TPG) strategy, which enhances the temporal modeling capability of video features based on the motion sensitivity of skeleton sequences. Experimental results on the RG and Fis-V datasets demonstrate that SG-STVR outperforms other approaches, reflecting its competitiveness in long-term AQA task.

Weakly supervised text-based person re-identification aims to retrieve specific pedestrians based on textual descriptions without identity labels available during training. This task remains challenging due to the inherent cross-modal heterogeneity and lack of identity annotations. There is a common issue of modality gap in vision language models, which in turn affects the performance of downstream tasks such as cross-modal retrieval and multimodal clustering. Specifically, in our research and experiments, we found that there is a problem of inter-modal misalignment between image and text modalities. However, existing methods rely on mutual enhancement strategies between image and text clustering, leading to the accumulation of clustering noise and affecting the final retrieval performance. To address this issue, we propose a Consensus Labelling: Prompt-guided Clustering refinement (CLPC) framework for weakly supervised text-based person re-identification. Specifically, we introduce a textual inversion network to learn a pseudo token that captures visual context, which is then integrated into natural language sentences as personalized textual prompt. To further improve clustering quality, we introduce a Nearest Neighbor-Guided Pseudo Label Mining (NGPM) method, which uses the clusters derived from personalized textual prompts to refine the clustering of image features. Additionally, we design a Dynamic Margin Triplet (DMT) loss, where the margin is adaptively adjusted using a sigmoid-based function to enhance the model’s ability to distinguish hard negative samples. We have also introduce a Normalized Distribution Matching (NDM) loss to minimize the KL divergence between the image-text matching scores and the normalized soft matching scores. The extensive experimental results on three public datasets have demonstrated the superiority of our method. Our code is available at https://github.com/LeviWeiZhi/CLPC.

In partial label learning (PLL) tasks, each training instance is assigned a candidate label set, with only one label being correct. Previous studies on PLL have focused on scenarios where the class label set remains fixed, i.e., the label set for test data is the same as that used during training. However, in many real-world applications, the environment is dynamic, and new labels may emerge, requiring methods that can detect and classify these new labels. Moreover, previous methods typically learn from partial label data by manipulating the same feature set, which may be suboptimal as it overlooks the semantic relationships between instances and labels. To this end, we develop a novel PLL approach called Label-Specific feature-based Partial label learning with Emerging new Labels (LSPEL), which works by iteratively learning label-specific features during the label disambiguation process to support new label detection and model update. It consists of three key components: (1) model training based on label-specific feature learning, (2) construction of a new label detector that works in conjunction with the classifier to predict known labels, and (3) model updating and induction to further enhance the prediction results for known labels. Extensive experiments on synthetic and real-world PL datasets demonstrate that LSPEL is effective in handling emerging new labels.

Prior work on few-shot action recognition predominantly adopts two strategies: spatio-temporal separated frame matching and multi-stream multi-modal networks. However, each suffering from either incomplete spatio-temporal modeling or an over-reliance on additional annotation data. To address these limitations, we propose a Contrastive Single-Stream Spatio-Temporal joint modeling Few-Shot Action Recognition (CS3T-FSAR) model. In terms of spatio-temporal modeling, our approach directly constructs high-quality three-dimensional spatio-temporal representations to fully capture the global associations among video frames. Regarding the loss function design, we integrate a triplet loss to achieve precise matching while reducing both inference cost and computational complexity. Ultimately, our method achieves significant performance improvements across four benchmark datasets, demonstrating its competitiveness in few-shot action recognition.

Online Action Detection (OAD) is a fundamental task in video action understanding, aiming to predict ongoing actions in real-time from untrimmed video streams. However, existing models face difficulties in distinguishing feature differences between action frames and neutral transition frames. To address this challenge, we propose the Irrelevant Frames Filtering Network (IFFN), a dual-branch architecture comprising an action branch and an irrelevant frames filtering branch. The IFFN employs an Irrelevant Frame Filtering Unit (IFFU) to extract informative foreground frames from streaming videos. To further enhance the model’s discriminative ability, we introduce a novel training strategy that utilizes opposing training objectives and weight-sharing mechanisms to better distinguish neutral transition frames from foreground frames. We evaluate IFFN on three benchmark datasets: the THUMOS’14, TVSeries, and MultiTHUMOS. Experimental results demonstrate that IFFN outperforms all models used for comparison. Furthermore, we integrate IFFN into OadTR, LSTR, and MiniROAD, which are representative models based on Transformer and RNN architectures. Experiments demonstrate that IFFN can serve as a plug-and-play module to consistently enhance the performance of OAD models.

Accurate recognition and localization of 3D objects is a fundamental research problem in 3D computer vision. Benefiting from transformation-free point cloud processing and flexible receptive fields, point-based methods have become accurate in 3D point cloud modeling, but still fall behind voxel-based competitors in 3D detection. We observe that the set abstraction module, commonly utilized by point-based methods for downsampling points, tends to retain excessive irrelevant background information, thus hindering the effective learning of features for object detection tasks. To address this issue, we propose MSSA, a Multi-representation Semantics-augmented Set Abstraction for 3D object detection. Specifically, we first design a backbone network to encode different representation features of point clouds, which extracts point-wise features through PointNet to preserve fine-grained geometric structure features, and adopts VoxelNet to extract voxel features and BEV features to enhance the semantic features of key points. Second, to efficiently fuse different representation features of keypoints, we propose a Point feature-guided Voxel feature and BEV feature fusion (PVB-Fusion) module to adaptively fuse multi-representation features and remove noise. At last, a novel Multi-representation Semantic-guided Farthest Point Sampling (MS-FPS) algorithm is designed to help set abstraction modules progressively downsample point clouds, thereby improving instance recall and detection performance with more important foreground points. We evaluate MSSA on the widely used KITTI dataset and the more challenging nuScenes dataset. Experimental results show that compared to PointRCNN, our method improves the AP of “moderate” level for three classes of objects by 7.02%, 6.76%, and 5.44%, respectively. Compared to the advanced point-voxel-based method PV-RCNN, our method improves the AP of “moderate” level by 1.23%, 2.84%, and 0.55% for the three classes, respectively.

Multilabel feature selection (MFS) has received widespread attention in various big data applications. However, most of the existing methods either explicitly or implicitly assume that all labels are given in advance before feature selection starts; or that all labels are independent. In fact, in many practical applications, the available labels usually arrive dynamically, and they may be interdependent with each other. Moreover, labels may be generated dynamically in a minibatch manner, which makes it more difficult to explore label dependency. In this article, we propose a novel fuzzy mutual information-based multilabel feature selection approach MSDS, which is able to solve single streaming label, minibatch streaming labels, and exploit label dependency simultaneously. In specific, we first promote fuzzy mutual information to be suitable for multilabel learning. This model can effectively consider the relationship between two labels, and has good applicability for measuring the relationship between multiple labels. Then, we analyze feature relevance and feature redundancy based on the combination of label dependency and streaming labels, which helps to facilitate the selection of high-quality feature subsets. Finally, a feature conversion is designed to fuse the representative features of new arrival streaming labels. Comprehensive experiments on twelve multilabel datasets clearly reveal the superiority of the proposed method against two streaming labels based algorithms and five state-of-the-art static label space based algorithms.

Transformer-based and interaction point-based methods have demonstrated promising performance and potential in human-object interaction detection. However, due to differences in structure and properties, direct integration of these two types of models is not feasible. Recent Transformer-based methods divide the decoder into two branches: an instance decoder for human-object pair detection and a classification decoder for interaction recognition. While the attention mechanism within the Transformer enhances the connection between localization and classification, this paper focuses on further improving HOI detection performance by increasing the intrinsic correlation between instance and action features. To address these challenges, this paper proposes a novel Transformer-based HOI Detection framework. In the proposed method, the decoder contains three parts: learnable query generator, instance decoder, and interaction classifier. The learnable query generator aims to build an effective query to guide the instance decoder and interaction classifier to learn more accurate instance and interaction features. These features are then applied to update the query generator for the next layer. Especially, inspired by the interaction point-based HOI and object detection methods, this paper introduces the prior bounding boxes, keypoints detection and spatial relation feature to build the novel learnable query generator. Finally, the proposed method is verified on HICO-DET and V-COCO datasets. The experimental results show that the proposed method has the better performance compared with the state-of-the-art methods.

Abstract Multi‐scale feature fusion is often used to address the issue of scale variations in object detection. However, most of the proposed network architectures only combine the features of two adjacent levels sequentially, so the first fusion nodes in both top‐down and bottom‐up pathways must be blank nodes that only have one input with no feature fusion. In this work, cross‐scale feature fusion connection (CFFC) is proposed which aims to enhance the entire feature hierarchy by propagating the features of each level more efficiently. The proposed method reuses and aggregates all the features of other scales to the blank nodes in both top‐down and bottom‐up pathways. Furthermore, the authors remove the 1 × 1 convolutional layer and replace the shortcut with concatenation before fusing multiple features. These concatenated feature maps are then supervised by the channel attention block at the fusion nodes. This modification allows the network to learn the important degree of each level in concatenated feature maps along the channel dimension. It is also observed that the proposed method alleviates the inconsistency in feature pyramids with fewer parameters. The performance of a YOLO object detector equipped with the proposed method on the COCO test‐dev 2017 is evaluated. The results show that the proposed method outperforms other architectures presented in the literature.

Describing video content using natural language is an important part of video understanding. It needs to not only understand the spatial information on video, but also capture the motion information. Meanwhile, video captioning is a cross-modal problem between vision and language. Traditional video captioning methods follow the encoder-decoder framework that transfers the video to sentence. But the semantic alignment from sentence to video is ignored. Hence, finding a discriminative visual representation as well as narrowing the semantic gap between video and text has great influence on generating accurate sentences. In this paper, we propose an approach based on multi-feature fusion refine network (MFRN), which can not only capture the spatial information and motion information by exploiting multi-feature fusion, but also can get better semantic aligning of different models by designing a refiner to explore the sentence to video stream. The main novelties and advantages of our method are: (1) multi-feature fusion: Both two-dimension convolutional neural networks and three-dimension convolutional neural networks pre-trained on ImageNet and Kinetic respectively are used to construct spatial information and motion information, and then fused to get better visual representation. (2) Sematic alignment refiner: the refiner is designed to restrain the decoder and reproduce the video features to narrow semantic gap between different modal. Experiments on two widely used datasets demonstrate our approach achieves state-of-the-art performance in terms of BLEU@4, METEOR, ROUGE and CIDEr metrics.

Current methods for human-object interaction detection often use the spatial relation between a human and an object as an interaction pattern. However, this strategy is relatively simple and has low discrimination in similar interactions. To solve this drawback, the spatial relation between skeletons and objects is proposed to model the interaction pattern and improve the detection accuracy. First, the skeleton-object interaction pattern image is extracted for each interaction proposal. Second, a deep neural network is applied to learn the interaction features from these images. Finally, the interaction feature is added to the human-object interaction detection network by a multistream structure. In the experiments, we evaluate the proposed method on the HICO-DET and V-COCO datasets. Experimental results show that the proposed method can achieve the best performance compared with state-of-art methods.

The local key features in video are important for improving the accuracy of human action recognition. However, most end-to-end methods focus on global feature learning from videos, while few works consider the enhancement of the local information in a feature. In this article, we discuss how to automatically enhance the ability to discriminate the local information in an action feature and improve the accuracy of action recognition. To address these problems, we assume that the critical level of each region for the action recognition task is different and will not change with the region location shuffle. We therefore propose a novel action recognition method called the shuffle-invariant network. In the proposed method, the shuffled video is generated by regular region cutting and random confusion to enhance the input data. The proposed network adopts the multitask framework, which includes one feature backbone network and three task branches: local critical feature shuffle-invariant learning, adversarial learning, and an action classification network. To enhance the local features, the feature response of each region is predicted by a local critical feature learning network. To train this network, an L 1-based critical feature shuffle-invariant loss is defined to ensure that the ordered feature response list of these regions remains unchanged after region location shuffle. Then, the adversarial learning is applied to eliminate the noise caused by the region shuffle. Finally, the action classification network combines these two tasks to jointly guide the training of the feature backbone network and obtain more effective action features. In the testing phase, only the action classification network is applied to identify the action category of the input video. We verify the proposed method on the HMDB51 and UCF101 action datasets. Several ablation experiments are constructed to verify the effectiveness of each module. The experimental results show that our approach achieves the state-of-the-art performance.

Abstract Generally, it is regarded as challenge work to grasp the drawing style of an ancient masterpiece in Chinese painting learning. This paper presents a novel approach to the generation of a Chinese ink painting in a certain style and animating its brushwork process with expert skills. In order to demonstrate the techniques of brush and ink inside a stroke, a serials of geometric properties of a brush stroke, are first extracted, then through rational deformation calculation, the best stroke source is mapped onto the stroking path, which is sketched by the user, and finally a new Chinese painting can be synthesized by style migration and natural stroke composition. So with the generated strokes, the lifelike brushwork process of the new painting can be represented dramatically. Actually, by showing the authentic painting process, the tool we implemented helps the learners, who have no profound skills and knowledge in domain of Chinese painting, master the essence of a great painting style, and also provides an easy way to art creation and the comprehension of mysterious Chinese traditional art.

In mainstream approaches for 3D human action recognition, depth and skeleton features are combined to improve recognition accuracy. However, this strategy results in high feature dimensions and low discrimination due to redundant feature vectors. To solve this drawback, a multi-feature selection approach for 3D human action recognition is proposed in this paper. First, three novel single-modal features are proposed to describe depth appearance, depth motion, and skeleton motion. Second, a classification entropy of random forest is used to evaluate the discrimination of the depth appearance based features. Finally, one of the three features is selected to recognize the sample according to the discrimination evaluation. Experimental results show that the proposed multi-feature selection approach significantly outperforms other approaches based on single-modal feature and feature fusion.

Action recognition is one of the hottest research topics in computer vision. Recent methods represent actions based on global or local video features. These approaches, however, lack semantic structure and may not provide a deep insight into the essence of an action. In this work, the authors argue that semantic clues, such as joint positions and part‐level motion clustering, help verify actions. To this end, a meta‐action descriptor for action recognition in RGBD video is proposed in this study. Specifically, two discrimination‐based strategies – dynamic and discriminative part clustering – are introduced to improve accuracy. Experiments conducted on the MSR Action 3D dataset show that the proposed method significantly outperforms the methods without joint position semantic.

Label imbalance and the insufficiency of labeled training samples are major obstacles in most methods for counting people in images or videos. In this work, a sparse representation-based semi-supervised regression method is proposed to count people in images with limited data. The basic idea is to predict the unlabeled training data, select reliable samples to expand the labeled training set, and retrain the regression model. In the algorithm, the initial regression model, which is learned from the labeled training data, is used to predict the number of people in the unlabeled training dataset. Then, the unlabeled training samples are regarded as an over-complete dictionary. Each feature of the labeled training data can be expressed as a sparse linear approximation of the unlabeled data. In turn, the labels of the labeled training data can be estimated based on a sparse reconstruction in feature space. The label confidence in labeling an unlabeled sample is estimated by calculating the reconstruction error. The training set is updated by selecting unlabeled samples with minimal reconstruction errors, and the regression model is retrained on the new training set. A co-training style method is applied during the training process. The experimental results demonstrate that the proposed method has a low mean square error and mean absolute error compared with those of state-of-the-art people-counting benchmarks.

In this study, the authors investigate the possibility of boosting action recognition performance by exploiting the associated scene context. Towards this end, the authors model a scene as a mid‐level ‘middle layer’ in order to bridge action descriptors and action categories. This is achieved via a scene topic model, in which hybrid visual descriptors, including spatial–temporal action features and scene descriptors, are first extracted from a video sequence. Then, the authors learn a joint probability distribution between scene and action using a naive Bayes nearest neighbour algorithm, which is adopted to jointly infer the action categories online by combining off‐the‐shelf action recognition algorithms. The authors demonstrate the advantages of their approach by comparing it with state‐of‐the‐art approaches using several action recognition benchmarks.

Recognizing human actions is not alone, as hinted by the scene herein. In this paper, we investigate the possibility to boost the action recognition performance by exploiting their scene context associated. To this end, we model the scene as a mid-level “hidden layer” to bridge action descriptors and action categories. This is achieved via a scene topic model, in which hybrid visual descriptors including spatiotemporal action features and scene descriptors are first extracted from the video sequence. Then, we learn a joint probability distribution between scene and action by a Naive Bayesian N-earest Neighbor algorithm, which is adopted to jointly infer the action categories online by combining off-the-shelf action recognition algorithms. We demonstrate our merits by comparing to state-of-the-arts in several action recognition benchmarks.