Large Vision-Language Models (LVLMs) suffer from the high computational cost of the attention mechanism caused by the large number of visual tokens. Token reduction has emerged as a promising approach to reduce the complexity by eliminating redundant visual tokens. However, existing token reduction methods struggle to preserve task-relevant tokens and eliminate irrelevant ones. This is due to the attention biases of LVLMs, where tokens with high attention scores are not always the critical ones. Such biases force existing methods into a dilemma: they face either high performance degradation or limited inference acceleration. This issue becomes more severe in fine-grained perception tasks, which rely heavily on the fine-grained information stored in specific visual tokens. To address the above issue, we propose an unbiased fine-grained token reduction method named FinePruner, which explores the attention patterns of LVLMs at the attention-head-level to mitigate the interference of attention biases. Concretely, we first conducted comparative studies to validate the impact of tokens corresponding to visual objects on final task performance, which established the conclusion that these tokens should be preserved while others can be pruned. Also, a series of visualizations unveils the changing patterns of LVLMs' attention biases across layers and attention heads. Based on the patterns of attention biases, the pipeline of FinePruner is divided into two stages. The first stage, named Instruction-Agnostic Clustering, clusters visual tokens into groups according to their embeddings to exclude the attention biases. The second stage, named Attention-Refined Pruning, selects attention heads with less bias by the divergence, which are used to identify the preserved tokens. Experiments on VQA benchmarks and fine-grained benchmarks demonstrate that our FinePruner achieves better accuracy-efficiency tradeoffs than state-of-the-art methods. The code is available at https://github.com/PKU-ICST-MIPL/FinePruner_TIP2026.

Compared with vision-based methods, millimeter-wave (mmWave) radar offers a non-invasive and privacy-preserving approach to human activity recognition (HAR), making it a robust solution for tasks requiring reliable performance under challenging conditions. Its ability to operate effectively in low-light or partially occluded environments renders it particularly suitable for applications such as smart healthcare and human-computer interaction. This paper introduces STFNet, an end-to-end spatiotemporal feature fusion network for human activity recognition using mmWave radar point clouds. The proposed method first employs a multi-scale spatial feature encoding module, which integrates multiple customized adaptive spatial feature extraction blocks at different scales to capture multi-level spatial information. A subsequent Transformer-based network then fuses these multi-scale features to enrich spatial representations. Next, a Bidirectional Gated Recurrent Unit (BiGRU) module is introduced to model the temporal dynamics present in the point cloud data. Finally, a Squeeze-and-Excitation Residual Network (SE-ResNet) is utilized to adaptively calibrate channel importance in the concatenated spatiotemporal features, significantly boosting recognition accuracy for complex activities. Experimental evaluation on the MMActivity dataset demonstrates that STFNet achieves a classification accuracy of 98.65%, outperforming state-of-the-art methods and confirming its effectiveness and superiority.

Existing action quality assessment (AQA) methods mainly learn deep representations at the video level for scoring diverse actions. Due to the lack of a fine-grained understanding of actions in videos, they harshly suffer from low credibility and interpretability, thus insufficient for stringent applications, such as Olympic diving events. We argue that a fine-grained understanding of actions requires the model to perceive and parse actions in both time and space, which is also the key to the credibility and inter-pretability of the AQA technique. Based on this insight, we propose a new fine-grained spatial-temporal action parser named FineParser. It learns human-centric foreground action representations by focusing on target action regions within each frame and exploiting their fine-grained alignments in time and space to minimize the impact of in-valid backgrounds during the assessment. In addition, we construct fine-grained annotations of human-centric fore-ground action masks for the FineDiving dataset, called FineDiving-HM. With refined annotations on diverse target action procedures, FineDiving-HM can promote the development of real-world AQA systems. Through extensive experiments, we demonstrate the effectiveness of FineParser, which outperforms state-of-the-art methods while supporting more tasks of fine-grained action understanding. Data and code are available at https://github.com/PKU-ICST-MIPL/FineParser_CVPR2024.

Non-contact vital signs monitoring with frequency modulated continuous wave (FMCW) MIMO radar has attracted increasing attention. However, in practical applications, existing methods cannot accurately and robustly measure vital signs due to the influence of indoor multipath propagation. Therefore, this paper proposes a joint design method of waveform parameter optimization and transmit-receive weighting based on FMCW-MIMO radar to suppress multipath interference. This method uses quasi-orthogonal FMCW with low range sidelobe level, which can separate undesired multipath from the desired signal and suppress it by adaptive transmit-receive weighting. Theoretical and simulation results show that the method has good performance in indoor multipath environments with mean errors of 0.608 (bpm) for respiration rate and 3.164 (bpm) for heart rate.

Open-vocabulary detection (OVD) aims to detect novel objects without instance-level annotations to achieve open-world object detection at a lower cost. Existing OVD methods mainly rely on the powerful open-vocabulary image-text alignment capability of Vision-Language Pretrained Models (VLM) such as CLIP. However, CLIP is trained on image-text pairs and lacks the perceptual ability for local regions within an image, resulting in the gap between image and region representations. Directly using CLIP for OVD causes inaccurate region classification. We find the image-region gap is primarily caused by the deformation of region feature maps during region of interest (RoI) extraction. To mitigate the inaccurate region classification in OVD, we propose a new Shape-Invariant Adapter named SIA-OVD to bridge the image-region gap in the OVD task. SIA-OVD learns a set of feature adapters for regions with different shapes and designs a new adapter allocation mechanism to select the optimal adapter for each region. The adapted region representations can align better with text representations learned by CLIP. Extensive experiments demonstrate that SIA-OVD effectively improves the classification accuracy for regions by addressing the gap between images and regions caused by shape deformation. SIA-OVD achieves substantial improvements over representative methods on the COCO-OVD benchmark. The code is available at https://github.com/PKU-ICST-MIPL/SIA-OVD_ACMMM2024.

Automatic Modulation Classification (AMC) is a crucial task in the field of wireless communication, allowing for the identification of the modulation scheme of a received radio signal without prior knowledge of the communication system. Recently, AMC approaches based on Deep Learning (DL) have achieved outstanding results. However, the majority of current DL-based AMC methods face challenges in achieving high recognition accuracy while remaining computationally efficient. Some researchers have designed autoencoder-based models to generate low-dimensional temporal feature embeddings of the radio signal, thereby reducing the number of model parameters while maintaining high performance in recognizing modulation formats. However, when further improving AMC performance via learning low-dimensional spatial-temporal feature representations, traditional autoencoder models require both a convolutional decoder and an LSTM decoder to reconstruct temporal and spatial features separately, which unavoidably raises the model parameters. In this paper, we propose a spatiotemporal feature sharing reconstructing network (STARNet) to simultaneously extract low-dimensional spatial and temporal feature representations of radio signals using a single autoencoder structure, thereby reducing the number of model parameters and improving AMC performance. Additionally, we construct a Hybrid Attentive Ghost (HA-Ghost) to automatically extract discriminative radio signal spatial information according to signal reconstruction performance. Extensive experiments on benchmark datasets demonstrate that the proposed STARNet achieves an average modulation classification accuracy of 63.64%, outperforming previous state-of-the-art models. Despite extracting more types of features, STARNet has only 14,860 parameters, which is smaller than existing spatiotemporal autoencoder-based methods.