Lip movement-based user identification holds significant promise for public security and intelligent surveillance due to its dynamic patterns, forgery resistance, and individual distinctiveness. Recently, millimeter-wave radar has been employed for contactless identification, offering advantages such as light insensitivity, privacy preservation, and sensitivity to fine motion. However, its limited wall penetration and vulnerability to occlusion present ongoing challenges. Moreover, existing recognition approaches rely heavily on supervised learning, demanding large labeled datasets and exhibiting poor generalization across domains. To overcome these limitations, we propose Lip-TWCDID, a lip movement-based cross-domain user identification system using 1–2 GHz MIMO radar. The use of low-frequency signals enhances penetration, while the MIMO architecture improves spatial resolution, enabling stable detection of fine-grained micro-Doppler signatures of lip movements through a 22 cm brick wall. To reduce dependence on labeled data and improve domain generalization, we introduce a novel unsupervised domain adaptation (UDA) framework, consistency-adversarial-contrastive learning (CACL), which integrates pseudo-label consistency learning, domain adversarial training, and pseudo-supervised contrastive learning. Specifically, pseudo-label consistency enforces prediction consistency under input perturbations, improving robustness; domain adversarial training introduces a domain discriminator to encourage domaininvariant feature learning and align feature distributions; pseudosupervised contrastive learning leverages high-confidence pseudolabels to perform contrastive learning in the feature space, enhancing inter-class separability and intra-class compactness. By jointly optimizing these components, CACL effectively adapts to unlabeled target domains while minimizing annotation costs. Extensive experiments demonstrate that CACL outperforms state-of-the-art UDA methods and significantly improves the generalization and robustness of through-wall user identification.

Dongsheng Zhu, Weixian Shi, Zhengliang Shi, Zhaochun Ren, Shuaiqiang Wang, Lingyong Yan, Dawei Yin. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.

Contrastive learning has recently achieved compelling performance in unsupervised sentence representation. As an essential element, data augmentation protocols, however, have not been well explored. The pioneering work SimCSE resorting to a simple dropout mechanism (viewed as continuous augmentation) surprisingly dominates discrete augmentations such as cropping, word deletion, and synonym replacement as reported. To understand the underlying rationales, we revisit existing approaches and attempt to hypothesize the desiderata of reasonable data augmentation methods: balance of semantic consistency and expression diversity. We then develop three simple yet effective discrete sentence augmentation schemes: punctuation insertion, modal verbs, and double negation. They act as minimal noises at lexical level to produce diverse forms of sentences. Furthermore, standard negation is capitalized on to generate negative samples for alleviating feature suppression involved in contrastive learning. We experimented extensively with semantic textual similarity on diverse datasets. The results support the superiority of the proposed methods consistently. Our key code is available at https://github.com/Zhudongsheng75/SDA

Dongsheng Zhu, Daniel Tang, Weidong Han, Jinghui Lu, Yukun Zhao, Guoliang Xing, Junfeng Wang, Dawei Yin. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2024.

Recently emerged radio frequency-based lip-reading recognition technologies leverage their independence from lighting and penetration capabilities to expand the applications of lip-reading. Unlike visual-based lip-reading, this technology penetrates barriers, such as masks, glass, and wood to detect lip movements. However, the previous studies utilizing devices like millimeter-wave radar face limitations due to frequency and power consumption, which restrict the types of penetrable materials and application scenarios. Although low-frequency through-wall radar offers significant penetration capabilities, it has reduced sensitivity to small movements, posing a challenge for lip-reading recognition. Moreover, the high cost of radar equipment and the scarcity of commercially available devices hinder the technology’s development. To address these challenges, we propose TWLip, a word-level lip-reading recognition system utilizing coherent single-input and single-output through-wall radar to detect tiny lip movements behind walls. We utilize I/Q 3-D curves derived from the radar signals corresponding to lip movements as the network input. These curves reflect the amplitude, frequency, and rotational characteristics of lip movements in the complex plane. Furthermore, we designed the IQResNet, built with 1-D preactivation residual bottleneck units, to extract and classify lip-movement features from I/Q 3-D curves. We propose a data-augmentation method for radar lip-reading to enhance model efficacy and generalizability. We created a through-wall radar lip-reading data set containing 20 words from eight volunteers, totaling 9583 samples. The TWLip demonstrated the ability to recognize these words through a 24 cm brick wall from two meters away with 88.51% accuracy, validating the algorithm’s superiority through detailed comparative studies.

Current methods for prompt learning in zeroshot scenarios widely rely on a development set with sufficient human-annotated data to select the best-performing prompt template a posteriori.This is not ideal because in a real-world zero-shot scenario of practical relevance, no labelled data is available.Thus, we propose a simple yet effective method for screening reasonable prompt templates in zero-shot text classification: Perplexity Selection (Perplection).We hypothesize that language discrepancy can be used to measure the efficacy of prompt templates, and thereby develop a substantiated perplexity-based scheme allowing for forecasting the performance of prompt templates in advance.Experiments show that our method leads to improved prediction performance in a realistic zero-shot setting, eliminating the need for any labelled examples.PPL Acc.(%) PPL Acc.(%) PPL Acc.(%) PPL Acc.(%) DOUBAN 24.61 57.12 40.93 50.98 28.80 56.68 71.01 51.31 WEIBO 19.78 61.79 30.37 51.16 22.34 58.35 44.45 50.92WAIMAI 16.44 67.80 23.34 53.15 19.68 69.72 36.07 48.49ECOMMERCE 14.07 73.12 18.45 55.68 16.88 67.

Human localization and behavior recognition (HLBR) is an important research topic in wireless sensing and computer vision. Most studies mainly use cameras and millimeter-wave radars, which can achieve fine-grained human localization; however, they cannot solve the problem of wall occlusion. Although methods that use Wi-Fi can penetrate walls, they cannot detect the precise localization of people due to their narrow bandwidth limitation. The above problems limit the application of HLBR in real life. This paper proposes a real-time, through-wall, multi-human localization and behavior recognition system, RTWLBR. In this system, we design a multiple-input multiple-output (MIMO) radar in the 1-2 GHz frequency range. An ultrawide band (UWB) positioning device and a camera are attached to the self-developed, through-wall radar system to capture the simultaneous localization, behavior information, and radar reflection heatmap of people behind a wall, where location information and behavior information are jointly embedded and encoded as a confidence matrix, which serves as a supervision signal for neural network training. A multi-feature fusion network based on a 3D convolutional neural network (3D CNN) is designed to learn human localization and behavior information from radar heatmaps. A joint heatmap compression autoencoder network is utilized to assist network training to reduce quantization error and running costs. The experimental results show that the RTWLBR method can perform real-time localization and behavior recognition of target humans behind a 24 cm brick wall.

Human localization and behavior recognition (HLBR) is an important research topic in wireless sensing and computer vision. In existing work, most of the approaches using sensors such as cameras and mmWave radar cannot solve the wall occlusion problem, while the approaches using Wi-Fi can penetrate the wall but cannot locate human precisely due to its bandwidth limitation. All these limit the application of HLBR in reality. In this paper, we propose TWLBR, a real-time detection system for inferring the localization and behavior of human behind brick walls from radar heatmaps. In this system, we design a multiple-input multiple-output (MIMO) radar in the frequency range of 1–2 GHz and a multi-feature fusion network based on a 3D convolutional neural network (CNN) and a transformer. The network takes four radar heatmaps with background removal as input and outputs the location and behavior of the target humans. Our experiments show that TWLBR can locate and recognize the behavior of target humans behind a 24 cm brick wall with a localization accuracy of 6.2 cm and a behavior recognition accuracy of 96.37%, which is better than existing methods.