Lip movement-based user identification holds significant promise for public security and intelligent surveillance due to its dynamic patterns, forgery resistance, and individual distinctiveness. Recently, millimeter-wave radar has been employed for contactless identification, offering advantages such as light insensitivity, privacy preservation, and sensitivity to fine motion. However, its limited wall penetration and vulnerability to occlusion present ongoing challenges. Moreover, existing recognition approaches rely heavily on supervised learning, demanding large labeled datasets and exhibiting poor generalization across domains. To overcome these limitations, we propose Lip-TWCDID, a lip movement-based cross-domain user identification system using 1–2 GHz MIMO radar. The use of low-frequency signals enhances penetration, while the MIMO architecture improves spatial resolution, enabling stable detection of fine-grained micro-Doppler signatures of lip movements through a 22 cm brick wall. To reduce dependence on labeled data and improve domain generalization, we introduce a novel unsupervised domain adaptation (UDA) framework, consistency-adversarial-contrastive learning (CACL), which integrates pseudo-label consistency learning, domain adversarial training, and pseudo-supervised contrastive learning. Specifically, pseudo-label consistency enforces prediction consistency under input perturbations, improving robustness; domain adversarial training introduces a domain discriminator to encourage domaininvariant feature learning and align feature distributions; pseudosupervised contrastive learning leverages high-confidence pseudolabels to perform contrastive learning in the feature space, enhancing inter-class separability and intra-class compactness. By jointly optimizing these components, CACL effectively adapts to unlabeled target domains while minimizing annotation costs. Extensive experiments demonstrate that CACL outperforms state-of-the-art UDA methods and significantly improves the generalization and robustness of through-wall user identification.

Person reidentification (Re-ID) plays an increasingly important role in the development of smart cities and public security systems. Typical person Re-ID is generally used to query and retrieve pedestrians across cameras in the form of images or videos. For the sake of privacy protection and invariability to resolution, light, and occlusion, Re-ID performs excellent prospect by using millimeter-wave radars. Existing radio frequency (RF) person Re-ID approaches either suffer from relatively unreliable accuracy due to the sparse characteristics of point clouds or additionally rely on other sensing task (e.g., 3-D skeleton prediction) to avoid overfitting. In this article, we present mmReID, an RF person Re-ID system which integrates frequency-modulated continuous wave (FMCW) mmWave radar time-velocity micro-Doppler imaging heatmaps with different frequencies into the proposed dual-stream multilayer feature fusion network named ConvSnet. ConvSnet effectively fuses shallow and deep features at different levels, and adopts the attention module with intramodal aggregation to extract the contextual relevance of velocity features. In addition, we construct and publish a dataset mmReIData based on mmWave radar for person Re-ID, consisting of sampling RF data from 41 pedestrians. Experimental results show that our proposed ConvSnet network achieves the best performance against other state-of-the-art networks both in person identification and Re-ID tasks. Further ablation studies indicate the effectiveness of each component of the ConvSnet network.

In current mainstream radar-based personnel identification technologies, the identification process typically relies on detecting the Doppler effect or the intensity of radar reflection signals. However, this method encounters limitations when there are multiple people within the radar detection area, as it cannot precisely locate each person in the real space. Moreover, when these signals are directly input into neural networks for learning, the networks tend to capture macroscopic information, such as body reflections and velocity, while overlooking detailed information crucial for identification, such as body posture and gait. To overcome this challenge, this study proposes a real-time through-wall multiperson identification system based on MIMO radar, named RTMP-ID. This system employs a global-local dual-branch structure to learn fine-grained identity information. The local branch focuses on introducing a radar-based human posture estimation network, aiming to accurately extract sequences of human postures. Based on these sequences, an identity feature extraction model is constructed to derive individual identity information from the postures. Meanwhile, the global branch integrates the posture information obtained from the local branch in a feedback manner, further accurately extracting the target reflection regions for feature extraction. By fusing the features extracted by both branches, the system achieves accurate identification of individuals. Our research results emphasized the critical importance of introducing 3-D pose sequences to enhance the robustness and accuracy of multitarget person identification. We conducted experiments across three scenarios and achieved a maximum recognition rate of 97.4%, even in the presence of stationary individuals.

In recent years, the recognition accuracies of deep learning-based biometric recognition methods, which rely on large amounts of biometric data for training, have significantly increased. However, in practical applications, biometric data are often distributed in small and fragmented amounts among various local clients. Implementing distributed biometric recognition is therefore greatly important. Most existing distributed biometric methods are implemented by federated learning and have achieved great success. However, the conversion from traditional local learning to distributed learning with multiterminal cooperation poses a series of security hazards, such as Byzantine attacks, inference attacks, etc, that have not been addressed. To address the issues, in this paper, a decentralized federated learning links (FedLink) for distributed biometric recognition is proposed, which is resistant to malicious attacks such as Byzantine attacks. Additionally, we validate the performance and security of FedLink by using two biometric traits, fingerprint and finger vein, on the NUPT-FPV dataset. The experimental results demonstrate that the FedLinks has excellent recognition accuracy and performs comparably to the unattacked model when subjected to various degrees of Byzantine attacks.

Abstract Recently, the application of finger vein recognition has become popular. Studies have shown finger vein presentation attacks increasingly threaten these recognition devices. As a result, research on finger vein presentation attack detection (fvPAD) methods has received much attention. However, the current fvPAD methods have two limitations. (1) Most terminal devices cannot train fvPAD models independently due to a lack of data. (2) Several research institutes can train fvPAD models; however, these models perform poorly when applied to terminal devices due to inadequate generalisation. Consequently, it is difficult for threatened terminal devices to obtain an effective fvPAD model. To address this problem, the method of federated finger vein presentation attack detection for various clients is proposed, which is the first study that introduces federated learning (FL) to fvPAD. In the proposed method, the differences in data volume and computing power between clients are considered. Traditional FL clients are expanded into two categories: institutional and terminal clients. For institutional clients, an improved triplet training mode with FL is designed to enhance model generalisation. For terminal clients, their inability is solved to obtain effective fvPAD models. Finally, extensive experiments are conducted on three datasets, which demonstrate the superiority of our method.

Recently emerged radio frequency-based lip-reading recognition technologies leverage their independence from lighting and penetration capabilities to expand the applications of lip-reading. Unlike visual-based lip-reading, this technology penetrates barriers, such as masks, glass, and wood to detect lip movements. However, the previous studies utilizing devices like millimeter-wave radar face limitations due to frequency and power consumption, which restrict the types of penetrable materials and application scenarios. Although low-frequency through-wall radar offers significant penetration capabilities, it has reduced sensitivity to small movements, posing a challenge for lip-reading recognition. Moreover, the high cost of radar equipment and the scarcity of commercially available devices hinder the technology’s development. To address these challenges, we propose TWLip, a word-level lip-reading recognition system utilizing coherent single-input and single-output through-wall radar to detect tiny lip movements behind walls. We utilize I/Q 3-D curves derived from the radar signals corresponding to lip movements as the network input. These curves reflect the amplitude, frequency, and rotational characteristics of lip movements in the complex plane. Furthermore, we designed the IQResNet, built with 1-D preactivation residual bottleneck units, to extract and classify lip-movement features from I/Q 3-D curves. We propose a data-augmentation method for radar lip-reading to enhance model efficacy and generalizability. We created a through-wall radar lip-reading data set containing 20 words from eight volunteers, totaling 9583 samples. The TWLip demonstrated the ability to recognize these words through a 24 cm brick wall from two meters away with 88.51% accuracy, validating the algorithm’s superiority through detailed comparative studies.

By using directional sensor nodes such as cameras, wireless multimedia sensor networks are typically used in traffic monitoring, target tracking and other fields. However, most sensor nodes powered by batteries are strictly limited in energy and cannot achieve long-term frontal tracking and monitoring of moving objects. To solve these problems, a model of directed sensor nodes with solar energy harvesting is introduced in this paper, where the energy of such nodes is no longer limited to batteries and the directed sensing area enables better frontal tracking. On this basis, we propose a distributed algorithm for directional task assignment called EN-DADA. EN-DADA is a task assignment algorithm with energy harvesting and orientation awareness, including a task classification phase and node bidding phase. The task is first classified to determine the candidate node set that can execute the task, and then the task assignment is determined according to the monitoring income of each node in the candidate node set. Experimental results show that the proposed task assignment algorithm has advantages in terms of task revenue and network lifetime when using the same energy harvesting model.

Abstract Today, a directional wireless multimedia sensor network is a popular environment for solving the task assignment problem. Achieving long‐term frontal monitoring of moving objects is a crucial challenge for scholars in this field. Utilizing directional sensors equipped with energy harvesting is an effective technique to enhance network performance. In this way, the energy of nodes is no longer limited to batteries and can result in better frontal monitoring. In this method, each sensor categorizes tasks based on its own energy, allowing the determination of task execution nodes through bidding. The present study proposes a distributed algorithm for directional task assignment, EN‐DADA, based on energy harvesting. The task was first classified to determine the candidate node set that could execute the task, and then the task assignment was determined according to the monitoring income of each node in the candidate node set. The comparative analysis confirmed that the proposed method had advantages in terms of task revenue and network lifetime when using the same energy harvesting model.

Human localization and behavior recognition (HLBR) is an important research topic in wireless sensing and computer vision. Most studies mainly use cameras and millimeter-wave radars, which can achieve fine-grained human localization; however, they cannot solve the problem of wall occlusion. Although methods that use Wi-Fi can penetrate walls, they cannot detect the precise localization of people due to their narrow bandwidth limitation. The above problems limit the application of HLBR in real life. This paper proposes a real-time, through-wall, multi-human localization and behavior recognition system, RTWLBR. In this system, we design a multiple-input multiple-output (MIMO) radar in the 1-2 GHz frequency range. An ultrawide band (UWB) positioning device and a camera are attached to the self-developed, through-wall radar system to capture the simultaneous localization, behavior information, and radar reflection heatmap of people behind a wall, where location information and behavior information are jointly embedded and encoded as a confidence matrix, which serves as a supervision signal for neural network training. A multi-feature fusion network based on a 3D convolutional neural network (3D CNN) is designed to learn human localization and behavior information from radar heatmaps. A joint heatmap compression autoencoder network is utilized to assist network training to reduce quantization error and running costs. The experimental results show that the RTWLBR method can perform real-time localization and behavior recognition of target humans behind a 24 cm brick wall.

Human localization and behavior recognition (HLBR) is an important research topic in wireless sensing and computer vision. In existing work, most of the approaches using sensors such as cameras and mmWave radar cannot solve the wall occlusion problem, while the approaches using Wi-Fi can penetrate the wall but cannot locate human precisely due to its bandwidth limitation. All these limit the application of HLBR in reality. In this paper, we propose TWLBR, a real-time detection system for inferring the localization and behavior of human behind brick walls from radar heatmaps. In this system, we design a multiple-input multiple-output (MIMO) radar in the frequency range of 1–2 GHz and a multi-feature fusion network based on a 3D convolutional neural network (CNN) and a transformer. The network takes four radar heatmaps with background removal as input and outputs the location and behavior of the target humans. Our experiments show that TWLBR can locate and recognize the behavior of target humans behind a 24 cm brick wall with a localization accuracy of 6.2 cm and a behavior recognition accuracy of 96.37%, which is better than existing methods.

Compared with single biometric recognition, multimodal biometric recognition based on fingerprint and finger vein has been widely considered because of its convenient sample collection, high security and accurate recognition. However, according to our investigation, there is no public dataset of fingerprint and finger vein collected at the same time. The existing work uses fingerprint datasets and finger vein datasets from different sources for research, besides the researchers data from building their own equipment, which lacks consideration of practical applications. This is not conducive to the promotion of multibiometric technology based on finger. To promote research on multimodal biometric recognition based on fingerprint and finger vein, we design a finger collection device and introduce a new dataset, NUPT-FPV. It is the first public dataset to collect fingerprint and finger vein simultaneously in real-world applications. NUPT-FPV obtained 840 finger information from 140 volunteers, each finger was collected 20 times (collected in two sessions), and 33600 fingerprint and finger vein images were obtained. In addition, we propose a novel multimodal fusion method based on a convolutional neural network as a benchmark. Extensive experiments were conducted to verify the necessity of our dataset. Through the released dataset and benchmark, we hope to further promote the development of multimodal biometrics based on fingerprint and finger vein.

Abstract To improve the segmentation performance on videos with large object motion or deformation, a novel scheme is proposed which has two branches. In one branch, the attention mechanism is first utilized to highlight objects‐related features. Then, to well consider the temporal coherence of videos, Conv3D is integrated to capture short‐term temporal features, and the designed attention residual convolutional long–short‐term memory is adopted to capture the long–short‐term temporal information of objects under the interference of redundant video frames. Meanwhile, considering the negative effect of background motion, in another branch, the optical flow‐based prediction model is introduced to predict objects regions in subsequent video frames with the annotated initial frame. At last, based on the fused results of two branches, the global thresholds and noising area clean method are employed to obtain segmented objects. The experiments on DAVIS2016 and CDnet2014 exhibit the competitive performance of the proposed scheme.

Wi‐Fi‐based indoor localization has received extensive attention in wireless sensing. However, most Wi‐Fi‐based indoor localization systems have complex models and high localization delays, which limit the universality of these localization methods. To solve these problems, a depthwise separable convolution‐based passive indoor localization system (DSCP) is proposed. DSCP is a lightweight fingerprint‐based localization system that includes an offline training phase and an online localization phase. In the offline training phase, the indoor scenario is first divided into different areas to set training locations for collecting CSI. Then, the amplitude differences of these CSI subcarriers are extracted to construct location fingerprints, thereby training the convolutional neural network (CNN). In the online localization phase, CSI data are first collected at the test locations, and then, the location fingerprint is extracted and finally fed to the trained network to obtain the predicted location. The experimental results show that DSCP has a short training time and a low localization delay. DSCP achieves a high localization accuracy, above 97%, and a small median localization distance error of 0.69 m in typical indoor scenarios.

Wi-Fi-based indoor localization has received extensive attention in the academic community. However, most WiFi-based indoor localization systems have complex models and high localization delays, which limit the universality of these localization methods. To solve these problems, we propose a depthwise separable convolution based passive indoor localization system (DSCP) using Wi-Fi channel state information (CSI). DSCP is a fingerprint-based localization system, which includes an offline training phase and an online localization phase. In the offline training phase, the indoor scenario is first divided into different areas to set training locations for collecting CSI. Then amplitude differences of these CSI subcarriers are extracted for constructing location fingerprints, thereby training the CNN. In the online localization phase, CSI data is first collected at the test locations, then the location fingerprint is extracted and finally fed to the trained network to obtain the predicted location. The experimental results show that DSCP has a short training time and a low localization delay. DSCP achieves a high localization accuracy, upper than 97%, and a small median localization distance error of 0.98 m in open indoor scenarios.

Video segmentation has become a fundamental of various multimedia applications. Spatiotemporal coherence is important for video segmentation. In this study, to balance the spatiotemporal coherence in scenes with deformation or large motion, the authors propose a novel segmentation scheme based on the absorbing Markov chain (AMC) model named directed graph segmentation based on AMC. In their study, they first generate primary proposals per frame. Then, they train weight models by using a part of primary proposals with their features and feature scores. Next, they construct a directed AMC graph, in which states are the generated primary proposals and edge weights are decided by trained weight models. They subsequently perform the first proposal selection per frame by thresholding the modified absorbed time. Afterwards, they design a reselection algorithm to filter the selected proposals and ensure the proposals, which are the most likely to be the motion object in each frame, to be selected as candidates. Finally, they employ the graph‐cuts based optimisation algorithm to generate refined per pixel segmentation by using object and background models built by candidate proposals under the concept of Gaussian mixture models. Experimental results demonstrate that the proposed scheme shows competitive performance compared with advanced algorithms.

A distributed image compression scheme based on solar energy harvesting is proposed to address the problem of image transmission in wireless multimedia sensor networks. Two-level clustering management is adopted. The camera node-normal node cluster enables camera nodes to gather and send collected raw images to the corresponding normal nodes for compression, and the normal node cluster enables the normal nodes to send the compressed images to the corresponding cluster head node. The re-clustering and dynamic adjustment methods for normal nodes are proposed to adaptive adjustment the operation mode in the working chain. Simulation results show that the proposed distributed image compression scheme can effectively balance the energy consumption of the network. Compared with the existing image transmission schemes, the proposed scheme can transmit more and higher quality images and ensure the survival of the network.

In Wyner–Ziv (WZ) video coding, side information (SI), which is a decoder estimation of the original frame, plays a key role in overall compression performance. Many researchers have focused on SI in the past decade to develop efficient SI generation algorithms. In this study, the authors propose an algorithm combined with naive Bayesian theory to create a generic model that can complete the generation of SI in the WZ video coding framework. The proposed scheme first utilises samples to build the initial model, after which the algorithm filters the samples and models according to the threshold . Then, the algorithm takes the filtered samples and models as conditions to build the generic model. Finally, the proposed scheme completes the generation of SI with the motion vectors obtained from the generated model. Experimental results show that the proposed algorithm achieves better rate‐distortion performance and improves peak signal‐to‐noise ratio by up to 0.5 and 2 dB compared with state‐of‐the‐art techniques.

As an advanced form of wireless sensor networks, directional sensor networks (DSNs) are used in a range of areas monitoring and surveillance applications. In this paper, we study sensor scheduling problems based on DSNs for subarea multi-demands of coverage, also called the p-coverage. With regard to different subarea coverage requirements p, the initial area is divided into small subareas and each subarea has its own coverage requirement. Starting from the analysis of sensing model of rotatable and directional sensors, we propose an algorithm to generate sensor's field of view area based on grid division. Then we devise a distributed Direction Adjustable Sensor-scheduling Algorithm (DASA), which consists of selecting step and connecting step to solve the multi-demands of coverage, i.e., the p-coverage problem. The objective of DASA is to prolong the network lifetime with considering the connectivity of network. A set of experiments are performed to investigate the proposed DASA. By comparing with other algorithms, the simulation results show that DASA performs better in subarea lifetime, overall network lifetime and sensor selection efficiency than other algorithms.