Millimeter-wave (mmWave) radar offers significant potential for fine-grained sensing, yet transitioning from controlled laboratory environments to dynamic real-world applications remains challenging. Existing methods face a dichotomy: real-time point clouds sacrifice crucial signal details needed for sophisticated tasks, whereas information-rich raw data sensing imposes prohibitive transmission and computation overheads, often limiting analysis to offline settings and hindering real-time viability. To this end, we present CoSense, areal-timeedge-end collaborative sensing system built on commodity mmWave radar (end) and edge intelligence. We first introduce a novel dual-stream data acquisition mechanism via realizing radar driver-level interfaces, enabling simultaneous transmission of point clouds and raw data. To bridge the fidelity-latency trade-off, we implement an adaptive transmission strategy via firmware modifications, selectively forwarding raw data segments (corresponding to regions of interest identified in the point cloud) for detailed fine-grained analysis, while continuously delivering point clouds for low-latency coarse-grained sensing and control loops. Furthermore, we incorporate closed-loop feedback beamforming, dynamically steering the radar beam based on real-time tracking to counteract motion-induced misalignment and enhance signal fidelity. Extensive evaluations under dynamic conditions demonstrate that CoSense successfully achieves real-time fine-grained sensing with high fidelity and manageable overhead.

Speech enhancement is crucial for reliable communication in noisy environments. However, the lack of a priori knowledge about target speech characteristics in conventional systems often leads to erroneous extraction of interfering speech as desired signals during noise suppression, significantly compromising system performance. Recently, researchers have proposed to use the side-channel signal as an assist to denoise noisy speech. This paper proposes BoneSE, a multimodal speech enhancement approach using bone conduction. The basic idea of BoneSE is to perceive the bone-conducted sound with an IMU sensor embedded in the Commercial Off-The-Shelf (COTS) earphone and then leverage the correlations between the bone conduction signal and audio signal for speech enhancement. However, the lack of high-frequency components in the IMU modality brings data imbalance and hinders data fusion. To address this challenge, we explore and model the relationship between multimodal signals and design a fusion module according to the time and frequency correlation. Moreover, to balance fast processing and denoising performance, we propose bone conduction-based noise level metrics to measure the noise level. To accommodate different noise levels, we propose an adaptive model selection approach based on reinforcement learning to select the proper denoising model, thereby optimizing the latency. Experiments on two datasets show that the proposed method performs favorably against state-of-the-art methods and can enhance speech in low SNR scenarios.

Class-Incremental Learning (CIL) enables learning systems to continuously adapt to evolving data streams. With the advancement of pre-training, leveraging pre-trained vision-language models (e.g., CLIP) offers a promising starting point for CIL. However, CLIP makes decisions by matching visual embeddings to class names, overlooking the rich contextual information conveyed through language. For instance, the concept of ``cat'' can be decomposed into features like tail, fur, and face for recognition. Besides, since the model is continually updated, these detailed features are overwritten in CIL, requiring external knowledge for compensation. In this paper, we introduce ExterNal knowledGe INjEction (ENGINE) for CLIP-based CIL. To enhance knowledge transfer from outside the dataset, we propose a dual-branch injection tuning framework that encodes informative knowledge from both visual and textual modalities. The visual branch is enhanced with data augmentation to enrich the visual features, while the textual branch leverages GPT-4 to rewrite discriminative descriptors. In addition to this on-the-fly knowledge injection, we also implement post-tuning knowledge by re-ranking the prediction results during inference. With the injected knowledge, the model can better capture informative features for downstream tasks as data evolves. Extensive experiments demonstrate the state-of-the-art performance of ENGINE. Code is available at: https://github.com/LAMDA-CL/ICCV25-ENGINE

With the rapid development of smart health care, accurate heart rate variability (HRV) estimation for the early detection of diseases has become a hot research topic. Advanced work uses the wireless signal to estimate the heartbeat in a contact-free way, which usually cannot separate multiple users or work in a dynamic environment. In this article, we propose a lightweight heartbeat-sensing method based on RFID tag pairs, which focuses on HRV extraction in a more general sensing scenario. Based on the tag-pair design, we build a novel heartbeat and respiration model to describe the signal relationship between the two tags from the time and space domains. Based on the model, we propose a Calibrated Temporal-Spatial IQ-Shaping-based signal cancellation algorithm to cancel the respiration and extract the heartbeat. To remove the interference in dynamic measurement, we build an IQ-based signal model via a Principal Component Analysis-based interference estimation. To reduce the statistical error in HRV extraction, we further design a neural network to predict the HRV index. We have implemented a system prototype in a real environment with COTS RFID devices. Extensive experiments show that our system can achieve a median RMSSD error of 7.51 ms, which satisfies the medical demand in HRV measurement.

Class-Incremental Learning (CIL) or continual learning is a desired capability in the real world, which requires a learning system to adapt to new tasks without forgetting former ones. While traditional CIL methods focus on visual information to grasp core features, recent advances in Vision-Language Models (VLM) have shown promising capabilities in learning generalizable representations with the aid of textual information. However, when continually trained with new classes, VLMs often suffer from catastrophic forgetting of former knowledge. Applying VLMs to CIL poses two major challenges: 1) how to adapt the model without forgetting and 2) how to make full use of the multi-modal information. To this end, we propose PROjectiOn Fusion (Proof) that enables VLMs to learn without forgetting. To handle the first challenge, we propose training task-specific projections based on the frozen image/text encoders. When facing new tasks, new projections are expanded, and former projections are fixed, alleviating the forgetting of old concepts. For the second challenge, we propose the fusion module to better utilize the cross-modality information. By jointly adjusting visual and textual features, the model can capture better task-specific semantic information that facilitates recognition. Extensive experiments on nine benchmark datasets with various continual learning scenarios and various VLMs validate that Proof achieves state-of-the-art performance.

With the rise of intelligent systems like assisted driving and robotics, all-weather target identification and 3D localization systems have become crucial for reliable obstacle avoidance and navigation. However, vision-based methods struggle to provide accurate target locations under low light or bad weather. Radar-based solutions like mmWave radar and LiDAR are robust but hindered by high costs and challenges in recognizing target identities at scale. In this paper, we propose alow-cost, all-weather target identification and 3D localization systembased onLED-tags, which system can address the needs of intelligent systems for obstacle avoidance in complex environments. We explore the backscatter communication of LED devices and design adual-modal LED-Tag, which includes two features: a backscatter RF signal detectable by RF devices and visual light spot information detectable by cameras, both sharing the same ID. To enhance the limited backscatter capability, we propose amulti-branch parallel modelthat enhances the signal strength using beamforming synthesis and achannel adjustment mechanismto improve robustness in complex environments, ensuring accurate 3D localization. For multi-target identification, we design an LED-tag encoding system, assigning each tag a unique encoding sequence. Each target's identity can be recognized with our customizedID decoding method, which leverages prior information and time-domain sampling characteristics. Extensive experimental results show that the backscatter communication and target detection range of LED-tags can reach15m. Moreover, the system achieves anaverage localization error of 7.3cm within a 5m range, demonstrating the system's excellent performance in terms of practicality and accuracy.

Nowadays, object detection and 3D tracking have become key technologies for intelligent system or robot navigation to realize automatic obstacle avoidance and target detection, especially in low-light and night vision scenarios. In this paper, we explore the backscattering capability of LEDs and implement a multi-modal tag LED-tag to realize object detection and 3D tracking. Our basic idea is to utilize the fact that feeding modulation signals to an LED-tag can generate both RF and visual features. We fuse the depth of field information perceived from the RF domain and the pixel coordinates obtained from the visual domain to derive a 3D position by matching the decoded ID. In the RF domain, the depth of field is acquired through ultra-wideband channel measurements and estimated phase. In the visual domain, the pixel coordinate in the XOY coordinates can be extract from the image and mapped into 2D spatial coordinates. To address the limited backscatter capability of the LED-tag, we propose a multiple parallel branch model to increase backscatter paths for amplifying the LED-tag's backscattering intensity. Additionally, we propose a decoding ID scheme that utilizes a priori knowledge and repetitive samples to restore the IDs whose encoding frequency is higher than four times the sampling rate. We have implemented a prototype system and evaluated its performance in real-world environments. Extensive experimental results show that LED-tag can backscatter RF signals ranging up to 15m. Besides, the system achieves an average position error of 8cm within the range of 3m.

Ultra-high precision motion sensing leveraging computer vision (CV) is a key technology in many high-precision AR/VR applications such as precise industrial manufacture and image-guided surgery, yet conventional CV can be challenged by moiré-based sensing mechanism, thanks to moiré pattern's high sensitivity to six degrees of freedom (6-DoF) pose changes. Unfortunately, existing moiré-based solutions, in their infancy, cannot deal with complicated curvilinear moiré patterns caused by various perspective angles. In this paper, we propose a generalized moiré-based mechanism, MoiréVision, towards practical adoptions; it relies on high-frequency gratings as visual marker to help extract the fine-grained feature points for ultra-high precision motion sensing. As the foundation of general moiré-based sensing, we propose a formulation to characterize "uncontrolled" curvilinear moiré patterns in practical scenarios. To deal with the problem of moiré feature interference in practice, we propose a Gabor-based algorithm to separate overlapped curvilinear moiré patterns from two dimensions. Furthermore, to extract fine-grained feature points for high-precision motion sensing, we propose a bending function-based model and a resolution-enhanced strategy to reconstruct detailed texture of moiré markers and extract moiré feature points at sub-pixel level. Extensive experimental results show that MoiréVision greatly enhances the usability and generalizability of moiré-based sensing systems in real-world applications.

Detection and assessment of micro vibrations are crucial tasks in both industrial settings and daily life. However, vibration sensors attached to the target vibrator may introduce potential resonance, and wireless detection methods suffer from severe multipath interference. Fortunately, moiré-based sensing methods have gained recognition in recent years due to their ability to perceive micro motion changes. In this paper, we propose MoiréVib, a micro-vibration detection solution based on moiré patterns for dynamic and high-frequency environments. We attach a printed marker with periodic gratings to the surface of vibration devices to generate moiré patterns, which can amplify micro vibrations due to their low-frequency magnification effect. However, moiré pattern's changes caused by micro vibrations are often overwhelmed by random pixel-level noises, and the limited frame rate of the camera fails to capture high-frequency moiré features. To deal with these problems, we propose a spectrum-based method to refine and enhance the dynamic and micro moiré features. Additionally, we propose a dual-frame-rate-based fusion mechanism to realize high-frequency reconstruction of moiré features. Extensive experimental results show that MoiréVib can realize a median amplitude detection error of 4.37 μm and achieve frequency detection up to 300Hz with a frame rate range of 10~30 fps.

In the realm of AR applications and particularly camera-to-screen interactions, camera tracking stands as a crucial technology. However, the ever-increasing demand for tracking accuracy makes it essential to explore a six-degrees of freedom (6-DoF) tracking technology with ultra-high precision to facilitate micro-motion sensing. In this paper, we propose a novel sensing method MoiréTracker to achieve camera’s 6-DoF pose tracking with ultra-high precision. MoiréTracker outputs camera’s continuous 3-DoF trajectory and 3-DoF posture changes according to the captured moiré patterns, which can be produced by the superposition of camera’s Color Filter Array (CFA) and the projection of screen raster on the CFA plane. Thanks to moiré pattern’s high sensitivity to 6-DoF motions, we characterize the relationship between moiré features and camera’s micro pose changes, so as to realize the continuous 6-DoF pose tracking for camera with ultra-high precision. Moreover, our proposal involves a thumbnail-based method aimed at expanding the working range of MoiréTracker, enabling the pervasive camera-to-screen interactions. We implement a prototype system and evaluate its performance in real-world environments. Extensive experiment results show that MoiréTracker achieves the average trajectory error of 1.20 cm and the posture error of 1.07°.

With the rapid development of smart factories, it is necessary to provide lightweight and real-time working condition monitoring for remote facility sensors to reduce labor costs and ensure safe operation. In the high-voltage power grid scenario, it is crucial to detect the working state of the electric knife-edge switch, i.e., the switch's rotation angle in 3D space, which controls the on-off state of the high-voltage lines. However, existing solutions are mainly in a contact way, which is easy to cause accidents in high risk scenarios such as high-voltage transmission lines. In this paper, we propose EdgeMonitor, a contactless monitoring system for knife-edge switches, which can provide robust and real-time working state detection based on lightweight edge devices. EdgeMonitor includes a lightweight edge device, a monocular camera, and an inertial sensor. Specifically, we propose a lightweight two-stage strategy first to extract the switch's feature information, then perform continuous state tracking for the switch's rotation angle in 3D space. Experiment results show that EdgeMonitor achieves the knife-edge switches detection error within 2.5° with a latency of less than 50ms.

With the popularity of online meetings equipped with speakers, voice privacy security has drawn increasing attention because eavesdropping on the speakers can quickly obtain sensitive information. In this paper, we propose mmEavesdropper, a mmWave based eavesdropping system, which focuses on augmenting the micro-vibration signal via theoretical models for voice recovery. Particularly, to augment the receiving signal of the target vibration, we propose to use beam-forming to facilitate the directional augmentation by suppressing other orientations and use Chirp-Z transform to facilitate the distance augmentation by increasing the range resolution compared with traditional FFT. To augment the vibration signal in the IQ plane, we build a theoretical model to analyze the distortion and propose a segmentation-based fitting method to calibrate the vibration signal. To augment the spectrum for sound recovery, we propose to combine multiple channels and leverage an encoder-decoder based neural network to reconstruct the spectrogram for voice recovery. We perform extensive experiments on mmEavesdropper and the results show that mmEavesdropper can reach the accuracy of 93% on digit and letter recognition. Moreover, mmEavesdropper can reconstruct the voice with an average SNR of 5dB and peak SNR of 17dB.

With the popularity of smartwatches, users can access private information stored in the device by simply touching the watch screen. However, smartwatches also expose users to the risk of information leakage because they lack proper authentication schemes. This paper proposes PalmEcho, a multimodal authentication scheme for smartwatches. The basic idea of PalmEcho is to capture the vibration and the sound generated from user’s beating gestures, and then fuse multimodal signals to extract unique features for user authentication. However, signals of beating gestures are short in the time domain, which makes it hard to extract effective features. To address this challenge, our work reveals that the spectral energy distribution of the generated sound is unique to each user and provides rich information for authentication. Moreover, conventional classification networks require large amounts of user data for training, which is not convenient in the user authentication scenario. To address this challenge, we design a prototypical network called BeatNet, which allows users to register with a few samples. Experimental results show that PalmEcho can reach an average F1-score of 94%.

Nowadays, authentication systems are usually required to provide continuous, contactless, and non-intrusive services. In this paper, we proposeRF-Badge, a vital sign-based authentication scheme on human subjects to meet the above requirements by using RFID technology. We consider two biometric features with individual diversity to characterize the vital sign of users, including themovement effectfrom respiration and thereflection effectfrom organs, especially the heart. To derive the movement effect from respiration, we build a phase-based geometric model to restore the fine-grained badge moving trace as the feature. To derive the reflection effect from human internal organs, we extract the reflection signal from the original signal and generate the spectrum as the feature. Besides, to deal with the feature deviation in different physical conditions of users, we propose a multi-condition network (MCNet) to further guarantee the generalization of RF-Badge. We implement a prototype system and evaluate the performance in real environments. The experiment results show that our system achieves the average false positive rate (FPR) of 3.9 percent and false negative rate (FNR) of 3.3 percent for continuous authentication within four signal cycles.

As a key component of most machines, the status of the rotation shaft is a crucial issue in the factories, which affects both the industrial safety and the product quality. Tracking the rotation angle can efficiently monitor the status of the rotation shaft, but traditional solutions either rely on the specialized sensors, suffering from intrusive transformation, or use the computer vision-based solutions, suffering from poor light conditions. In this paper, we present a non-contacting low-cost angle tracking solution, RF-Protractor, to track the rotation shaft based on the surrounding RFID tags. Particularly, instead of directly attaching the tags to the shaft, which may lead to serious miss reading problems due to metal interference, we deploy the tags beside the shaft and leverage the polarization effect of the reflection signal from the shaft for angle tracking. To improve the polarization effect, we exploit the linear polarization feature by using the linear shaft turntable or placing a light aluminum foil on the shaft turntable, which requires no transformation of the shaft. We firstly build a polarization model to quantify the relationship between the rotation angle and the reflection signal. To extract the accurate reflection signal, we then propose to combine the signals of multiple tags to cancel the reflection effect and then estimate the environment-related parameter to calibrate the model. Finally, we propose to leverage both the power trend and the IQ signal to estimate the rotation direction and the rotation angle. We have implemented a real system and the extensive experiments in the real environment confirm the effectiveness of RF-Protractor, which achieves an average error of about 3.1° in angle tracking.

Camera tracking has become a key technology for various application scenarios, especially for AR-based camera-to-screen interaction. Demand for subtle motion detection in camera tracking makes it essential to explore the six degrees of freedom (6-DoF) pose detection with ultra-high precision. In this paper, we propose a novel sensing method MoiréPose to achieve ultra-high precision on the camera's 6-DoF pose estimation. The purpose of MoiréPose is to derive the camera's 3-DoF position and 3-DoF posture relative to the screen according to the captured moiré pattern, which is produced by the superposition of the camera's Color Filter Array (CFA) and the screen raster projected onto the CFA layer. Based on moiré pattern's high sensitivity to 6-DoF pose movement and robustness to the environmental interference in the frequency domain, we propose a spectrogram-based method to realize the camera's 6-DoF detection with ultra-high precision. Moreover, we propose a thumbnail-based method to effectively extend the working range of MoiréPose, so as to realize pervasive camera-to-screen interaction. We have implemented a prototype system and evaluate the performance in real-world environments. Extensive experiment results show that MoiréPose achieves an average position error of 7.5mm and an overall posture error of 1.66°.

Multi-subject video analysis is one of the most important problems in the field of visual perception for human activity recognition on multiple subjects nowadays. However, multi-subject video analysis is difficult to achieve real-time performance at the edge due to the limited resources of edge devices and the high complexity of the Convolutional Neural Networks (CNN) model used in this task. The common processing method is to upload the video data to the cloud. However, due to the influence of network bandwidth, the transmission time is not fixed, and the latency cannot be guaranteed. Thus, statically deployed model configurations cannot meet some dynamically changing scenarios. To address these challenges, in this paper, we propose an edge-cloud collaboration processing system for multi-subject video stream analysis, which can dynamically configure and optimize the related configurations according to specific scenarios. Specifically, we provide an adaptive configuration optimization solution based on context awareness for edge devices with limited resources such that multi-subject video stream analysis can be processed completely at the edge. For other complex scenarios, we propose an edge-cloud collaboration method to achieve task segmentation and collaboration to meet the performance requirements of the complex scenarios. Experimental results show that our method can achieve an average accuracy of 91.3% and the latency of less than 78ms with arbitrary runtime state.

Nowadays, detecting and evaluating the internal structure of packages becomes a crucial task for logistics systems to guarantee reliability and security. However, prior solutions such as X-ray diffraction and WiFi-based detection are not suitable for this purpose. X-ray-based methods usually require manual analysis or image processing algorithms with high complexity, while WiFi-based solutions may fail to detect complex structures due to the significant error of the RF-signal features. In this article, we propose RF-Detector, a low-cost RFID solution for performing three-dimensional (3D) structure detection of items contained in the packages, including the item orientations and relative locations. We thoroughly investigate a brand-new sensing model for RFID-based 3D structure detection, i.e., revolving scanning. We propose not only the fundamental revolving model but also a novel calibration method for the undesired deployments. We have implemented a prototype system to evaluate the performance of RF-Detector. Extensive evaluations in real settings show the effectiveness of RF-Detector, achieving very high accuracy of the internal 3D structure detection.

Nowadays, detecting and evaluating the internal structure of packages becomes a crucial task for logistics systems to guarantee the reliability and security. However, prior solutions such as X-ray diffraction and WiFi-based detection are not suitable for this purpose. X-ray-based methods usually require manual analysis or image processing algorithms with high complexity, while WiFi-based solutions may fail to detect complex structures due to the significant error of the RF-signal features. In this paper, we propose RF-Detector, a low-cost RFID solution for performing 3D structure detection of items contained in the packages, including the item orientations and relative locations. We thoroughly investigate a brand-new sensing model for RFID-based 3D structure detection, i.e., revolving scanning. We propose not only the fundamental revolving model but also a novel calibration method towards the undesired deployment. We have implemented a prototype system to evaluate the performance of RF-Detector. Extensive evaluations in real settings show the effectiveness of RF-Detector, achieving very high accuracy of the internal 3D structure detection.