Human semantic segmentation facilitates the recognition of different parts of the human body and is essential for applications such as sports analysis and fall detection. To integrate human semantic segmentation into the domain of radio front-end sensing, this article introduces mmSeg, an innovative system that leverages commercial millimeter-wave radar for human semantic segmentation. However, the inherent propagation characteristics of mmWave signals often result in highly sparse point clouds with limited semantic information and the entanglement of temporal-topological features, making human semantic segmentation a challenging task. To address these challenges, mmSeg (i) first introduces a radar cross-section (RCS) calculation method suitable for commercial millimeter-wave radar to enhance the semantic information of radar point clouds at a coarse granularity; (ii) further designs a temporal-topological decoupling network to obtain the fine-grained human semantic segmentation results; (iii) constructs an efficient loss function for end-to-end training, based on an adjacency matrix graph to improve the segmentation performance. We evaluate mmSeg on our self-built millimeter-wave dataset HSS and a public dataset MM-Fi. mmSeg achieves an average point cloud segmentation accuracy of 87.74% on the HSS dataset and 84.18% on the MM-Fi dataset, outperforming the existing methods in both cases.

Human activity recognition (HAR) requires extracting accurate spatial-temporal features with human movements. A mmWave radar point cloud-based HAR system suffers from sparsity and variable-size problems due to the physical features of the mmWave signal. Existing works usually borrow the preprocessing algorithms for the vision-based systems with dense point clouds, which may not be optimal for mmWave radar systems. In this work, we proposed a graph representation with a discrete dynamic graph neural network (DDGNN) to explore the spatial-temporal representation of human movement-related features. Specifically, we designed a star graph to describe the high-dimensional relative relationship between a manually added static center point and the dynamic mmWave radar points in the same and consecutive frames. We then adopted DDGNN to learn the features residing in the star graph with variable sizes. Experimental results demonstrated that our approach outperformed other baseline methods using real-world HAR datasets. Our system achieved an overall classification accuracy of 94.27%, which gets the near-optimal performance with a vision-based skeleton data accuracy of 97.25%. We also conducted an inference test on Raspberry Pi 4 to demonstrate its effectiveness on resource-constraint platforms. We provided a comprehensive ablation study for variable DDGNN structures to validate our model design. Our system also outperformed three recent radar-specific methods without requiring resampling or frame aggregators.

Real-time semantic recognition is crucial for virtual reality (VR) applications, but the efficient fusion of multi-modal data poses significant challenges under resource-constrained VR scenarios. While integrating millimeter-wave (mmWave) radar point clouds with vision data offers a promising solution, existing methods often suffer from excessive data overhead and degraded accuracy due to redundant and noisy information. To address this limitation, this paper presents VR-PCT, a multi-modal transformer for edge-client collaborative VR semantic recognition that fuses mmWave radar point cloud and vision data for VR applications. VR-PCT introduces a novel collaborative design where VR clients perform lightweight semantic region detection while VR edge processes multi-modal VR semantic recognition. Through efficient edge-client collaboration, VR-PCT optimizes the transmission of mmWave point cloud and vision data by transmitting only the VR semantic region of vision data instead of the entire video. Additionally, it incorporates adaptive cross-modal data selection and fusion strategies to achieve real-time semantic recognition while significantly reducing data redundancy. Across 22 participants engaged in four experimental scenes utilizing VR devices from three different manufacturers, our evaluation demonstrates that VR-PCT achieves 97.6% recognition accuracy while reducing transmission overhead by 81.5% compared to existing approaches. These results highlight the effectiveness of VR-PCT in enabling efficient and accurate multi-modal VR semantic recognition for VR applications. The code and data of VR-PCT are released onhttps://github.com/luoyumei1-a/VR-PCT.

As an important branch of IoT applications, Human activity recognition (HAR) is widely used in daily life, particularly through vision-based methods. However, vision-based HAR has serious privacy issues. How to better and low-cost protect the privacy of users who have already installed the relevant devices is a problem that needs to be solved. To address this challenge, we can solve it by transforming video to privacy-preserving mmWave data. Existing studies have primarily focused on synthesizing micro-Doppler data from video, but there is a lack of methods for synthesizing range-Doppler data. Thus, we present a comprehensive method for synthesizing range-Doppler data from videos and subsequently utilize this synthetic data for HAR. Experimentally, we deploy our range-Doppler synthesis method and classification model on a custom dataset. Experimental results indicate that the model trained with synthetic data achieves accuracy on the custom dataset by 95.7%, which is comparable to the accuracy of vision-based HAR works, and demonstrate that the scheme proposed in this paper achieves privacy-preserving HAR.

Millimeter-wave (mmWave) radar is emerging as a promising sensor for various human sensing tasks. Deep learning is frequently applied in radar-based applications, which typically require extensive data collection and labeling. In this demo, we present a low-cost hardware setup and a cross-platform software pipeline that automatically captures radar data of human activities, labels ground truth, and tests inference models in real time. The effectiveness of the testbed is demonstrated through real-time human pose estimation.

This paper presents Mission, the first-of-this-kind cross-modal reidentification (ReID) design for mmWave Radar and RGB cameras. Given a person of interest detected by Radar in camera-restricted scenarios, Mission can identify the image of the person from cameras that are ubiquitously deployed in camera-allowed areas. We envision that cross Vison-RF ReID can significantly enrich mmWave human sensing with a wide spectrum of applications in security surveillance, tracking, and personalized services. Technically, we introduce a novel method for cross-modal similarity estimation that exploits inherent synergies between fine-grained 2D images and coarse-grained 3D Radar point clouds to effectively overcome their modal discrepancy. Through extensive experiments, we demonstrated that our proposed system can achieve 85% top-1 accuracy and 90% top-5 accuracy among 58 volunteers.

Low-Power Wide-Area Networks (LPWANs), extensively utilized for connecting billions of IoT devices, encounter wireless interference challenges in unlicensed frequency bands. Cutting-edge research suggests employing Received Signal Strength Indication (RSSI) sequences for error detection to mitigate interference-related issues. Nevertheless, the effectiveness of this method significantly declines under low signal-to-noise ratios (SNRs). Additionally, long-range communication often results in low SNR received signals, sometimes even below the noise floor. Targeting this fundamental issue, this article proposes the LPWAN packet technique, broadly applicable across diverse scenarios through edge–cloud collaboration. On the edge side, we propose an innovative architecture that fully exploits spatial distribution and interference independence in the field. Rather than utilizing resource-intensive RSSI-based error detection, we leverage a lightweight coding scheme for error detection at the Long Range (LoRa) edge, forwarding correct frames to the cloud. On the cloud side, packet recovery is achieved utilizing group-weighted voting. We design and implement ECRLoRa with commercially available devices (SemTech’s SX1278 and SX1302 LoRa chipsets) and assess its performance in low SNR environments. Our thorough evaluation demonstrates that our approach attains a Packet Recovery Ratio of 96% with low SNR (i.e., below −10 dB), resulting in 1.8× throughput, 7.5× faster recovery time, and 4.92× average accuracy compared to state-of-the-art cloud-optimized application layer solutions.

The successful operation of autonomous vehicles hinges on their ability to accurately identify objects in their vicinity, particularly living targets such as bikers and pedestrians. However, visual interference inherent in real-world environments, such as omnipresent billboards, poses substantial challenges to extant vision-based detection technologies. These visual interference exhibit similar visual attributes to living targets, leading to erroneous identification. We address this problem by harnessing the capabilities of mmWave radar, a vital sensor in autonomous vehicles, in combination with vision technology, thereby contributing a unique solution for liveness target detection. We propose a methodology that extracts features from the mmWave radar signal to achieve end-to-end liveness target detection by integrating the mmWave radar and vision technology. This proposed methodology is implemented and evaluated on the commodity mmWave radar IWR6843ISK-ODS and vision sensor Logitech camera. Our extensive evaluation reveals that the proposed method accomplishes liveness target detection with a mean average precision of 98.1%, surpassing the performance of existing studies.

This paper presents a framework for semantic segmentation on sparse sequential point clouds of millimeter-wave radar. Compared with cameras and lidars, millimeter-wave radars have the advantage of not revealing privacy, having a strong anti-interference ability, and having long detection distance. The sparsity and capturing temporal-topological features of mmWave data is still a problem. However, the issue of capturing the temporal-topological coupling features under the human semantic segmentation task prevents previous advanced segmentation methods (e.g PointNet, PointCNN, Point Transformer) from being well utilized in practical scenarios. To address the challenge caused by the sparsity and temporal-topological feature of the data, we (i) introduce graph structure and topological features to the point cloud, (ii) propose a semantic segmentation framework including a global feature-extracting module and a sequential feature-extracting module. In addition, we design an efficient and more fitting loss function for a better training process and segmentation results based on graph clustering. Experimentally, we deploy representative semantic segmentation algorithms (Transformer, GCNN, etc.) on a custom dataset. Experimental results indicate that our model achieves mean accuracy on the custom dataset by 82.31% and outperforms the state-of-the-art algorithms. Moreover, to validate the model’s robustness, we deploy our model on the well-known S3DIS dataset. On the S3DIS dataset, our model achieves mean accuracy by 92.6%, outperforming baseline algorithms.

Recent researches have mitigated interference by utilizing cloud assistance or cloud-edge collaboration for Low-Power Wide-Area Networks. However, the issue of long interference recovery time prevents these methods from being well utilized in practical scenarios. In this paper, we propose a novel method, called FDR, for Edge-Cloud collaborative interference mitigation with Fuzzy Detection Recovery, which recovers errors in real-time. Our design (i) utilizes gateways and cloud servers and (ii) reduces data transmissions with fuzzy detection codes for real-time error recovery. In our design, each gateway detects and reports the fuzzy positions of errors to the cloud. Then the cloud restores packets with fuzzy detection results. FDR takes the advantage of both the computational ability of the cloud and the error detection benefit of each gateway. We design and implement FDR with commodity devices including LoRa SX1280 and the USRP-B210 platform. Experimental results show that FDR reduces recovery time by 78.53% compared with the state-of-art, and recovers interfered data packets accurately when the packet damage rate reaches 45.72%.

loT devices have evolved from providing remote connection to being an essential component of the Metaverse. The integration of loT and vision technologies has been incubating emerging applications such as vision-enhanced device tracking and remote education/medicine/maintenance. Despite the exciting vision, practical challenges include coordinate transformation, angle estimation, target mapping, and personal error. Instead of proposing yet-another localization approach, we propose a novel vision-enhanced device tracking system, called VisBLE. VisBLE takes advantage of the new localization capability introduced in BLE 5.1 and advances in vision technologies for high accuracy, robust, and intuitive BLE device tracking. There are two novel technical mechanisms: i) a rotation-based wireless localization mechanism that accurately and robustly locates the BLE transmitter in the camera coordinate and ii) a homography-based matching mechanism that identifies target BLE devices with high accuracy on the camera screen. We prototype VisBLE and deploy it on the smartphone (i.e., Nexus 5X) and development board (i.e., CC26X2 + BOOSTXL-AOA). Our results show that VisBLE outperforms the state of the art in both angular accuracy and position accuracy.