Gesture recognition has significant applications in areas such as assisted living, e-health, and human-device interactions. Moving beyond conventional computer vision techniques, recent studies have increasingly adopted ubiquitous methods like Wi-Fi and acoustic signals, which provide a cost-effective solution for device deployment. In this paper, we explore these two ubiquitous techniques to enhance gesture recognition, focusing on overcoming challenges associated with multi-modal fusion. To harmonize information from these inherently different signal types, we propose a tailored fusion strategy specifically designed for Wi-Fi and acoustic signals. Traditional multi-modal fusion methods often lack a theoretical framework due to insufficient analysis of the fundamental characteristics of different signals. To address this gap, we introduce the concept of the Hybrid Zone, a novel theoretical framework that models the interaction and fusion of acoustic and Wi-Fi sensing signals. The Hybrid Zone offers a unified perspective on the interaction between acoustic and Wi-Fi sensing areas and delivers insights into the granular synthesis of their velocity profiles. Our experimental results demonstrate strong performance, achieving a gesture recognition accuracy rate of 94.69%.

Gesture recognition has significant applications such as assisted living, e-health, and human-device interactions. Moving away from conventional computer vision techniques, recent studies are turning towards ubiquitous methods such as Wi-Fi and acoustic signals, which offer device deployment at minimal costs. In this paper, we explore these two ubiquitous techniques to enhance gesture recognition, addressing challenges related to multi-modal fusion. Due to the inherent differences in signal types, we employ a tailored method to harmonize information from these distinct signals. Traditional multi-modal fusion methods often lack theoretical models due to insufficient analysis of the foundational characteristics of different signals. In particular, we propose a concept, Hybrid Zone, which is a theoretical model illustrating the fusion of acoustic and Wi-Fi sensing. Hybrid zone offers a comprehensive perspective on the fusion of acoustic and Wi-Fi sensing areas. Moreover, it provides an intricate view of the synthesis of acoustic and Wi-Fi velocities at a granular level. Our experimental results have been promising, achieving a high accuracy rate of 93.75% in gesture recognition.

This paper studies the problem of 6D pose estimation, which is practically important in various application scenarios such as robotic-based object grasping, obstacle avoidance in autonomous driving scene, and object integration in mixed reality. However, existing methods suffer from at least one of the five major limitations: dependence on object identification, complex deployment, difficulty in data collection, low accuracy, and incomplete estimation. To overcome the above limitations, this paper proposes an RC6D system, which is the first to estimate 6D poses by fusing RFID and Computer Vision (CV) data with multi-modal deep learning techniques. In RC6D, we first detect 2D keypoints through a deep learning approach. We then propose a novel RFID-CV fusion neural network to predict the depth of the scene, and use the estimated depth information to expand the 2D keypoints to 3D keypoints. Finally, we model the coordinate correspondences between the detected 2D-3D keypoints, which is applied to estimate the 6D pose of the target object. When implementing RC6D, we mainly address the following three technical challenges. (i) To predict 6D poses without using the CAD model, we propose a network architecture for monocular depth estimation. (ii) To train the neural network for 6D pose estimation without time-consuming 6D labeling, we use an unsupervised learning algorithm based on 2D-3D point pair matching. (iii) To detect the subject of the object without identification, we leverage optical flow to restrict the object and RFID to directly obtain its information. The experimental results show that the localization error of RC6D is less than 10 cm with a probability higher than 90.64% and its orientation estimation error is less than 10° with a probability higher than 79.63%. Hence, the proposed RC6D system performs much better than the state-of-the-art related solutions.