Fine-grained visual referring and grounding are critical for enhancing scene understanding and enabling various real-world vision-language applications. Although recent studies have extended multimodal large language models (MLLMs) to these tasks, they still face significant challenges in fine-grained multi-target scenarios. To address this, we propose MTRAG, a pixel-level multi-target referring and grounding framework that leverages semantic-spatial collaboration. Specifically, we introduce a Channel Extension Mechanism (CEM) that enables a global image encoder to extract global semantics and multi-region representations while retaining background context, without extra region feature extractors. Moreover, we introduce a grounding branch for pixel-level grounding and design a Hybrid Adapter (HA) to fuse semantic features from the MLLM branch with spatial information from the grounding branch, thereby enhancing the semantic-spatial alignment. For training, we meticulously curate MTRAG-D, a dataset comprising single- and multi-target referring and grounding samples derived from existing datasets and newly synthesized free-form multi-target referring instruction-following data. We also present MTR-Bench, a benchmark for systematic evaluation of multi-target referring. Extensive experiments across five core tasks, including single- and multi-target referring and grounding as well as image-level captioning, show that MTRAG consistently outperforms strong baselines on both multi- and single-target tasks, while maintaining competitive image-level understanding. The code is available at https://github.com/deng-ai-lab/MTRAG.

Recent advances in LLMs have shown exceptional reasoning capability. However, their ability to integrate physical model knowledge for real-world signal interpretation remains largely unexplored. We introduce Wi-Chat, an LLM-powered Wi-Fi-based human activity recognition system. By embedding Wi-Fi sensing principles into prompts, we show that LLMs can infer human activities through Wi-Fi signals in a zero-shot manner without complex signal processing.

In this work, we propose a WiFi vision-based approach to person re-identification (Re-ID) indoors. Our approach leverages the advances of WiFi to visualize a person and utilizes deep learning to help WiFi devices identify and recognize people. Specifically, we leverage multiple antennas on WiFi devices to estimate the two-dimensional angle of arrival (2D AoA) of the WiFi signal reflections to enable WiFi devices to "see'' a person. We then utilize deep learning techniques to extract a 3D mesh representation of a person and extract the body shape and walking patterns for person Re-ID. Our preliminary study shows that our system achieves high overall ranking accuracies. It also works under non-line-of-sight and different person appearance conditions, where the traditional camera vision-based systems do not work well.

Person re-identification (Re-ID) has become increasingly important as it supports a wide range of security applications. In this work, we propose a WiFi-based person Re-ID system in 3D space, which leverages the advances of WiFi and deep learning to extract the static body shape and dynamic walking patterns to recognize people. In particular, we leverage multiple antennas on WiFi devices to capture signal reflections of the human body and produce a WiFi image of a person. We then leverage deep learning to extract both the static body shape and dynamic walking patterns for person Re-ID. Our evaluation results show that our system achieves an overall rank-1 accuracy of 87.1%.

In this work, we present, Wi-Mesh, a WiFi vision-based 3D human mesh construction system. Our system leverages the advances of WiFi to visualize the shape and deformations of the human body for 3D mesh construction. In particular, it estimates the two-dimensional angle of arrival (2D AoA) of the WiFi signal reflections to enable WiFi devices to "see" the physical environment as we humans do. It then extracts only the images of the human body from the physical environment, and leverages deep learning models to digitize the extracted human body into 3D mesh representation. Experimental evaluation under various indoor environments shows that Wi-Mesh achieves an average vertices location error of 2.58cm and joint position error of 2.24cm.

Sensing the liquid level in a container is critical to building many smart home and mobile healthcare applications. This paper presents a liquid level sensing system that is low-cost, high accuracy, widely applicable to different daily liquids and containers, and can be easily integrated with existing smart home networks. Our system uses an existing home WiFi network and a low-cost transducer that is attached to the container to sense the resonance of the container for liquid level detection. We evaluate our system in home environments with various containers and liquids. Preliminary results show that our system achieves an accuracy of 97% for continuous prediction and an F-score of 0.968 for discrete prediction.

The ubiquitous WiFi signals provide us the opportunity to sense human activities and the physical environment. In this work, we take a layered approach to design a vision-based method for commodity WiFi sensing. Specifically, the next-generation WiFi supports a larger number of antennas that can provide spatial information of the signal reflections, which enables a vision-based approach for WiFi sensing. To better leverage the spatial formation of the signal reflections and fulfill emerging applications, we provide a holistic layered framework including hardware, physical, deep learning, and application layers as well as a case study. The proposed layered approach could enlighten the research on future WiFi sensing.

In this paper, we present, Wi-Mesh, a WiFi vision-based 3D human mesh construction system. Our system leverages the advances of WiFi to visualize the shape and deformations of the human body for 3D mesh construction. In particular, it leverages multiple transmitting and receiving antennas on WiFi devices to estimate the two-dimensional angle of arrival (2D AoA) of the WiFi signal reflections to enable WiFi devices to "see" the physical environment as we humans do. It then extracts only the images of the human body from the physical environment, and leverages deep learning models to digitize the extracted human body into a 3D mesh representation. Experimental evaluation under various indoor environments shows that Wi-Mesh achieves an average vertices location error of 2.81cm and joint position error of 2.4cm, which is comparable to the systems that utilize specialized and dedicated hardware. The proposed system has the advantage of reusing the WiFi devices that already exist in the environment for potential mass adoption. It can also work in non-line of sight (NLoS), poor lighting conditions, and baggy clothes, where the camera-based systems do not work well.

The prevalence of WiFi devices and ubiquitous coverage of WiFi networks provide us the opportunity to extend WiFi capabilities beyond communication, particularly in sensing the physical environment. In this article, we survey the evolution of WiFi sensing systems utilizing commodity devices over the past decade. It groups WiFi sensing systems into three main categories: 1) activity recognition (large scale and small scale); 2) object sensing; and 3) localization. We highlight the milestone work in each category and the underline techniques they adopted. Next, this work presents the challenges faced by existing WiFi sensing systems. Finally, we comprehensively discuss the future trending of commodity WiFi sensing.

Earables (ear wearable) are rapidly emerging as a new platform to enable a variety of personal applications. The traditional authentication methods thus become less applicable and inconvenient for earables due to their limited input interface. Earables, however, often feature rich around the head sensing capability that can be leveraged to capture new types of biometrics. In this work, we propose ToothSonic that leverages the toothprint-induced sonic effect produced by a user performing teeth gestures for user authentication. In particular, we design several representative teeth gestures that can produce effective sonic waves carrying the information of the toothprint. To reliably capture the acoustic toothprint, it leverages the occlusion effect of the ear canal and the inward-facing microphone of the earables. It then extracts multi-level acoustic features to represent the intrinsic acoustic toothprint for authentication. The key advantages of ToothSonic are that it is suitable for earables and is resistant to various spoofing attacks as the acoustic toothprint is captured via the private teeth-ear channel of the user that is unknown to others. Our preliminary studies with 20 participants show that ToothSonic achieves 97% accuracy with only three teeth gestures.

WiFi human sensing has become increasingly attractive in enabling emerging human-computer interaction applications. The corresponding technique has gradually evolved from the classification of multiple activity types to more fine-grained tracking of 3D human poses. However, existing WiFi-based 3D human pose tracking is limited to a set of predefined activities. In this work, we present Winect, a 3D human pose tracking system for free-form activity using commodity WiFi devices. Our system tracks free-form activity by estimating a 3D skeleton pose that consists of a set of joints of the human body. In particular, Winect first identifies the moving limbs by leveraging the signals reflected off the human body and separates the entangled signals for each limb. Then, our system tracks each limb and constructs a 3D skeleton of the body by modeling the inherent relationship between the movements of the limb and the corresponding joints. Our evaluation results show that Winect achieves centimeter-level accuracy for free-form activity tracking under various environments.

Biometric-based authentication is gaining increasing attention for wearables and mobile applications. Meanwhile, the growing adoption of sensors in wearables also provides opportunities to capture novel wearable biometrics. In this work, we propose EarDynamic, an ear canal deformation based user authentication using ear wearables (earables). EarDynamic provides continuous and passive user authentication and is transparent to users. It leverages ear canal deformation that combines the unique static geometry and dynamic motions of the ear canal when the user is speaking for authentication. It utilizes an acoustic sensing approach to capture the ear canal deformation with the built-in microphone and speaker of the earables. Specifically, it first emits well-designed inaudible beep signals and records the reflected signals from the ear canal. It then analyzes the reflected signals and extracts fine-grained acoustic features that correspond to the ear canal deformation for user authentication. Our experimental evaluation shows that EarDynamic can achieve a recall of 97.38% and an F1 score of 96.84%.

This paper presents GoPose, a 3D skeleton-based human pose estimation system that uses commodity WiFi devices at home. Our system leverages the WiFi signals reflected off the human body for 3D pose estimation. In contrast to prior systems that need dedicated sensors, our system does not require a user to wear any sensors and can reuse the WiFi devices that already exist in a home environment for mass adoption. To realize such a system, we leverage the 2D AoA estimation of the signals reflected from the human body and the deep learning techniques. Preliminary results show GoPose achieves a high accuracy of 4.5cm in various scenarios.

The voice user interface (VUI) has been progressively used to authenticate users to numerous devices and applications. Such massive adoption of VUIs in IoT environments like individual homes and businesses arises extensive privacy and security concerns. Latest VUIs adopting traditional voice authentication methods are vulnerable to spoofing attacks, where a malicious party spoofs the VUIs with pre-recorded or synthesized voice commands of the genuine user. In this paper, we design VibLive, a continuous liveness detection system for secure VUIs in IoT environments. The underlying principle of VibLive is to catch the dissimilarities between bone-conducted vibrations and air-conducted voices when human speaks for liveness detection. VibLive is a text-independent system that verifies live users and detects spoofing attacks without requiring users to enroll specific passphrases. Moreover, VibLive is practical and transparent as it requires neither additional operations nor extra hardwares, other than a loudspeaker and a microphone that are commonly equipped on VUIs. Our evaluation with 25 participants under different IoT intended experiment settings shows that VibLive is highly effective with over 97% detection accuracy. Results also show that VibLive is robust to various use scenarios.