We present a real-time multimodal sensing and feedback system that enables closed-loop interaction using wearable devices. Our implementation leverages OmniBuds—a pair of true wireless stereo (TWS) earbuds equipped with dual 9-axis inertial measurement units (IMUs), optical heart rate sensors, and skin temperature sensors, one set in each ear. These sensors continuously stream motion and physiological data via Bluetooth Low Energy (BLE) to a mobile computing platform, where lightweight inference models classify user actions and assess physiological states. Based on this analysis, the system provides real-time auditory feedback through the same earbuds, enabling responsive, human-in-the-loop interaction. As a demonstration, we apply this system to an interactive control scenario based on the T-Rex Chrome Dino game, where users control the avatar using head and body motion captured by the earbuds. Jumping and ducking are recognized in real time through IMU signals, while heart rate and skin temperature dynamically modulates the game speed. The system delivers auditory guidance via the OmniBuds' speakers to help users adapt their actions during game-play. This framework demonstrates the potential of multimodal wearable sensing and closed-loop feedback for embodied interaction, real-time behavioral adaptation, and health-aware interactive systems.

Wearable technology is advancing health monitoring by enabling real-time, privacy-preserving physiological analysis. However, traditional devices often rely on cloud processing, restricting access to raw sensor data and limiting the progress of health-related research. To overcome these limitations, we introduce OmniBuds, a programmable earable research platform that enables multimodal bio-sensing and on-device learning while providing direct access to raw physiological data, fostering advancements in health perception and wearable intelligence. It integrates PPG, temperature, IMUs, and multiple microphones, leveraging an embedded ML accelerator for efficient real-time processing. This paper presents its design, architecture, and applications, demonstrating its potential to shape the future of health-aware wearables.

Backscattering enables battery-less and perpetually operating Internet-of-Things (IoT) devices to sweep through ambient electromagnetic waves (in terrestrial networks) or acoustic waves (in underwater networks) rather than generate new waves. The emerging underwater acoustic backscatter communication (UABC) has the potential to avoid the need for underwater nodes to actively emit acoustic signals, thus, reducing their energy consumption. However, there is a lack of a clear design, comprehensive modelling or analysis of UABC systems. This paper studies a UABC system where a high-power transceiver transmits acoustic signals and a piezoelectric UABC node modulates and reflects the incident acoustic signals back to the transceiver. Specifically, a detailed new design for UABC nodes is proposed and a circuit model for the nodes is developed to analyze their acoustic impedance under different materials and operating frequencies. Based on the circuit model, a node load design is introduced to maximize the acoustic reflection coefficients; and a comprehensive communication analytical framework is established and is used to analyze how different UABC parameters, such as piezoelectric material properties, operating frequencies of UABC nodes, transmission distances, ambient noise intensities, carrier wave frequencies, and wind speeds, affect the bit error rate of UABC. Simulation results validate the effectiveness of the proposed designs and models as well as identify the key parameters influencing the UABC performance.

Reconfigurable intelligent surfaces (RISs) and millimeter-wave (mmWave) communications have been considered for providing wireless connectivity to mobile robots used in industrial plants and other indoor environments. However, the existing works have not sufficiently studied how the number and deployment locations of RISs should be optimized for serving a mobile robot. In this article, we study RIS-assisted mmWave communications for a robot moving around fixed obstacles in an indoor industrial environment. For a fixed total number of reflecting elements, we formulate an optimization problem to minimize the transmission energy consumption of the access point (AP) while ensuring the robot’s received signal-to-noise ratio (SNR) above a threshold throughout its journey by jointly optimizing the number, positions, and phase shifts of RISs and the beamforming vectors of the AP. To solve the formulated nonconvex optimization problem, we devise an iterative algorithm that decomposes it into two subproblems (i.e., optimizing the phase shifts of RISs and the beamforming vector of the AP, and optimizing the number and locations of RISs) and solves them alternately. Simulation results show that the proposed algorithm converges fast and can obtain the best number and locations of RISs that lead to a transmission energy consumption of the AP much lower than the benchmark schemes.

The increasingly stringent requirement on quality-of-experience in 5G/B5G communication systems has led to the emerging neural speech enhancement techniques, which however have been developed in isolation from the existing expert-rule based models of speech pronunciation and distortion, such as the classic Linear Predictive Coding (LPC) speech model because it is difficult to integrate the models with auto-differentiable machine learning frameworks. In this paper, to improve the efficiency of neural speech enhancement, we introduce an LPC-based speech enhancement (LPCSE) architecture, which leverages the strong inductive biases in the LPC speech model in conjunction with the expressive power of neural networks. Differentiable end-to-end learning is achieved in LPCSE via two novel blocks: a block that utilizes the expert rules to reduce the computational overhead when integrating the LPC speech model into neural networks, and a block that ensures the stability of the model and avoids exploding gradients in end-to-end training by mapping the Linear prediction coefficients to the filter poles. The experimental results show that LPCSE successfully restores the formants of the speeches distorted by transmission loss, and outperforms two existing neural speech enhancement methods of comparable neural network sizes in terms of the Perceptual evaluation of speech quality (PESQ) and Short-Time Objective Intelligibility (STOI) on the LJ Speech corpus.

Motion tracking technologies have been widely used in mobile interaction applications, such as Virtual Reality (VR), healthy monitoring, and virtual touch control. Compared with dedicated hardware devices, mobile phones use reliable speakers and microphones, and can serve as ubiquitous devices for cheap acoustic-based motion tracking solutions. However, for complex indoor environments, it is very difficult for acoustic-based methods to achieve accurate motion tracking due to multipath fading and limited sampling rate at mobile devices. In this paper, a new parameter named Multipath Effect Ratio (MER) is defined to indicate the multipath fading effect on received signals at different frequencies. Based on MER, a novel multipath effect mitigating technique is developed to calculate the phase change of acoustic signals and track the corresponding moving distance by using multiple speakers. A Phase-based Acoustic Motion Tracking (PAMT) method is then proposed and implemented on standard Android smartphones. Experiment results show, without any specialized hardware, PAMT can achieve an impressive millimeter-level accuracy for localization and motion tracking applications in multipath fading environments. Specifically, the measurement errors are less than 2mm and 4mm in one-dimensional and two-dimensional scenarios, respectively.

Motion tracking is attractive in what concerns a smart city environment, where citizens have to interact with Internet of Things (IoT) infrastructures spread all around one particular city. Motion tracking is important for smart services and location-based services in smart cities, since it provides natural ways for users to interact with the IoT infrastructures, such as the ability to recognize of a wide range of hand motion in real-time. Compared with dedicated hardware devices, ubiquitous devices with reliable speakers and microphones can be developed to achieve cheap acoustic-based motion tracking, which is appropriate for low-power and low-cost IoT applications. However, for complex urban environments, it is very difficult for acoustic-based methods to achieve accurate motion tracking due to multipath fading and limited sampling rate at mobile devices. In this paper, a new parameter called multipath dispersion vector (MDV) is proposed to estimate and mitigate the impact of multipath fading on received signals using extreme learning machine. Based on MDV, a robust acoustic motion tracking (RAMTEL) method is proposed to calculate the moving distance based on the phase change of acoustic signals, and track the corresponding motion in 2-D plane by using multiple speakers. The method is then proposed and implemented on standard Android smartphones. Experiment results show, without any specialized hardware, RAMTEL can achieve an impressive millimeter-level accuracy for localization and motion tracking applications in multipath fading environments. Specifically, the measurement errors are less than 2 and 4 mm in 1-D and 2-D scenarios, respectively.

Motion tracking, as a mechanism of mobile interaction, allows devices to get fine-gained user input by locating the real-time position of target devices (e.g., smart phones, smart watches) in the air. With the proliferation of mobile devices and smart multimedia devices (e.g., smart TV, home audio system), the ubiquitous speakers and microphones in the devices provide more diverse ways of acoustic-based mobile interaction. In this demonstration, we propose a fine-gained motion tracking system, which can be developed on commercial mobile devices and track the devices with millimeter level (mm-level) accuracy. We first compensate the phase offset between receiver and audio source at each frequency. We then use the acoustic phase change at receiver to achieve accurate distance measurement. Finally, we implement our system on off-the-shelf devices, and achieve a fine-gained motion tracking in two-dimensional space. Our experiments show that our system achieves high accuracy as well as high sensitivity: our system could detect the sight and slow movement caused by human breathing for example.