Indoor self-localization has become a highly desirable system function for smartphones. The existing systems based on imaging, radio frequency, and geomagnetic sensing may have sub-optimal performance when their limiting factors prevail. In this paper, we present a new indoor simultaneous localization and mapping (SLAM) system that is based on the smartphone's built-in audio hardware and inertial measurement unit (IMU). Our system uses a smartphone's loudspeaker to emit near-inaudible chirps and then the microphone to record the acoustic echoes from the indoor environment. The echoes contain the smartphone's location information with sub-meter granularity. To enable SLAM, we apply contrastive learning to train an echoic location feature (ELF) extractor, such that the loop closures on the smartphone's trajectory can be accurately detected from the associated ELF trace. The detection results effectively regulate the IMU-based trajectory reconstruction. The reconstructed trajectories are used fortrajectory map superimpositionandroom geometry reconstruction. Extensive experiments show that our SLAM achieves median localization errors of$\text{0.1}\,\text{m}$,$\text{0.53}\,\text{m}$, and$\text{0.4}\,\text{m}$in a living room, an office, and a shopping mall, and outperforms both the Wi-Fi and geomagnetic SLAM systems. The room geometry reconstruction achieves up to 4× lower errors compared with the latest echo-based approaches.

Recent advances in machine learning have inspired the development of deep neural network (DNN)-based smart sensing applications for the Artificial Internet of Things (AIoT). However, the effectiveness of DNNs relies on the availability of large, labeled data to uncover useful feature representations. The widespread use of DNN models in computer vision (CV), natural language processing (NLP), and voice sensing can be attributed to the massively available labeled training datasets. Despite the abundance of IoT sensing data, the human-uninterpretable property of AIoT data makes it difficult to construct labeled datasets for DNN model training. Additionally, variations in sensor hardware or DNN models’ deployment environments introduce domain shifts, making generalized machine learning algorithms even more difficult to develop. The scarcity of labeled training data and run-time domain shifts are two main challenges in developing effective machine learning algorithms for AIoT sensing. The goal of my research is to address the above challenges for AIoT sensing applications. Two main research methodologies are involved. The first is to leverage the latest state-of-the-art machine learning techniques to develop effective models for smart sensing. The second approach involves integrating known prior knowledge into machine learning algorithms to develop more accurate and reliable DNN models for AIoT sensing applications.

Runtime domain shifts from the training phase caused by sensor characteristic variation incur performance drops of the deep learning-based sensing systems. To address this problem, existing transfer learning techniques require substantial target-domain data and incur high post-deployment overhead. Differently, we propose to exploit the first principle governing the domain shift to reduce the demand for target-domain data. Specifically, our proposed approach called PhyAug uses the first principle fitted with few labeled or unlabeled data pairs collected by the source sensor and the target sensor to transform the existing source-domain training data into the augmented target-domain data for calibrating the deep neural networks. In two audio sensing case studies of keyword spotting and automatic speech recognition, PhyAug recovers the recognition accuracy losses due to microphones’ characteristic variations by 37% to 72% with 5-second unlabeled data collected from the target microphones. In a case study of acoustics-based room recognition, PhyAug recovers the recognition accuracy loss caused by smartphone microphone variation by 33% to 80%. In the last case study of fisheye image recognition, PhyAug reduces the image recognition error due to the camera-induced distortions by 72%.

Indoor self-localization is a highly demanded system function for smartphones. The current solutions based on inertial, radio frequency, and geomagnetic sensing may have degraded performance when their limiting factors take effect. In this paper, we present a new indoor simultaneous localization and mapping (SLAM) system that utilizes the smartphone's built-in audio hardware and inertial measurement unit (IMU). Our system uses a smartphone's loud-speaker to emit near-inaudible chirps and then the microphone to record the acoustic echoes from the indoor environment. Our profiling measurements show that the echoes carry location information with sub-meter granularity. To enable SLAM, we apply contrastive learning to construct an echoic location feature (ELF) extractor, such that the loop closures on the smartphone's trajectory can be accurately detected from the associated ELF trace. The detection results effectively regulate the IMU-based trajectory reconstruction. Extensive experiments show that our ELF-based SLAM achieves median localization errors of 0.1 m, 0.53 m, and 0.4m on the reconstructed trajectories in a living room, an office, and a shopping mall, and outperforms the Wi-Fi and geomagnetic SLAM systems.

Run-time domain shifts from training-phase domains are common in sensing systems designed with deep learning. The shifts can be caused by sensor characteristic variations and/or discrepancies between the design-phase model and the actual model of the sensed physical process. To address these issues, existing transfer learning techniques require substantial target-domain data and thus incur high post-deployment overhead. This paper proposes to exploit the first principle governing the domain shift to reduce the demand on target-domain data. Specifically, our proposed approach called PhyAug uses the first principle fitted with few labeled or unlabeled source/target-domain data pairs to transform the existing source-domain training data into augmented data for updating the deep neural networks. In two case studies of keyword spotting and DeepSpeech2-based automatic speech recognition, with 5-second unlabeled data collected from the target microphones, PhyAug recovers the recognition accuracy losses due to microphone characteristic variations by 37% to 72%. In a case study of seismic source localization with TDoA fingerprints, by exploiting the first principle of signal propagation in uneven media, PhyAug only requires 3% to 8% of labeled TDoA measurements required by the vanilla fingerprinting approach in achieving the same localization accuracy.

Smartphone indoor location awareness is increasingly demanded by a variety of mobile applications. The existing solutions for accurate smartphone indoor localization rely on additional devices or pre-installed infrastructure (e.g., dense WiFi access points, Bluetooth beacons). In this demo, we present EchoLoc, an infrastructure-free smartphone indoor localization system using room acoustic response to a chirp emitted by the phone. EchoLoc consists of a mobile client for echo data collection and a cloud server hosting a deep neural network for location inference. EchoLoc achieves 95% accuracy in recognizing 101 locations in a large public indoor space and a median localization error of 0.5 m in a typical lab area. Demo video is available at https://youtu.be/5si0Cq6LzT4.