Human avatars, defined as animatable 3D digital people, have diverse applications, including teleconferencing, motion guidance, remote care, and gaming. With the widespread adoption of smart mobile devices, the demand for real-time viewing of dynamic digital humans on mobile platforms has grown significantly. However, existing avatar reconstruction methods, such as MagicStream and ExAvatar, struggle to produce high-fidelity avatars from monocular videos captured by mobile devices. Furthermore, most related research focuses on individual algorithmic components and does not provide an end-to-end pipeline that jointly addresses avatar reconstruction, 4D data transmission, and mobile rendering. In this work, we introduce HoloMobile, the first end-to-end mobile avatar system with two key capabilities: (1) High-fidelity avatar reconstruction from monocular videos captured on mobile phones, using a hybrid explicit-implicit model. (2) Efficient polynomial compression, transmission, and web-based rendering of dynamic 3D avatars on mobile devices. With avatar reconstruction performed on a PC or cloud with a single consumer-grade GPU, HoloMobile provides an end-to-end solution for mobile users: from data capture to streaming and real-time viewing. Evaluation on 20 self-captured avatars reveals that HoloMobile achieves an average PSNR of 30.36 dB, a compression rate of 99.57%, and over 200 FPS for mobile rendering, demonstrating superiority over existing baselines. The demo video of HoloMobile is available at https://youtu.be/kfOFV7AiP5A.

Multimodal human action recognition (HAR) utilizes complementary data for activity classification. Built on traditional HAR tasks, recent advances in Large Language Models (LLMs) enable detailed descriptions and causal reasoning of human actions, advancing new tasks of human action understanding (HAU) and human action reasoning (HARn). However, most LLMs, especially multimodal Large Vision-Language Models (LVLMs), struggle with modalities other than RGB images, like depth, IMU, ormmWave, due to a lack of large-scale datasets in these task domains. Existing HAR datasets provide only coarse-grained annotations, in-sufficient for depicting the detailed action dynamics required in HAU and HARn tasks. Simply combining annotations and generating captions with LLMs often lacks necessary logical and spatiotemporal consistency. In this paper, we introduce CUHK-X, a large-scale multi-modal dataset and benchmarks for HAR, HAU, and HARn. It includes 64,267 samples of 40 actions performed by 30 participants across two indoor environments, covering diverse daily scenarios. To address the challenge of spatiotemporal inconsistencies in captions, we propose a prompt-based scene creation method that leverages LLMs to generate logically connected activity sequences. CUHK-X also includes three benchmarks with six tasks to evaluate state-of-the-art models. Experimental results show average accuracies of 76.52% for HAR, 40.76% for HAU, and 70.25% for HARn. This large-scale multimodal dataset aims to empower the research community to apply, develop, and adapt data-intensive learning techniques for a wide range of human activity-related tasks.

In recent years, Deep Neural Networks (DNNs) have been increasingly adopted in a wide range of time-critical applications running on edge platforms equipped with heterogeneous multiprocessors. Given the limited resources available on these platforms, efficiently utilizing both CPU and GPU resources for time-sensitive DNN inference is crucial. However, this cross-processor inference paradigm poses significant challenges due to inherent performance imbalances between different processors. In this paper, we introduce BlastNet, a system that leverages duo-blocks—a novel model inference abstraction designed to enable highly efficient cross-processor, time-sensitive DNN inference. Each duo-block features a dual model structure, facilitating fine-grained, alternate inference across different processors. Duo-blocks are optimized during design and dynamically scheduled at runtime to maximize the resource utilization of CPU and GPU. To address memory constraints on edge devices, we also propose a duo-block selection algorithm that selectively constructs duo-blocks based on performance gains. BlastNet is implemented on an indoor autonomous driving platform and three popular edge platforms. Extensive evaluations demonstrate that BlastNet reduces the deadline missing rate by$35.07\,\%$with only a mere$1.63 \%$loss in model accuracy.

Artificial intelligence (AI) models have been widely deployed on edge devices, enabling various IoT applications. However, lightweight on-device AI models on resource-limited edge devices hinder their adaptability to dynamic environments and tasks. Despite the superior generalization capabilities of recently developed Foundation Models (FMs), utilizing their extensive knowledge on the resource-constrained edge platforms remains unexplored. In this work, we introduce DeepEdgeFM, an edge-cloud collaborative system with FMs that enables open-set learning, simultaneously achieving generalizability and efficiency for IoT applications. DeepEdgeFM employs a spatiotemporalaware semantic customization approach that leverages spatial, temporal, and domain-specific knowledge from FMs to continuously customize edge models using unlabeled sensor data in emerging IoT environments. Meanwhile, DeepEdgeFM utilizes a dynamic model switching strategy to selectively query the knowledge of FMs based on sensor-data uncertainty and real-time network fluctuations. We implement DeepEdgeFM on five FMs and multi-modal large language models (MLLMs), covering four types of sensor data modalities. We evaluate DeepEdgeFM on two edge platforms, five public datasets, and two self-collected datasets covering both indoor and outdoor real-world environments. The results show that DeepEdgeFM outperforms state-ofthe- art baselines, achieving up to an 18.6% accuracy gain and a 38.6

Human activity monitoring in the water is essential for pool management and drowning prevention. Existing camera-based solutions pose significant concerns about privacy and extra installation costs. Although sonars have been widely used for underwater sensing in open aquatic environments such as oceans and lakes, monitoring human activities with sonars in a pool setup is challenging. In this work, we propose AquaScan, the first scanning sonar-based underwater sensing system for human activity monitoring. To overcome the low frame rate, we propose a novel scanning strategy and apply an image reconstruction method to accelerate the scanning speed without compromising the performance of motion detection. We develop a novel signal processing pipeline based on a physical model to remove noises and localize human subjects. We extract features like motion, time, and spatial information from sonar images and develop a state-transfer-based activity recognition system to recognize five common water activities. We deployed AquaScan on three public swimming pools for a total period of 94 hours. The evaluation results show that AquaScan can successfully recognize the five activities in the water at about 91.5%.

Recent advances in Large Language Models (LLMs) have propelled intelligent agents from reactive responses to proactive support. While promising, existing proactive agents either rely exclusively on observations from enclosed environments (e.g., desktop UIs) with direct LLM inference or employ rule-based proactive notifications, leading to suboptimal user intent understanding and limited functionality for proactive service. In this paper, we introduce ContextAgent, the first context-aware proactive agent that incorporates extensive sensory contexts surrounding humans to enhance the proactivity of LLM agents. ContextAgent first extracts multi-dimensional contexts from massive sensory perceptions on wearables (e.g., video and audio) to understand user intentions. ContextAgent then leverages the sensory contexts and personas from historical data to predict the necessity for proactive services. When proactive assistance is needed, ContextAgent further automatically calls the necessary tools to assist users unobtrusively. To evaluate this new task, we curate ContextAgentBench, the first benchmark for evaluating context-aware proactive LLM agents, covering 1,000 samples across nine daily scenarios and twenty tools. Experiments on ContextAgentBench show that ContextAgent outperforms baselines by achieving up to 8.5% and 6.0% higher accuracy in proactive predictions and tool calling, respectively. We hope our research can inspire the development of more advanced, human-centric, proactive AI assistants. The code and dataset are publicly available at https://github.com/openaiotlab/ContextAgent.

An increasing number of environments, such as smart homes and factories, are being equipped with multiple sensor systems to enable diverse intelligent applications. However, most existing sensor coordination systems require manually predefined rules, limiting their ability to handle flexible and complex tasks. While recent approaches leverage large language models (LLMs) to interact with external APIs, they struggle to fully understand the capabilities and data dependencies of practical sensor systems. This paper introduces TaskSense, a novel system that coordinates multiple sensor systems in response to users' complex queries. TaskSense introduces a sensor language that automatically translates the capabilities and data dependencies of sensor systems into vocabularies and grammar rules that can be understood by LLMs. It then interprets user intentions into executable task plans for sensor systems using this sensor language in combination with LLMs. Meanwhile, TaskSense checks the solvability of user queries and verifies the correctness of task plan dependencies. To further enhance robustness, TaskSense incorporates a dynamic plan execution mechanism that adjusts plans based on real-time feedback from sensor data availability, data quality and execution results. TaskSense is deployed on real-world smart home systems, utilizing six popular LLMs. The system is evaluated across 4 scenarios involving 9 types of sensor systems, over 60 APIs, 170 tasks and 5 types of data modalities. Results show that TaskSense achieves up to 2× higher planning accuracy and a 75% increase in answer accuracy using the similar amount of tokens compared with baseline approaches.

Most of the existing OFDM backscatter systems adopt phase-modulated schemes to embed tag data, suffering from symbol-level modulation limitation, heavy synchronization accuracy reliance, and small tolerability to symbol time offset (STO) / carrier frequency (CFO) offset. We introduce SubScatter, the first subcarrier-level frequency-modulated OFDM backscatter which is able to tolerate bigger synchronization errors, STO, and CFO. The unique feature of SubScatter is our subcarrier shift keying (SSK) modulation. This method pushes the modulation granularity to the subcarrier by encoding and mapping tag data into different subcarrier patterns. We also design a tandem frequency shift (TFS) scheme that enables SSK with low cost and low power. Furthermore, we design SubScatter+ that shows these advantages while providing an even higher throughput without requiring more subcarrier patterns. We prototype and test SubScatter and SubScatter+, and the results show that our systems outperforms prior works in terms of effectiveness and robustness. Specifically, SubScatter has 743 kbps throughput that is 3.1 times and 14.9 times higher than RapidRider and MOXcatter, respectively. It also has a lower BER under noise and interferences which is over 6 times better than RapidRider or MOXcatter. Moreover, our proposed SubScatter+ could increase the throughput of SubScatter by 30%.

The demand for fully on-device health monitoring is huge and urgent. However, deploying Large Foundation Models conventionally relies on cloud-based computing services, which poses privacy concerns. Driven by the belief of delivering personalised healthcare to family members, this study presents the development of an innovative on-device machine learning system, CaringFM. This family caring system utilizes privacy-protecting sensors and an edge-deployed Foundation Model(FM) to offer a convenient and low-cost solution for chronic disease prediction and health condition monitoring at home. In particular, CaringFM provides general health suggestions and personalized medical information while ensuring high privacy by processing and preserving all data locally.

Large Language Models (LLMs) have shown significant potential in performing inferences on various tasks using heterogeneous sensors with minimal human intervention. Despite their promise, challenges such as high inference overhead and limitations on resource-constrained edge devices remain. Additionally, model hallucinations, particularly those arising from cognitive biases when interpreting numerical data, hinder performance. This work introduces a novel technique, embedding interpolation, to enhance LLMs' understanding of sensor measurements and mitigate inference overhead on edge devices. By computing embeddings through pre-computed boundary embeddings instead of directly from the input, we improve efficiency and accuracy. The effective-ness of this approach is demonstrated through visualizations with image generation models.

Alzheimer's Disease (AD) and related dementia are a growing global health challenge due to the aging population. In this paper, we present ADMarker, the first end-to-end system that integrates multi-modal sensors and new federated learning algorithms for detecting multidimensional AD digital biomarkers in natural living environments. ADMarker features a novel three-stage multi-modal federated learning architecture that can accurately detect digital biomarkers in a privacy-preserving manner. Our approach collectively addresses several major real-world challenges, such as limited data labels, data heterogeneity, and limited computing resources. We built a compact multi-modality hardware system and deployed it in a four-week clinical trial involving 91 elderly participants. The results indicate that ADMarker can accurately detect a comprehensive set of digital biomarkers with up to 93.8% accuracy and identify early AD with an average of 88.9% accuracy. ADMarker offers a new platform that can allow AD clinicians to characterize and track the complex correlation between multidimensional interpretable digital biomarkers, demographic factors of patients, and AD diagnosis in a longitudinal manner.

Recently, smart roadside infrastructure (SRI) has demonstrated the potential of achieving fully autonomous driving systems. To explore the potential of infrastructure-assisted autonomous driving, this paper presents the design and deployment of Soar, the first end-to-end SRI system specifically designed to support autonomous driving systems. Soar consists of both software and hardware components carefully designed to overcome various system and physical challenges. Soar can leverage the existing operational infrastructure like street lampposts for a lower barrier of adoption. Soar adopts a new communication architecture that comprises a bi-directional multi-hop I2I network and a downlink I2V broadcast service, which are designed based on off-the-shelf 802.11ac interfaces in an integrated manner. Soar also features a hierarchical DL task management framework to achieve desirable load balancing among nodes and enable them to collaborate efficiently to run multiple data-intensive autonomous driving applications. We deployed a total of 18 Soar nodes on existing lampposts on campus, which have been operational for over two years. Our real-world evaluation shows that Soar can support a diverse set of autonomous driving applications and achieve desirable real-time performance and high communication reliability. Our findings and experiences in this work offer key insights into the development and deployment of next-generation smart roadside infrastructure and autonomous driving systems.

Drowning poses a significant threat to human life, making underwater human activity monitoring necessary for pool management. However, it is challenging for existing methods to recognize the aquatic activities of diverse users continuously without privacy concerns. In this paper, we propose a novel sonar-based pool monitoring system that localizes swimmers and recognizes pool activities. Our system is deployed and evaluated in a real-world pool, outperforming baselines in terms of activity recognition.

Despite the extensive use of sensors enabling intelligent applications, the complementary potential of co-existing sensor systems is often not fully utilized, limiting more advanced applications. This paper introduces a novel solution using Large Language Models (LLMs) to coordinate sensor systems for handling complex user queries. It defines a sensor language for sensor systems, including vocabulary set and grammar rules, analogous to natural language components, enabling LLMs to translate user intentions into sensor coordination plans. Preliminary results show that our approach significantly outperforms the existing solution at plan generation, execution and response generation stages.

Federated learning in multi-modal systems faces challenges due to modality heterogeneity, where edge devices have different sensor setups. Labeling multi-modal data is labor-intensive and impractical, leading to the label scarcity issue on edge clients. This paper presents a novel semi-supervised federated learning framework to address these issues. It uses complementary data, like RGB images and depth sensors, with a pseudo-labeling algorithm to improve cross-modal learning. Applied to the human action recognition task, the framework outperforms baselines. It enables efficient federated learning, handling labeling difficulties and missing modalities, offering robust performance in real-world scenarios.

Indoor self-localization has become a highly desirable system function for smartphones. The existing systems based on imaging, radio frequency, and geomagnetic sensing may have sub-optimal performance when their limiting factors prevail. In this paper, we present a new indoor simultaneous localization and mapping (SLAM) system that is based on the smartphone's built-in audio hardware and inertial measurement unit (IMU). Our system uses a smartphone's loudspeaker to emit near-inaudible chirps and then the microphone to record the acoustic echoes from the indoor environment. The echoes contain the smartphone's location information with sub-meter granularity. To enable SLAM, we apply contrastive learning to train an echoic location feature (ELF) extractor, such that the loop closures on the smartphone's trajectory can be accurately detected from the associated ELF trace. The detection results effectively regulate the IMU-based trajectory reconstruction. The reconstructed trajectories are used fortrajectory map superimpositionandroom geometry reconstruction. Extensive experiments show that our SLAM achieves median localization errors of$\text{0.1}\,\text{m}$,$\text{0.53}\,\text{m}$, and$\text{0.4}\,\text{m}$in a living room, an office, and a shopping mall, and outperforms both the Wi-Fi and geomagnetic SLAM systems. The room geometry reconstruction achieves up to 4× lower errors compared with the latest echo-based approaches.

Recent years have witnessed the emergence of a new class of cooperative edge systems in which a large number of edge nodes can collaborate through local peer-to-peer connectivity. In this paper, we propose CoEdge, a novel cooperative edge system that can support concurrent data/compute-intensive deep learning (DL) models for distributed real-time applications such as city-scale traffic monitoring and autonomous driving. First, CoEdge includes a hierarchical DL task scheduling framework that dispatches DL tasks to edge nodes based on their computational profiles, communication overhead, and real-time requirements. Second, CoEdge can dramatically increase the execution efficiency of DL models by batching sensor data and aggregating the inferences of the same model. Finally, we propose a new edge containerization approach that enables an edge node to execute concurrent DL tasks by partitioning the CPU and GPU workloads into different containers. We extensively evaluate CoEdge on a self-deployed smart lamppost testbed on a university campus. Our results show that CoEdge can achieve up to reduction on deadline missing rate compared to baselines.

HD map is a key enabling technology towards fully autonomous driving. We propose VI-Map, the first system that leverages roadside infrastructure to enhance real-time HD mapping for autonomous driving. The core concept of VI-Map is to exploit the unique cumulative observations made by roadside infrastructure to build and maintain an accurate and current HD map. This HD map is then fused with on-vehicle HD maps in real time, resulting in a more comprehensive and up-to-date HD map. By extracting concise bird-eye-view features from infrastructure observations and utilizing vectorized map representations, VI-Map incurs low compute and communication overhead. We conducted end-to-end evaluations of VI-Map on a real-world testbed and a simulator. Experiment results show that VI-Map can construct decentimeter-level (up to 0.3 m) HD maps and achieve real-time (up to a delay of 42 ms) map fusion between driving vehicles and roadside infrastructure. This represents a significant improvement of 2.8× and 3× in map accuracy and coverage compared to the state-of-the-art online HD mapping approaches. A video demo of VI-Map on our real-world testbed is available at https://youtu.be/p2RO65R5Ezg.

In recent years, the deployment of unmanned vehicle delivery services has increased unprecedentedly, leading to a need for enhanced security due to the risk of leaving high-value packages to an unauthorized third party during pickup or delivery. Existing authentication methods such as QR code and one-time password are inadequate, as they are susceptible to attacks and provide only one-way authentication. This paper, for the first time to our best knowledge, proposes Wave-for-Safe (W4S) --- a novel mutual authentication system that utilizes multi-modal sensors on both the user's smartphone and the unmanned vehicle. W4S uses random hand-waving by the legitimate user to achieve robust authentication by obtaining highly correlated sensory data measured by the Inertial Measurement Unit (IMU) in the smartphone and sensors in the unmanned vehicle (e.g., mmWave radar and camera). We propose several novel methods to overcome challenges such as heterogeneous data processing, asynchronization, and imitating attacks. The prototype is implemented on an unmanned vehicle and various smartphones, and evaluation in different real-world scenarios shows that W4S achieves an equal error rate below 0.013 against various attacks.

Head-mounted wearables are rapidly growing in popularity. However, a gap exists in providing robust voice-related applications like conversation or command control in complex environments, such as competing speakers and strong noises. The compact design of HMWs introduces non-trivial challenges to existing speech enhancement systems that use microphone recording only. In this paper, we handle this problem by using bone vibration conducted through the head skull. The principle is that the accelerometer is widely installed on head-mounted wearables and can capture the clean user's voice. Hence, we develop VibVoice, a lightweight multi-modal speech enhancement system for head-mounted wearables. We design a two-branch encoder-decoder deep neural network to fuse the high-level features of the two modalities and reconstruct clean speech. To address the issue of insufficient paired data for training, we extensively measure the bone conduction effect from a limited dataset to extract the physical impulse function for cross-modal data augmentation. We evaluate VibVoice on a dataset collected in real world and compare it with two state-of-the-art baselines. Results show that VibVoice yields up to 21% better performance in PESQ and up to 26% better performance in SNR compared with the baseline with 72 times less paired data required. We also conduct a user study with 35 participants, in which 87% participants prefer VibVoice compared with the baseline. In addition, VibVoice requires 4 to 31 times less execution time compared with baselines on mobile devices. The demo audio of VibVoice is available at https://www.youtube.com/watch?v=8_-s_C_NGRI.

Deep Learning (DL) models have been widely deployed on IoT devices with the help of advancements in DL algorithms and chips. However, the limited resources of edge devices make these on-device DL models hard to be generalizable to diverse environments and tasks. Although the recently emerged foundation models (FMs) show impressive generalization power, how to effectively leverage the rich knowledge of FMs on resource-limited edge devices is still not explored. In this paper, we propose EdgeFM, a novel edge-cloud cooperative system with open-set recognition capability. EdgeFM selectively uploads unlabeled data to query the FM on the cloud and customizes the specific knowledge and architectures for edge models. Meanwhile, EdgeFM conducts dynamic model switching at run-time taking into account both data uncertainty and dynamic network variations, which ensures the accuracy always close to the original FM. We implement EdgeFM using two FMs on two edge platforms. We evaluate EdgeFM on three public datasets and two self-collected datasets. Results show that EdgeFM can reduce the end-to-end latency up to 3.2x and achieve 34.3% accuracy increase compared with the baseline.

Today, there is an increasing number of smartphones supporting wireless charging that leverages electromagnetic induction to transmit power from a wireless charger to the charging smartphone. In this paper, we report a new contactless and context-aware wireless-charging side-channel attack, which captures two physical phenomena (i.e., the coil whine and the magnetic field perturbation) generated during this wireless charging process and further infers the user interactions on the charging smartphone. We design and implement a three-stage attack framework, dubbed WISERS, to demonstrate the practicality of this new side channel. WISERS first captures the coil whine and the magnetic field perturbation emitted by the wireless charger, then infers (i) inter-interface switches (e.g., switching from the home screen to an app interface) and (ii) intra-interface activities (e.g., keyboard inputs inside an app) to build user interaction contexts, and further reveals sensitive information. We extensively evaluate the effectiveness of WISERS with popular smartphones and commercial-off-the-shelf (COTS) wireless chargers. Our evaluation results suggest that WISERS can achieve over 90.4% accuracy in inferring sensitive information, such as screen-unlocking passcode and app launch. In addition, our study also shows that WISERS is resilient to a list of impact factors.

This paper presents TouchAuth, a new touch-to-access device authentication approach using induced body electric potentials (iBEPs) caused by the indoor ambient electric field that is mainly emitted from the building's electrical network. The design of TouchAuth is based on the electrostatics of iBEP generation and a resulting property, i.e., the iBEPs at two close locations on the same human body are similar, whereas those from different human bodies are distinct. Extensive experiments verify the above property and show that TouchAuth achieves high-profile receiver operating characteristics in implementing the touch-to-access policy. Our experiments also show that a range of possible interfering sources including appliances’ electromagnetic emanations and noise injections into the power network do not affect the performance of TouchAuth. A key advantage of TouchAuth is that the iBEP sensing requires a simple analog-to-digital converter only, which is widely available on microcontrollers. Compared with the existing approaches including intra-body communication and physiological sensing, TouchAuth is a low-cost, faster, and easy-to-use approach for authorized users to access the smart objects found in indoor environments.

Runtime domain shifts from the training phase caused by sensor characteristic variation incur performance drops of the deep learning-based sensing systems. To address this problem, existing transfer learning techniques require substantial target-domain data and incur high post-deployment overhead. Differently, we propose to exploit the first principle governing the domain shift to reduce the demand for target-domain data. Specifically, our proposed approach called PhyAug uses the first principle fitted with few labeled or unlabeled data pairs collected by the source sensor and the target sensor to transform the existing source-domain training data into the augmented target-domain data for calibrating the deep neural networks. In two audio sensing case studies of keyword spotting and automatic speech recognition, PhyAug recovers the recognition accuracy losses due to microphones’ characteristic variations by 37% to 72% with 5-second unlabeled data collected from the target microphones. In a case study of acoustics-based room recognition, PhyAug recovers the recognition accuracy loss caused by smartphone microphone variation by 33% to 80%. In the last case study of fisheye image recognition, PhyAug reduces the image recognition error due to the camera-induced distortions by 72%.

Drowning is a major cause of unintentional deaths in swimming pools. Most swimming pools hire lifeguards for continuous surveil-lance, which is labor-intensive and hence unfeasible for small private pools. The existing unmanned surveillance solutions like camera array requires non-trivial installations, only work in certain conditions (e.g., with adequate ambient lighting), or raise privacy concerns. This demo presents SwimSonar, the first practical drowning detection system based on underwater sonar. SwimSonar employs an active ultrasonic sonar and features a novel sonar scanning strategy that balances the time and accuracy. Lastly, SwimSonar leverages a deep neural network for accurate drowning detection. Our experiments in real swimming pools show that the system achieves 88 % classification accuracy with a scan time of 1.5 seconds.

Federated Learning (FL) is an emerging learning paradigm that enables the collaborative learning of different nodes without ex-posing the raw data. However, a critical challenge faced by the current federated learning algorithms in real-world applications is the long-tailed data distribution, i.e., in both local and global views, the numbers of classes are often highly imbalanced. This would lead to poor model accuracy on some rare but vital classes, e.g., those related to safety in health and autonomous driving applications. In this paper, we propose BalanceFL, a federated learning frame-work that can robustly learn both common and rare classes from a long-tailed real-world dataset, addressing both the global and local data imbalance at the same time. Specifically, instead of letting nodes upload a class-drifted model trained on imbalanced private data, we design a novel local update scheme that rectifies the class imbalance, forcing the local model to behave as if it were trained on ideal uniform distributed data. To evaluate the performance of BalanceFL, we first adapt two public datasets to the long-tailed federated learning setting, and then collect a real-life IMU dataset for action recognition, which includes more than 10,000 data sam-ples and naturally exhibits the global long tail effect and the local imbalance. On all of these three datasets, BalanceFL outperforms state-of-the-art federated learning approaches by a large margin.

Indoor localization by leveraging the existing residential instru-ments has been widely explored. Given the properties of tempo-ral stability and spatial distinctness, the electromagnetic radiation (EMR) from the powerline network is a promising signal for location sensing. In this demo, we present a three-sensor setup to capture the powerline EMR signal from the three-dimensional (3D) space and formulate a new powerline EMR feature to implement the simultaneous localization and mapping (SLAM). Compared with the single sensor setup, our proposed approach can improve the localization accuracy to decimeter level.

Infrastructure-assisted autonomous driving is an emerging paradigm that expects to significantly improve the driving safety of autonomous vehicles. The key enabling technology for this vision is to fuse LiDAR results from the roadside infrastructure and the vehicle to improve the vehicle's perception in real time. In this work, we propose VIPS, a novel lightweight system that can achieve decimeter-level and real-time (up to 100 ms) perception fusion between driving vehicles and roadside infrastructure. The key idea of VIPS is to exploit highly efficient matching of graph structures that encode objects' lean representations as well as their relationships, such as locations, semantics, sizes, and spatial distribution. Moreover, by leveraging the tracked motion trajectories, VIPS can maintain the spatial and temporal consistency of the scene, which effectively mitigates the impact of asynchronous data frames and unpredictable communication/compute delays. We implement VIPS end-to-end based on a campus smart lamppost testbed. To evaluate the performance of VIPS under diverse situations, we also collect two new multi-view point cloud datasets using the smart lamppost testbed and an autonomous driving simulator, respectively. Experiment results show that VIPS can extend the vehicle's perception range by 140% within 58 ms on average, and delivers a 4X improvement in perception fusion accuracy and 47X data transmission saving over existing approaches. A video demo of VIPS based on the lamppost dataset is available at https://youtu.be/zW4oi_EWOu0.

Massive fraudulent and phishing robocalls present threats to societies. The integration of artificial intelligence technologies, including dialogue and voice generation systems, renders the robocalls more deceptive. Existing countermeasures such as caller ID, call provenance, voiceprint, and fake voice detection have respective limitations and are heavyweight for end users' smartphones. This paper studies detecting the acoustic echo channel on the remote end of a call based on the received voice. The positive detection result evidencing the physical setup of an audio system is indicative of a human caller. However, the acoustic echo cancellation mechanisms of most audio systems and the use of earphone/headset diminish echoes significantly. To address these issues, the proposed Telesonar transmits short chirps during the vulnerable time of echo cancellation, detects the tiny echo remnants from the received voice, and passively analyzes the timing of caller's breath sounds to confirm a human caller. Extensive real experiments under a wide range of settings show that Telesonar correctly recognizes human callers with a rate of over 95%, while wrongly recognizing voice robots as human with a rate of 3.8%.

Traffic camera is one of the most ubiquitous traffic facilities, providing high coverage of complex, accident-prone road sections such as intersections. This work leverages traffic cameras to improve the perception and localization performance of autonomous vehicles at intersections. In particular, vehicles can expand their range of perception by matching the images captured by both the traffic cameras and on-vehicle cameras. Moreover, a traffic camera can match its images to an existing high-definition map (HD map) to derive centimeter-level location of the vehicles in its field of view. To this end, we propose AutoMatch - a novel system for real-time image registration, which is a key enabling technology for traffic camera-assisted perception and localization of autonomous vehicles. Our key idea is to leverage landmark keypoints of distinctive structures such as ground signs at intersections to facilitate image registration between traffic cameras and HD maps or vehicles. By leveraging the strong structural characteristics of ground signs, AutoMatch can extract very few but precise landmark keypoints for registration, which effectively reduces the communication/compute overhead. We implement AutoMatch on a testbed consisting of a self-built autonomous car, drones for surveying and mapping, and real traffic cameras. In addition, we collect two new multi-view traffic image datasets at intersections, which contain images from 220 real operational traffic cameras in 22 cities. Experimental results show that AutoMatch achieves pixel-level image registration accuracy within 88 milliseconds, and delivers an 11.7× improvement in accuracy, 1.4× speedup in compute time, and 17.1× data transmission saving over existing approaches.

In recent years, Deep Neural Network (DNN) has been increasingly adopted by a wide range of time-critical applications running on edge platforms with heterogeneous multiprocessors. To meet the stringent timing requirements of these applications, heterogeneous CPU and GPU resources must be efficiently utilized for the inference of multiple DNN models. Such a cross-processor real-time DNN inference paradigm poses major challenges due to the inherent performance imbalance among different processors and the lack of real-time support for cross-processor inference from existing deep learning frameworks. In this work, we propose a new system named BlastNet that exploits duo-block - a new model inference abstraction to support highly efficient cross-processor real-time DNN inference. Each duo-block has a dual model structure, enabling efficient fine-grained inference alternatively across different processors. BlastNet employs a novel block-level Neural Architecture Search (NAS) technique to generate duo-blocks, which accounts for computing characteristics and communication overhead. The duo-blocks are optimized at design time and then dynamically scheduled to achieve high resource utilization of heterogeneous CPU and GPU at runtime. BlastNet is implemented on an indoor autonomous driving platform and three popular edge platforms. Extensive results show that BlastNet achieves 35.07 % less deadline missing rate with a mere 1.63% of model accuracy loss.

Indoor self-localization is a highly demanded system function for smartphones. The current solutions based on inertial, radio frequency, and geomagnetic sensing may have degraded performance when their limiting factors take effect. In this paper, we present a new indoor simultaneous localization and mapping (SLAM) system that utilizes the smartphone's built-in audio hardware and inertial measurement unit (IMU). Our system uses a smartphone's loud-speaker to emit near-inaudible chirps and then the microphone to record the acoustic echoes from the indoor environment. Our profiling measurements show that the echoes carry location information with sub-meter granularity. To enable SLAM, we apply contrastive learning to construct an echoic location feature (ELF) extractor, such that the loop closures on the smartphone's trajectory can be accurately detected from the associated ELF trace. The detection results effectively regulate the IMU-based trajectory reconstruction. Extensive experiments show that our ELF-based SLAM achieves median localization errors of 0.1 m, 0.53 m, and 0.4m on the reconstructed trajectories in a living room, an office, and a shopping mall, and outperforms the Wi-Fi and geomagnetic SLAM systems.

Deep learning-based visual sensing has achieved attractive accuracy but is shown vulnerable to adversarial attacks. Specifically, once the attackers obtain the deep model, they can construct adversarial examples to mislead the model to yield wrong classification results. Deployable adversarial examples such as small stickers pasted on the road signs and lanes have been shown effective in misleading advanced driver-assistance systems. Most existing countermeasures against adversarial examples build their security on the attackers’ ignorance of the defense mechanisms. Thus, they fall short of following Kerckhoffs’s principle and can be subverted once the attackers know the details of the defense. This article applies the strategy of moving target defense (MTD) to generate multiple new deep models after system deployment that will collaboratively detect and thwart adversarial examples. Our MTD design is based on the adversarial examples’ minor transferability across different models. The post-deployment of dynamically generated models significantly increase the bar of successful attacks. We also apply serial data fusion with early stopping to reduce the inference time by a factor of up to 5, as well as exploit hardware inference accelerators’ characteristics to strike better tradeoffs between inference time and power consumption. Evaluation based on three datasets including a road sign dataset and two GPU-equipped embedded computing boards shows the effectiveness and efficiency of our approach in counteracting the attack.

Run-time domain shifts from training-phase domains are common in sensing systems designed with deep learning. The shifts can be caused by sensor characteristic variations and/or discrepancies between the design-phase model and the actual model of the sensed physical process. To address these issues, existing transfer learning techniques require substantial target-domain data and thus incur high post-deployment overhead. This paper proposes to exploit the first principle governing the domain shift to reduce the demand on target-domain data. Specifically, our proposed approach called PhyAug uses the first principle fitted with few labeled or unlabeled source/target-domain data pairs to transform the existing source-domain training data into augmented data for updating the deep neural networks. In two case studies of keyword spotting and DeepSpeech2-based automatic speech recognition, with 5-second unlabeled data collected from the target microphones, PhyAug recovers the recognition accuracy losses due to microphone characteristic variations by 37% to 72%. In a case study of seismic source localization with TDoA fingerprints, by exploiting the first principle of signal propagation in uneven media, PhyAug only requires 3% to 8% of labeled TDoA measurements required by the vanilla fingerprinting approach in achieving the same localization accuracy.

Smartphone indoor location awareness is increasingly demanded by a variety of mobile applications. The existing solutions for accurate smartphone indoor localization rely on additional devices or pre-installed infrastructure (e.g., dense WiFi access points, Bluetooth beacons). In this demo, we present EchoLoc, an infrastructure-free smartphone indoor localization system using room acoustic response to a chirp emitted by the phone. EchoLoc consists of a mobile client for echo data collection and a cloud server hosting a deep neural network for location inference. EchoLoc achieves 95% accuracy in recognizing 101 locations in a large public indoor space and a median localization error of 0.5 m in a typical lab area. Demo video is available at https://youtu.be/5si0Cq6LzT4.

This paper presents TouchAuth, a new touch-to-access device authentication approach using induced body electric potentials (iBEPs) caused by the indoor ambient electric field that is mainly emitted from the building's electrical cabling. The design of TouchAuth is based on the electrostatics of iBEP generation and a resulting property, i.e., the iBEPs at two close locations on the same human body are similar, whereas those from different human bodies are distinct. Extensive experiments verify the above property and show that TouchAuth achieves high-profile receiver operating characteristics in implementing the touch-to-access policy. Our experiments also show that a range of possible interfering sources including appliances' electromagnetic emanations and noise injections into the power network do not affect the performance of TouchAuth. A key advantage of TouchAuth is that the iBEP sensing requires a simple analog-to-digital converter only, which is widely available on microcontrollers. Compared with existing approaches including intra-body communication and physiological sensing, TouchAuth is a low-cost, lightweight, and convenient approach for authorized users to access the smart objects found in indoor environments.

Deep learning-based visual sensing has achieved attractive accuracy but is shown vulnerable to adversarial example attacks. Specifically, once the attackers obtain the deep model, they can construct adversarial examples to mislead the model to yield wrong classification results. Deployable adversarial examples such as small stickers pasted on the road signs and lanes have been shown effective in misleading advanced driver-assistance systems. Many existing countermeasures against adversarial examples build their security on the attackers' ignorance of the defense mechanisms. Thus, they fall short of following Kerckhoffs's principle and can be subverted once the attackers know the details of the defense. This paper applies the strategy of moving target defense (MTD) to generate multiple new deep models after system deployment, that will collaboratively detect and thwart adversarial examples. Our MTD design is based on the adversarial examples' minor transferability across different models. The post-deployment dynamically generated models significantly increase the bar of successful attacks. We also apply serial data fusion with early stopping to reduce the inference time by a factor of up to 5. Evaluation based on four datasets including a road sign dataset and two GPU-equipped Jetson embedded computing platforms shows the effectiveness of our approach.

Design of clock synchronization for networked nodes faces a fundamental trade-off between synchronization accuracy and universality of heterogeneous platforms, because a high synchronization accuracy generally requires platform-dependent hardware-level network packet timestamping. This paper presents TouchSync, a new indoor clock synchronization approach for wearables that achieves millisecond accuracy while preserving universality in that it uses standard system calls only, such as reading system clock, sampling sensors, and sending/receiving network messages. The design of TouchSync is driven by a key finding from our extensive measurements that the skin electric potentials (SEPs) induced by powerline radiation are salient, periodic, and synchronous on a same wearer and even across different wearers. TouchSync integrates the SEP signal into the universal principle of Network Time Protocol and solves an integer ambiguity problem by fusing the ambiguous results in multiple synchronization rounds to conclude an accurate clock offset between two synchronizing wearables. With our shared code, TouchSync can be readily integrated into any wearable applications. Extensive evaluation based on our Arduino and TinyOS implementations shows that TouchSync's synchronization errors are below 3 and 7 milliseconds on the same wearer and between two wearers 10 kilometers apart, respectively.

Design of clock synchronization for networked nodes faces a fundamental trade-off between synchronization accuracy and universality for heterogeneous platforms, because a high synchronization accuracy generally requires platform-dependent hardware-level network packet timestamping. This paper presents TouchSync, a new indoor clock synchronization approach for wearables that achieves millisecond accuracy while preserving universality in that it uses standard system calls only, such as reading system clock, sampling sensors, and sending/receiving network messages. The design of TouchSync is driven by a key finding from our extensive measurements that the skin electric potentials (SEPs) induced by powerline radiation are salient, periodic, and synchronous on a same wearer and even across different wearers. TouchSync integrates the SEP signal into the universal principle of Network Time Protocol and solves an integer ambiguity problem by fusing the ambiguous results in multiple synchronization rounds to conclude an accurate clock offset between two synchronizing wearables. With our shared code, TouchSync can be readily integrated into any wearable applications. Extensive evaluation based on our Arduino and TinyOS implementations shows that TouchSync's synchronization errors are below 3 and 7 milliseconds on the same wearer and between two wearers 10 kilometers apart, respectively.