This paper presents E-Cube, a framework designed to enhance the efficiency of mobile video streaming systems. Our core insight is that content correlations among frames, which are critical for video streaming but challenging to extract in dynamic scenes, are often already available as intermediate outputs from other mobile computing subsystems. By adopting a cross-subsystem design, E-Cube repurposes these intermediate results obtained from event cameras to reduce computation, energy consumption, and bandwidth usage of existing video streaming systems, while simultaneously improving video quality. We implement E-Cube on a drone-embedded chip through software-hardware co-design, and plugged E-Cube into three prevalent drone video streaming systems for industrial inspection. Evaluations in one of the world's largest oil fields and on public datasets demonstrate E-Cube can save over 35% network bandwidth overhead and reduce drone video streaming energy consumption by over 12% for H.265 and AV1, and 47% for advanced H.266, all while achieving improved video quality.

As drone-based applications proliferate, paramount contactless sensing of airborne drones from the ground becomes indispensable. This work demonstrates concentrating on propeller rotational speed will substantially improve drone sensing performance and proposes an event-camera-based solution, EventPro. EventPro features two components: Count Every Rotation achieves accurate, real-time propeller speed estimation by mitigating ultra-high sensitivity of event cameras to environmental noise. Every Rotation Counts leverages these speeds to infer both internal and external drone dynamics. Extensive evaluations in real-world drone delivery scenarios show that EventPro achieves a sensing latency of 3 ms and a rotational speed estimation error of merely 0.23%. Additionally, EventPro infers drone flight commands with 96.5% precision and improves drone tracking accuracy by over 22% when combined with other sensing modalities. Demo: https://eventpro25.github.io/EventPro/.

This article introduces a radar perception learning framework guided by data collected from commonly equipped visual-inertial (VI) sensor suites on smart vehicles. Unlike existing approaches that rely on dense point clouds from 3D LiDARs, which are costly and not widely deployed, this method leverages the broader availability of VI data. However, visual images alone lack the ability to capture the three-dimensional motion of moving targets, which limits their effectiveness in supervising motion-related tasks. To overcome this limitation, the framework integrates multiple perception tasks such as odometry estimation, motion segmentation, and scene flow prediction into a unified learning process. The first component is an odometry estimation module that combines deterministic ego-motion models with data-driven learning results. This fusion helps accurately infer the scene flow of static background points while minimizing drift. The second component is a supervision signal extraction module that aligns optical and millimeter-wave radar measurements to guide the learning of radar scene flow and rigid transformations. This module improves the reliability of dynamic point supervision through joint constraints across sensing modalities. The third component introduces a feature-selection module designed for cross-modal learning. It enhances the accuracy of motion segmentation and enforces consistency between odometry and scene flow, resulting in more coherent radar perception outputs. Experimental evaluations show that this framework achieves superior performance in challenging conditions such as smoke-obscured environments. It surpasses state-of-the-art (SOTA) methods that depend on high-cost LiDAR systems. The implementation of VISC+ will be open-source athttps://github.com/weini-Eve/VISC

For precise, efficient, and safe drone landings, ground platforms should real-time, accurately locate descending drones and guide them to designated spots. While mmWave sensing combined with cameras improves localization accuracy, lower sampling frequency of traditional frame cameras compared to mmWave radar creates bottlenecks in system throughput. In this work, we upgrade traditional frame camera with event camera, a novel sensor that harmonizes in sampling frequency with mmWave radar within ground platform setup, and introduce mmE-Loc, a high-precision, low-latency ground localization system designed for precise drone landings. To fully exploit thetemporal consistencyandspatial complementaritybetween these two modalities, we propose two innovative modules:(i)the Consistency-instructed Collaborative Tracking module, which further leverages the drone's physical knowledge of periodic micro-motions and structure for accurate measurements extraction, and(ii)the Graph-informed Adaptive Joint Optimization module, which integrates drone motion information for efficient sensor fusion and drone localization. Extensive experiments (30+ hours) demonstrate that mmE-Loc attains 0.083$m$localization accuracy and 5.12$ms$end-to-end latency, outperforming four state-of-the-art methods by over 48% and 62%, respectively.

A heterogeneous micro aerial vehicles (MAV) swarm consists of resource-intensive but expensive advanced MAVs (AMAVs) and resource-limited but cost-effective basic MAVs (BMAVs), offering opportunities in diverse fields. Accurate and real-time localization is crucial for MAV swarms, but current practices lack a low-cost, high-precision, and real-time solution, especially for lightweight BMAVs. We find an opportunity to accomplish the task by transforming AMAVs into mobile localization infrastructures for BMAVs. However, translating this insight into a practical system is challenging due to issues in estimating locations with diverse and unknown localization errors of BMAVs, and allocating resources of AMAVs considering interconnected influential factors. This work introduces TransformLoc, a new framework that transforms AMAVs into mobile localization infrastructures, specifically designed for low-cost and resource-constrained BMAVs. We design an error-aware joint location estimation model to perform intermittent joint estimation for BMAVs and introduce a similarity-instructed adaptive grouping-scheduling strategy to allocate resources of AMAVs dynamically. TransformLoc achieves a collaborative, adaptive, and cost-effective localization system suitable for large-scale heterogeneous MAV swarms. We implement and validate TransformLoc on industrial drones. Results show it outperforms all baselines by up to 68% in localization performance, improving navigation success rates by 60%. Extensive robustness and ablation experiments further highlight superiority of its design.

Gas source localization is pivotal for the rapid mitigation of gas leakage disasters, where mobile robots emerge as a promising solution. However, existing methods predominantly schedule robots’ movements based on reactive stimuli or simplified gas plume models. These approaches typically excel in idealized, simulated environments but fall short in real-world gas environments characterized by their patchy distribution. In this work, we introduce SniffySquad , a multi-robot olfaction-based system designed to address the inherent patchiness in gas source localization. SniffySquad incorporates a patchiness-aware active sensing approach that enhances the quality of data collection and estimation. Moreover, it features an innovative collaborative role adaptation strategy to boost the efficiency of source-seeking endeavors. Extensive evaluations demonstrate that our system achieves an increase in the success rate by \(20\%+\) and an improvement in path efficiency by \(30\%+\) , outperforming state-of-the-art gas source localization solutions.

Event cameras excel in high temporal resolution and dynamic range but suffer from dense noise in rainy conditions. Existing event deraining methods face trade-offs between temporal precision, deraining effectiveness, and computational efficiency. In this paper, we propose PRE-Mamba, a novel point-based event camera deraining framework that fully exploits the spatiotemporal characteristics of raw event and rain. Our framework introduces a 4D event cloud representation that integrates dual temporal scales to preserve high temporal precision, a Spatio-Temporal Decoupling and Fusion module (STDF) that enhances deraining capability by enabling shallow decoupling and interaction of temporal and spatial information, and a Multi-Scale State Space Model (MS3M) that captures deeper rain dynamics across dual-temporal and multi-spatial scales with linear computational complexity. Enhanced by frequency-domain regularization, PRE-Mamba achieves superior performance (0.95 SR, 0.91 NR, and 0.4s/M events) with only 0.26M parameters on EventRain-27K, a comprehensive dataset with labeled synthetic and real-world sequences. Moreover, our method generalizes well across varying rain intensities, viewpoints, and even snowy conditions.

Graph-based social recommendation (SR) models suffer from various noises of the social graphs, hindering their recommendation performances. Either graph-level redundancy or graph-level missing will indeed influence the social graph structures, further influencing the message propagation procedure of graph neural networks (GNNs). Generative models, especially diffusion-based models, are usually used to reconstruct and recover the data in better quality from original data with noises. Motivated by it, a few works take attempts on it for social recommendation. However, they can only handle isotropic Gaussian noises but fail to leverage the anisotropic ones. Meanwhile the anisotropic relational structures in social graphs are commonly seen, so that existing models cannot sufficiently utilize the graph structures, which constraints the capacity of noise removal and recommendation performances. Compared to the diffusion strategy, the flow matching strategy shows better ability to handle the data with anisotropic noises since they can better preserve the data structures during the learning procedure. Inspired by it, we propose RecFlow which is the first flow-matching based SR model. Concretely, RecFlow performs flow matching on the structure representations of social graphs. Then, a conditional learning procedure is designed for optimization. Extensive performances prove the promising performances of our RecFlow from six aspects, including superiority, effectiveness, robustnesses, sensitivity, convergence and visualization.

Vision Language Models (VLMs) can enhance Internet of Things (IoT) applications by efficiently extracting valuable information from excessively long videos captured by IoT cameras. Due to the large volume of video data and the high computation overhead of VLMs, a practical deployment strategy is to transmit the video to the cloud only on demand and also deploy the VLMs on the cloud for video analytics. Yet, the interaction experience between humans and VLMs is degraded by the high latency in such cloud-edge collaboration applications. The latency is caused by both the video transmission process and the heavy VLM inference process. We propose SwiftReTaKe, a two-round transmission framework coupled with a low-latency pre-pruning strategy to reduce both network and inference latency. By first sending keyframes for relevance estimation and then adaptively transmitting informative frames, SwiftReTaKe minimizes data transfer and LLM computation. Compared to the state-of-the-art (SOTA) long video processing method, SwiftReTaKe reduces the latency by 6 times with only 3.33% accuracy drop.

Event cameras are increasingly integrated into SLAM systems to mitigate motion blur and enhance tracking performance in low-light environments. However, the event stream by nature lacks essential semantic and color information, posing a fundamental challenge to visual feature extraction and mapping. This paper introduces Ev-SLAM, a novel event-based visual SLAM system that harnesses implicit neural 3D scene representation to enhance visual feature extraction and SLAM performance. We design three modules to achieve this goal, including (1) a dynamic-slicing algorithm to address the impact of polarity neutralization in event frame generation; (2) a point cloud representation that minimizes redundant sampling in free space, significantly reducing the computational overhead of neural radiance field (NeRF) training; and (3) an optical-flow-based approach for coarse camera pose and depth estimation, coupled with a scale feedback mechanism to bootstrap NeRF training. Extensive evaluations demonstrate that Ev-SLAM outperforms SOTA baselines Point-SLAM and Co-SLAM, achieving up to an 82.9% improvement in tracking accuracy under normal-speed conditions and 85.2% under fast-speed conditions.

Despite advances in hardware acceleration, implementing AI on mobile devices is difficult when tight real-time latency bounds have to be met without compromising accuracy. A simple solution is edge offload: using a low-latency wireless network to perform the AI on a nearby cloudlet. This approach also avoids the software engineering effort of downsizing cloud-based AI. In this paper, we experimentally compare on-device and offloaded AI execution by introducing a set of new benchmarks for computer vision tasks. The results show that edge offload is Pareto-optimal across accuracy and latency. It also greatly reduces on-device energy usage.

This work proposes a mmWave radar’s scene flow estimation framework supervised by data from a widespread visual-inertial (VI) sensor suite, allowing crowdsourced training data from smart vehicles. Current scene flow estimation methods for mmWave radar are typically supervised by dense point clouds from 3D LiDARs, which are expensive and not widely available in smart vehicles. While VI data are more accessible, visual images alone cannot capture the 3D motions of moving objects, making it difficult to supervise their scene flow. Moreover, the temporal drift of VI rigid transformation also degenerates the scene flow estimation of static points. To address these challenges, we propose a drift-free rigid transformation estimator that fuses kinematic model-based ego-motions with neural network-learned results. It provides strong supervision signals to radar-based rigid transformation and infers the scene flow of static points. Then, we develop an optical-mmWave supervision extraction module that extracts the supervision signals of radar rigid transformation and scene flow. It strengthens the supervision by learning the scene flow of dynamic points with the joint constraints of optical and mmWave radar measurements. Extensive experiments demonstrate that, in smoke-filled environments, our method even outperforms state-of-the-art (SOTA) approaches using costly LiDARs.

Neural enhancement through super-resolution (SR) deep neural networks (DNNs) opens up new possibilities for ultra-high-definition (UHD) live streaming. Yet, the heavy SR DNN inference overhead leads to severe deployment challenges. To reduce the overhead, existing systems propose to apply DNN-based SR only on carefully selected anchor frames while upscaling non-anchor frames via the lightweight reusing-based SR approach. However, frame-level scheduling is coarse-grained and fails to deliver optimal efficiency. In this work, we propose Palantír, the first neural-enhanced UHD live streaming system with fine-grained patch-level scheduling.

Drones are increasingly used in applications such as last-mile delivery and infrastructure inspection, but their safe operation, especially in high-speed scenarios, remains a critical challenge. Existing vision- and LiDAR-based obstacle localization methods suffer from motion blur, latency, and low spatio-temporal resolution, making them inadequate for detecting and tracking fast-moving objects. In this work, we present HawkEye, a drone obstacle avoidance system that fuses event cameras and LiDAR to achieve high-frequency, accurate 3D tracking of dynamic objects. By leveraging the complementary strengths of both sensors, Hawkeye enables robust real-time sensing and safe evasive maneuvers, addressing a key requirement for the large-scale deployment of autonomous drones. Demo: https://wenhua00.github.io/HawkEye/.

A long-standing concern with GPS-based location sensing is its vulnerability to satellite signal loss. This may arise from adversarial attacks or natural causes such as urban canyons. Today, there are no real alternatives to GPS for providing absolute global coordinates. We address this concern by introducing TerraSLAM, a new global positioning system that uses a 3D GIS model to bridge relative and absolute coordinate systems, thereby enhancing visual SLAM to function as a global positioning solution. TerraSLAM offers localization accuracy and efficiency comparable to GPS-RTK even in GPS-denied settings. Extensive evaluation on drone localization scenarios shows that TerraSLAM achieves an average global positioning accuracy of 0.21m and a 99th percentile within 0.67m, outperforming advanced GPS solutions by over 70% (0.72m) and 80% (3.62m). Additionally, when integrated with ORB-SLAM3, the localization latency per frame is 16.7ms, achieving a 60% reduction compared to the baseline of 41.3ms. Code is available at https://github.com/cmusatyalab/TerraSLAM.

For precise, efficient, and safe drone landings, ground platforms should real-time, accurately locate descending drones and guide them to designated spots. While mmWave sensing combined with cameras improves localization accuracy, lower sampling frequency of traditional frame cameras compared to mmWave radar creates bottlenecks in system throughput. In this work, we replace traditional frame camera with event camera, a novel sensor that harmonizes in sampling frequency with mmWave radar within ground platform setup, and introduce mmE-Loc, a high-precision, low-latency ground localization system designed for drone landings. To fully leverage the temporal consistency and spatial complementarity between these modalities, we propose two innovative modules, consistency-instructed collaborative tracking and graph-informed adaptive joint optimization, for accurate drone measurement extraction and efficient sensor fusion. Real-world experiments in landing scenarios from a drone delivery company demonstrate that mmE-Loc outperforms SOTA methods in both accuracy and latency.

Fast and accurate obstacle avoidance is crucial to drone safety. Yet existing on-board sensor modules such as frame cameras and radars are ill-suited for doing so due to their low temporal resolution or limited field of view. This paper presentsBioDrone, a new design paradigm for drone obstacle avoidance using stereo event cameras. At the heart of BioDrone are three simple yet effective system designs inspired by the mammalian visual system, namely, a chiasm-inspired event filtering, a lateral geniculate nucleus (LGN)-inspired event matching, and a dorsal stream-inspired obstacle tracking. We implement BioDrone on FPGA through software-hardware co-design and deploy it on an industrial drone. In comparative experiments against two state-of-the-art event-based systems, BioDrone consistently achieves an obstacle detection rate of$> $90%, and an obstacle tracking error of$<$5.8 cm across all flight modes with an end-to-end latency of$<$6.4 ms, outperforming both baselines by over 44%.

Mobile air pollution sensing methods have emerged to collect air quality data with improved spatial and temporal resolutions. However, existing methodologies struggle to effectively process spatially mixed gas samples due to the highly dynamic fluctuations experienced by sensors, resulting in significant measurement deviations. We identify an opportunity to address this issue by exploring potential patterns within sensor measurements. To this end, we propose CatUA, a novel city-scale fine-grained air quality estimation system designed to deliver accurate mobile air quality data. First, we design AirBERT, a representation learning model specifically aimed at discerning mixed gas concentrations from sensor data. Second, we implement a Prompt-informed Training Strategy that leverages extensive unlabeled and minimal labeled city-scale data to enhance the performance of CatUA. Notably, the Auto-Prompt mechanism allows CatUA to conveniently acquire new knowledge tailored to specific downstream tasks. To ensure the practicality of CatUA, we have invested considerable effort in developing the software stack on our meticulously crafted Sensing Front-end, which has successfully gathered city-scale air quality data for over 1,200 hours. Experiments conducted on the collected data demonstrate that CatUA reduces sensing errors by 96.9% with a latency of only 44.9ms, outperforming the state-of-the-art baseline by 42.6%.

Time-Sensitive Networking (TSN) is foreseen as a foundational technology that enables Industry 4.0. It offers deterministic data transmission over Ethernet for critical applications such as industrial control and automotive systems. However, TSN is susceptible to hardware and software errors, necessitating an effective diagnostic system. Traditional network diagnostic tools are inadequate for TSN fault localization and classification due to the tightly coupled traffic and high precision requirements in TSN. In response, this paper presents TSNCard, a cross-cycle postcard-based diagnostic system tailored for Time-Aware Shaper (IEEE 802.1 Qbv) in TSN. TSNCard introduces a novel telemetry protocol that leverages the cyclical nature of TSN networks for data collection at each node. This protocol, coupled with dedicated analytic algorithms and hardware innovations within switches, forms a comprehensive system for TSN monitoring, fault localization and classification. Extensive experiments on both simulation and physical testbeds show that TSNCard can 100% detect fault location and type of the TSN misbehavior while adhering to industrial bandwidth restrictions. TSNCard not only bridges the gap in the TSN protocol stack, but also serves as a versatile toolkit for time-synchronized network analysis, paving the way for future research. The code is available athttps://github.com/MobiSense/TSNCard

We present a benchmark-driven experimental study of autonomous drone agility relative to edge offload pipeline attributes. This pipeline includes a monocular gimbal-actuated on-drone camera, hardware RTSP video encoding, 4G LTE wireless network transmission, and computer vision processing on a ground-based GPU-equipped cloudlet. Our parameterized and reproducible agility benchmarks stress the OODA (“Observe, Orient, Decide, Act”) loop of the drone on obstacle avoidance and object tracking tasks. We characterize the latency and throughput of components of this OODA loop through software profiling, and identify opportunities for optimization.

Drones have demonstrated their pivotal role in various applications such as search-and-rescue, smart logistics, and industrial inspection, with accurate localization playing an indispensable part. However, in high dynamic range and rapid motion scenarios, traditional visual sensors often face challenges in pose estimation. Event cameras, with their high temporal resolution, present a fresh opportunity for perception in such challenging environments. Current efforts resort to event-visual fusion to enhance the drone’s sensing capability. Yet, the lack of efficient event-visual fusion algorithms and corresponding acceleration hardware causes the potential of event cameras to remain underutilized. In this paper, we introduce EventBoost, an acceleration platform designed for drone-based applications with event-image fusion. We propose a suit of novel algorithms through software-hardware co-design on Zynq SoC, aimed at enhancing real-time localization precision and speed. EventBoost achieves enhanced visual fusion precision and markedly elevated processing efficiency. The performance comparison with two state-of-the-art systems shows EventBoost achieves 24.33% improvement in accuracy with 30 ms latency on resource-constrained platforms.

Edge-assisted visual SLAM stands as a pivotal enabler for emerging mobile applications, such as search-and-rescue, smart logistics, and industrial inspection. Limited by the computing capability of lightweight mobile devices like MAVs, current innovations balance system accuracy and efficiency by allocating lightweight and time-sensitive tracking tasks to mobile devices, while offloading the more resource-intensive yet delay-tolerant map optimization tasks to the edge. However, our pilot study in a large-scale oil field reveals several limitations of such a tracking-optimization decoupled paradigm, arising due to the disruption of inter-dependencies between the two tasks concerning data, resources, and threads.In this paper, we design and implement edgeSLAM2, an innovative system that reshapes the edge-assisted visual SLAM paradigm by tightly integrating tracking and partial-yet-crucial optimization on mobile. edgeSLAM2 harnesses the hierarchical and heterogeneous computing units offered by the latest commercial systems-on-chip (SoCs) to enhance the computational capacity of mobile devices, which in turn, allows edgeSLAM2 to design a suit of novel algorithms for map sync, optimization, and tracking that accommodate such architectural upgrade. By fully embracing the on-chip intelligence, edgeSLAM2 simultaneously enhances system accuracy and efficiency through software-hardware co-design. We deploy edgeSLAM2 on an industrial drone and conduct comprehensive experiments in a large-scale oil field over three months. The results show that edgeSLAM2 surpasses comparative methods by achieving an 80% reduction in bandwidth consumption, a 32% improvement in accuracy, and a 26% reduction in tracking delay.

A heterogeneous micro aerial vehicles (MAV) swarm consists of resource-intensive but expensive advanced MAVs (AMAVs) and resource-limited but cost-effective basic MAVs (BMAVs), offering opportunities in diverse fields. Accurate and real-time localization is crucial for MAV swarms, but current practices lack a low-cost, high-precision, and real-time solution, especially for lightweight BMAVs. We find an opportunity to accomplish the task by transforming AMAVs into mobile localization infrastructures for BMAVs. However, turning this insight into a practical system is non-trivial due to challenges in location estimation with BMAVs’ unknown and diverse localization errors and resource allocation of AMAVs given coupled influential factors. This study proposes TransformLoc, a new framework that transforms AMAVs into mobile localization infrastructures, specifically designed for low-cost and resource- constrained BMAVs. We first design an error-aware joint location estimation model to perform intermittent joint location estimation for BMAVs and then design a proximity-driven adaptive grouping-scheduling strategy to allocate resources of AMAVs dynamically. TransformLoc achieves a collaborative, adaptive, and cost-effective localization system suitable for large-scale heterogeneous MAV swarms. We implement TransformLoc on industrial drones and validate its performance. Results show that TransformLoc outperforms baselines including SOTA up to 68% in localization performance, motivating up to 60% navigation success rate improvement.

Time-Sensitive Networking (TSN) is foreseen as a foundational technology that enables Industry 4.0. It offers deterministic data transmission over Ethernet for critical applications such as industrial control and automotive systems. However, TSN is susceptible to hardware and software errors, necessitating an effective diagnostic system. Traditional network diagnostic tools are inadequate for TSN fault localization due to the unique characteristics of TSN. In response, this paper presents TSNCard, a cross-cycle postcard-based diagnostic system tailored for TSN. TSNCard introduces a novel telemetry protocol that leverages the cyclical nature of TSN networks for data collection at each node. This protocol, coupled with dedicated analytic algorithms and hardware innovations within switches, forms a comprehensive system for TSN monitoring and fault localization. Extensive experiments on both simulation and physical testbeds show that TSNCard can 100% localize the root cause of the TSN misbehavior while adhering to industrial bandwidth restrictions. TSNCard not only bridges the gap in the TSN protocol stack, but also serves as a versatile toolkit for time-synchronized network analysis, paving the way for future research.

Along with AIGC shines in CV and NLP, its potential in the wireless domain has also emerged in recent years. Yet, existing RF-oriented generative solutions are ill-suited for generating high-quality, time-series RF data due to limited representation capabilities. In this work, inspired by the stellar achievements of the diffusion model in CV and NLP, we adapt it to the RF domain and propose RF-Diffusion. To accommodate the unique characteristics of RF signals, we first introduce a novel Time-Frequency Diffusion theory to enhance the original diffusion model, enabling it to tap into the information within the time, frequency, and complex-valued domains of RF signals. On this basis, we propose a Hierarchical Diffusion Transformer to translate the theory into a practical generative DNN through elaborated design spanning network architecture, functional block, and complex-valued operator, making RF-Diffusion a versatile solution to generate diverse, high-quality, and time-series RF data. Performance comparison with three prevalent generative models demonstrates the RF-Diffusion's superior performance in synthesizing Wi-Fi and FMCW signals. We also showcase the versatility of RF-Diffusion in boosting Wi-Fi sensing systems and performing channel estimation in 5G networks.

Accurately localizing high-speed dynamic objects in 3D space with low latency is crucial for various robotic applications. Current methods face challenges due to extended exposure times and limited sensor resolution, hindering precise object detection and localization. Event cameras, known for their high temporal resolution and asynchronous nature, offer a promising solution. To leverage the potential of the event camera, we propose EventTracker, a novel framework that integrates event and depth measurements for precise and low-latency 3D localization and tracking of the high-speed dynamic object. EventTracker incorporates a collaborative object detection and tracking algorithm optimized for both event and depth data, overcoming detection and registration challenges. Additionally, a graph-instructed optimization algorithm enhances accuracy by fusing heterogeneous sensor data effectively. Experimental evaluation in dynamic environments demonstrates significant improvements in localization performance compared to baseline methods.

Event cameras excel in high-speed and high-dynamic-range scenarios but are highly sensitive to rain, which introduces significant noise while also revealing detailed rain features. This paper introduces a novel Event-based Rain-Background Decomposition Network that integrates Spiking Neural Networks (SNNs) and Convolutional Neural Networks (CNNs). By "Distilling Rain," we reconstruct a rain-free background for downstream tasks, and by "Collecting Rain," we extract the physical characteristics of rain. Experimental evaluations demonstrate the network's effectiveness in both background reconstruction and rain modeling. This work extends the capabilities of event cameras by mitigating the adverse effects of rain while also leveraging rain-induced noise to extract valuable environmental data, enhancing their utility in both challenging weather conditions and detailed environmental analysis.

Drone-based rapid and accurate environmental edge detection is highly advantageous for tasks such as disaster relief and autonomous navigation. Current methods, using radar or cameras, raise deployment costs and burden lightweight drones with high computational demands. In this paper, we propose AirTouch, a system that transforms the ground effect from a stability "foe" in traditional flight control views, into a "friend" for accurate and efficient edge detection. Our key insight is that analyzing drone sensor readings and flight commands allows us to detect ground effect changes. Such changes typically indicate the drone flying over an edge, making this information valuable for edge detection. We approach this insight through theoretical analysis, algorithm design, and implementation, fully leveraging the ground effect as a new sensing modality without compromising drone flight stability, thereby achieving accurate and efficient scene edge detection. Extensive evaluations demonstrate that our system achieves a high detection accuracy with mean detection distance errors of 0.051m, outperforming the baseline performance by 86%.

Mobile air pollution sensing methods are developed to collect air quality data with higher spatial-temporal resolutions. However, existing methods cannot process the spatially mixed gas samples effectively due to the highly dynamic temporal and spatial fluctuations experienced by the sensor, leading to significant measurement deviations. We find an opportunity to tackle the problem by exploring the potential patterns from sensor measurements. In light of this, we propose MobiAir, a novel city-scale fine-grained air quality estimation system to deliver accurate mobile air quality data. First, we design AirBERT, a representation learning model to discern mixed gas concentrations. Second, we design a knowledge-informed training strategy leveraging massive unlabeled city-scale data to enhance the AirBERT performance. To ensure the practicality of MobiAir, we have invested significant efforts in implementing the software stack on our meticulously crafted Sensing Front-end, which has successfully gathered air quality data at a city-scale for more than 1200 hours. Experiments conducted on collected data show that MobiAir reduces sensing errors by 96.7% with only 44.9ms latency, outperforming the SOTA baseline by 39.5%.

Multi-UAV systems have shown immense potential in handling complex tasks in large-scale, dynamic, and cold-start (i.e., limited prior knowledge) scenarios, such as wildfire suppression. Due to the dynamic and stochastic environmental conditions, the scheduling for sensing tasks (i.e., fire monitoring) and operation tasks (i.e., fire suppression) should be executed concurrently to enable real-time information collection and timely intervention of the environment. However, the planning inclinations of sensing and operation tasks are typically inconsistent and evolve over time, complicating the task of identifying the optimal strategy for each UAV. To solve this problem, this paper proposes SOScheduler, a collaborative multi-UAV scheduling framework for integrated sensing and operation in large-scale and dynamic wildfire environments. We introduce a spatio-temporal confidence-aware assessment model to dynamically and directly pinpoint locations that can optimally enhance the understanding of environmental dynamics and operational effectiveness, as well as a priority graph-instructed scalable scheduler to coordinate multi-UAV in an efficient manner. Experiments on real multi-UAV testbeds and large-scale physical feature-based simulations show that our SOScheduler reduces the fire expansion ratio by 59% and enhances the fire coverage ratio by 190% compared to state-of-the-art (SOTA) solutions.

Environment depth estimation by fusing camera and radar enables a broad spectrum of applications such as autonomous driving, environmental perception, context-aware localization and navigation. Various pioneering approaches have been proposed to achieve accurate and dense depth estimation by integrating vision and LiDAR through deep learning. However, due to the challenges of sparse sampling of in-vehicle LiDARs, high ground-truth annotation overhead, and severe dynamics in real environments, existing solutions have not yet achieved widespread deployment on commercial autonomous vehicles. In this paper, we propose LeoVR, a motion-inspired self-supervised visual-LiDAR fusion approach that enables accurate environment depth estimation. Leveraging the vehicle motion information, LeoVR employs two effective system frameworks to$(i)$optimize the depth estimation results, and$(ii)$provide supervision signals for DNN training. We fully implemented LeoVR on both a robotic testbed and a commercial vehicle and conducted extensive experiments over an 8-month period. The results demonstrate that LeoVR achieves remarkable performance with an average depth estimation error of 0.17$m$, outperforming existing state-of-the-art solutions by$\gt $45.9%. Besides, even cold-start in real environments by self-supervised training, LeoVR still achieves an average error of 0.2$m$, outperforming the related works by$\gt $47.8% and comparable to supervised training methods.

Edge-assisted visual SLAM plays a crucial role in enabling innovative mobile applications, such as autonomous swarm inspection, search-and-rescue, and smart logistics. Constrained by the computational capacities of lightweight mobile devices, current approaches delegate lightweight, time-sensitive tracking tasks to the mobile end while offloading resource-intensive, latency-tolerant map optimization tasks to the edge. However, our pilot study reveals several limitations of the tracking-optimization decoupled paradigm, stemming from the disruption of inter-dependencies between the two tasks. In this paper, we design and implement edgeSLAM2, an innovative system that reshapes the edge-assisted visual SLAM paradigm by tightly integrating tracking and partial-yet-crucial optimization on mobile. edgeSLAM2 harnesses the heterogeneous computing units offered by the commercial systems-on-chip (SoCs) to enhance the computational capacity of mobile devices, which in turn, allows edgeSLAM2 to design a suit of novel algorithms for map sync, optimization, and tracking that accommodate such architectural upgrade. By capitalizing on the full potential of on-chip intelligence, edgeSLAM2 supports both solitary and collaborative SLAM with accuracy and immediacy, underpinned by a cohesive software-hardware co-design. We deploy edgeSLAM2 on drones for industrial inspection. Comprehensive experiments in one of the world’s largest oil fields over three months demonstrate its superior performance.

The edge-based multi-agent visual SLAM is crucial for emerging mobile applications like search-and-rescue, inventory automation, and industrial inspection. It uses a central node to manage the global map and schedule tasks for agents. However, as the number of agents increases, the system faces scalability challenges due to operational overhead, such as data redundancy, bandwidth consumption, and localization errors. In this paper, we introduce, a framework designed to enhance the scalability of collaborative visual SLAM service in edge offloading settings. consists of three system modules: a change log-based server-client synchronization mechanism, a priority-aware task scheduler, and a lean global map representation. These modules work together to address the challenges of data explosion problems. is open-source and compatible with the robotic operating system (ROS). Existing visual SLAM applications could incorporate through SwarmAPI, a set of well-packaged APIs, to compose SwarmMap’s function modules to enhance their performance and capacity in multi-agent scenarios. Comprehensive evaluations and a three-month case study at one of the world’s largest oilfields demonstrate that can serve 2$\times$more agents ($>$20 agents) than the state-of-the-arts with the same resource overhead, meanwhile maintaining an average trajectory error of 38$cm$, outperforming existing works by$>$55%.

Drones have witnessed extensive popularity among diverse smart applications, and visual Simultaneous Localization and Mapping (SLAM) technology is commonly used to estimate the six-degrees-of-freedom pose for drone flight control systems. However, traditional image-based SLAM cannot ensure the flight safety of drones, especially in challenging environments such as high-speed flight and high dynamic range scenarios. The event camera, a new vision sensor, holds the potential to enable drones to overcome these challenging scenarios if fused with the image-based SLAM. Unfortunately, the computational demands of event-image fusion SLAM have grown manifold compared with image-based SLAM. Existing research on visual SLAM acceleration cannot achieve real-time operation of event-image fusion SLAM on on-board computing platforms for drones. To fill this gap, we present TrinitySLAM , a high-accuracy, real-time, low-energy consumption event-image fusion SLAM acceleration framework utilizing Xilinx Zynq, an on-board heterogeneous computing platform. The key innovations of TrinitySLAM include a fine-grained computation allocation strategy, several novel hardware–software co-acceleration designs, and an efficient data exchange mechanism. We fully implement TrinitySLAM on the latest Zynq UltraScale+ platform and evaluate its performance on one custom-made drone dataset and four official datasets covering various scenarios. Comprehensive experiments show that TrinitySLAM improves the pose estimation accuracy by 28% with half end-to-end latency and 1.2× energy consumption reduction compared with the most comparable state-of-the-art heterogeneous computing platform acceleration baseline.

Among numerous indoor localization systems, WiFi fingerprint-based localization has been one of the most attractive solutions, which is known to be free of extra infrastructure and specialized hardware. To push forward this approach for wide deployment, three crucial goals on high deployment ubiquity, high localization accuracy, and low maintenance cost are desirable. However, due to severe challenges about signal variation, device heterogeneity, and database degradation root in environmental dynamics, pioneer works usually make a trade-off among them. In this article, we propose iToLoc, a deep learning-based localization system that achieves all three goals simultaneously. Once trained, iToLoc will provide accurate localization service for everyone using different devices and under diverse network conditions, and automatically update itself to maintain reliable performance anytime. iToLoc is purely based on WiFi fingerprints without relying on specific infrastructures. The core components of iToLoc are a domain adversarial neural network and a co-training-based semi-supervised learning framework. Extensive experiments across 7 months with eight different devices demonstrate that iToLoc achieves remarkable performance with an accuracy of 1.92 m and >95% localization success rate. Even 7 months after the original fingerprint database was established, the rate still maintains >90%, which significantly outperforms previous works.

Industrial Identifiers (IID), such as GS1, Handle, and OID are fundamental to device identification in growing industrial networks. Appropriate resolution and addressing methods for those identifiers are designed for wide area networks (WAN). However, due to compatibility, efficiency, and security considerations, they are not suitable for local area networks (LANs) environments. Therefore, we propose a new industrial identification framework to handle LAN scenarios by industrial address. It mainly includes the Industrial Identifier Resolution Protocol (IIRP), which combines the IIRP table lookup, the IIRP request, and the response based on the data link layer frame transmission. It is implemented as a software plug-in on LAN devices without changing any network protocol or hardware, ensuring compatibility with existing network infrastructure. Our experiments also test the efficiency of the IIRP protocol.

Location awareness in environments is one of the key parts for drones’ applications and have been explored through various visual sensors. However, standard cameras easily suffer from motion blur under high moving speeds and low-quality image under poor illumination, which brings challenges for drones to perform motion tracking. Recently, a kind of bio-inspired sensors called event cameras emerge, offering advantages like high temporal resolution, high dynamic range and low latency, which motivate us to explore their potential to perform motion tracking in limited scenarios. In this paper, we propose FlyTracker, aiming at developing visual sensing ability for drones of both individual and circumambient location-relevant contextual, by using a monocular event camera. In FlyTracker, background-subtraction-based method is proposed to distinguish moving objects from background and fusion-based photometric features are carefully designed to obtain motion information. Through multilevel fusion of events and images, which are heterogeneous visual data, FlyTracker can effectively and reliably track the 6-DoF pose of the drone as well as monitor relative positions of moving obstacles. We evaluate performance of FlyTracker in different environments and the results show that FlyTracker is more accurate than the state-of-the-art baselines.

Fast and accurate obstacle avoidance is crucial to drone safety. Yet existing on-board sensor modules such as frame cameras and radars are ill-suited for doing so due to their low temporal resolution or limited field of view. This paper presents BioDrone, a new design paradigm for drone obstacle avoidance using stereo event cameras. At the heart of BioDrone is two simple yet effective system design inspired by the mammalian visual system, namely, a chiasm-inspired signal processing pipeline for fast event filtering and obstacle detection, and a lateral geniculate nucleus (LGN)-inspired event matching algorithm for accurate obstacle localization. To make BioDrone a practical solution, we further take significant engineering efforts to deploy the software stack on FPGA through software and hardware co-design. The performance comparison with two state-of-the-art event-based obstacle avoidance systems shows BioDrone achieves a consistently high obstacle detection rate of 96.1%. The average localization error of BioDrone is 6.8cm with a 4.7ms latency, outperforming both baselines by over 40%.

Rapid advances in programmable network devices catalyzed the development of in-network computing, which is foreseen as a key enabler to empower the intelligence of production lines and mechanical arms in Industry 4.0. Various pioneering approaches have demonstrated the significant benefits of moving simple yet delay-sensitive industrial control tasks performed by servers to network switches. However, our detailed field study at a top-tier auto glass factory reveals that current practice fails to achieve a real-time and deterministic intelligent decision closure as leaving those complex yet essential planning tasks still on edge or cloud. In this paper, we design and implement a brand-new industrial switch, named Netopia, on a commercial Zynq platform through software and hardware co-design. Netopia enables planning and control to simultaneously perform on a network switch during communication. At the core of Netopia are three simple yet effective modules - a determinism guarantee mechanism, a computing acceleration scheme, and a packet deterministic forwarding framework that work hand-in-hand to ensure mechanical arms obtain intelligent control commands with low and deterministic latency. Comprehensive evaluations in industrial environments demonstrate that Netopia achieves an average end-to-end intelligent decision latency of 3.0ms with a jitter < 0.4ms, reduced by > 86% over existing works.

Localization and navigation play a key role in many location-based services and have attracted numerous research efforts. In recent years, visual SLAM has been prevailing for autonomous driving. However, the ever-growing computation resources demanded by SLAM impede its applications to resource-constrained mobile devices. In this paper, we present the design, implementation, and evaluation ofedgeSLAM, an edge-assisted real-time semantic visual SLAM service running on mobile devices.edgeSLAMleverages the state-of-the-art semantic segmentation algorithm to enhance localization and mapping accuracy, and speeds up the computation-intensive SLAM and semantic segmentation algorithms by computation offloading. The key innovations ofedgeSLAMinclude an efficient computation offloading strategy, an opportunistic data sharing method, an adaptive task scheduling algorithm, and a multi-user support mechanism. We fully implementedgeSLAMand plan to open-source it. Extensive experiments are conducted under 3 datasets. The results show thatedgeSLAMcan run on mobile devices at 35fps and achieve 5cm localization accuracy from real-world experiments, outperforming existing solutions by more than 15%. We also demonstrate the usability ofedgeSLAMthrough 2 case studies of pedestrian localization and robot navigation. To the best of our knowledge,edgeSLAMis the first edge-assisted real-time semantic visual SLAM for mobile devices.

Indoor navigation is essential to a wide spectrum of applications in the era of mobile computing. Existing vision-based technologies suffer from both start-up costs and the absence of semantic information for navigation. We observe an opportunity to leverage pervasively deployed surveillance cameras to deal with the above drawbacks and revisit the problem of indoor navigation with a fresh perspective. In this paper, we proposeiSAT, a system that enables public surveillance cameras, as indoor navigating satellites, to locate users on the floorplan, tell users with semantic information about the surrounding environment, and guide users with navigation instructions. However, enabling public cameras to navigate is non-trivial due to 3 factors: absence of real scale, disparity of camera perspective, and lack of semantic information. To overcome these challenges,iSATleverages POI-assisted framework and adopts a novel coordinate transformation algorithm to associate public and mobile cameras, and further attaches semantic information to user location. Extensive experiments in 4 different scenarios show thatiSATachieves a localization accuracy of 0.48m and a navigation success rate of 90.5 percent, outperforming the state-of-th-art systems by$> 30\%$. Benefiting from our solution, all areas with public cameras can upgrade to smart spaces with visual navigation services.

Accurate and real-time instance segmentation on mobile devices enables a wide spectrum of applications such as augmented reality, context-aware inspection and environ-mental cognition. However, the computation resource demanded by instance segmentation impedes its deployment on resource-constrained commercial mobile devices. Prior studies enable smartphones to conduct computational-intensive tasks in real-time with the assistance of an edge server. However, simply applying an edge-assisted framework hardly achieves delightful segmentation performance due to the movements of devices and targets, pixel-level precision requirements, and huge computational overhead even for edge nodes. This work proposes edgeIS, an edge-assisted system that enables real-time and accurate instance segmentation on mobile devices. edgeIS embraces the mobile device sensing ability of surroundings and its own motion, and redesigns an innovative mobile-edge collaboration paradigm suitable for segmentation tasks. We implement edgeIS on a lightweight edge node and different mobile devices. Extensive experiments are conducted under four datasets. The results show that edgeIS can run on mobile devices in real-time and achieve a 0.92 segmentation IoU, outperforming existing state-of-the-art solutions. We further embed edgeIS in an AR-based inspection system deployed in an oil field and the performance of edgeIS meets the demand of the industrial scenario.

After years of boom, drones and their applications are now entering indoors. Six-degree-of-freedom (6-DoF) pose tracking is the core of drone flight control, but existing solutions cannot be directly applied to indoor scenarios due to insufficient accuracy, low robustness to adverse texture and light conditions, and signal obstruction in indoor scenarios. To overcome the above limitations, we propose Wi-Drone, a Wi-Fi standalone 6-DoF tracking system for indoor drone flight control. Wi-Drone takes full advantage of both exte-roceptive and proprioceptive measurements of Wi-Fi to estimate the drone's absolute pose and relative motion, and fuse them in a tight-coupling manner to achieve their complementary benefits. We implement Wi-Drone and integrate it into a flight control system. The evaluation results show that Wi-Drone achieves a real-time performance with the average location accuracy of 26.1 cm and the rotation accuracy of 3.8°, which demonstrates its competency of flight control, compared to visual-inertial-based flight control. Such results also outperform existing Wi-Fi-based tracking solutions in terms of both dimensionality and accuracy.

Environment depth estimation by fusing camera and radar enables a broad spectrum of applications such as autonomous driving, environmental perception, context-aware localization and navigation. Various pioneering approaches have been proposed to achieve accurate and dense depth estimation by integrating vision and LiDAR through deep learning. However, due to the challenges of sparse sampling of in-vehicle LiDARs, high ground-truth annotation overhead, and severe dynamics in real environments, existing solutions have not yet achieved widespread deployment on commercial autonomous vehicles. In this paper, we propose LeoVR, a visual-LiDAR fusion based self-supervised approach that enables accurate environment depth estimation. LeoVR digs into the vehicle's motion information and designs two effective system frameworks based on it to (i) optimize the depth estimation results, and (ii) provide supervision signals to train a DNN. We fully implement LeoVR on a robotic testbed and commercial vehicle to conduct extensive experiments across 6 months. The results demonstrate that LeoVR achieves remarkable performance with an average depth estimation error of 0.17m, outperforming existing state-of-the-art solutions by > 43%. Besides, even cold-start in real environments by self-supervised training, LeoVR still achieves an average error of 0.21m, outperforming the related works by > 45% and comparable to those supervised training methods.

Indoor localization has gained increasing attention in the era of the Internet of Things. Among various technologies, WiFi fingerprint-based localization has become a mainstream solution. However, RSS fingerprints suffer from critical drawbacks of spatial ambiguity and temporal instability that root in multipath effects and environmental dynamics, which degrade the performance of these systems and therefore impede their wide deployment in the real world. Pioneering works overcome these limitations at the costs of ubiquity as they mostly resort to additional information or extra user constraints. In this article, we present the design and implementation of ViViPlus, an indoor localization system purely based on WiFi fingerprints, which jointly mitigates spatial ambiguity and temporal instability and derives reliable performance without impairing the ubiquity. The key idea is to embrace the spatial awareness of RSS values in a novel form of RSS Spatial Gradient (RSG) matrix for enhanced WiFi fingerprints. We devise techniques for the representation, construction, and localization of the proposed fingerprint form and integrate them all in a practical system. Extensive experiments across 7 months in different environments demonstrate that ViViPlus significantly improves the accuracy in localization scenarios by about 30% to 50% compared with the state-of-the-art approaches.

Indoor localization has attracted increasingly attention in the era of Internet of Things. Single indoor localization method based on WiFi fingerprint, surveillance camera or pedestrian dead reckoning suffers from low accuracy, limited tracking region or accumulative errors. Pioneering works over-come these limitations at the costs of ubiquity as they mostly resort to additional information or extra user constraints. In the large indoor region, it is important to quickly get pedestrian detection and tracking. In this paper, an indoor localization and tracking system has been presented which integrates WiFi fingerprint, Vision of surveillance camera and Pedestrian Dead Reckoning(WVP system for short). This WVP system achieves high accuracy in dynamic indoor environment. Importantly, WVP employs a motion sequence-based matching algorithm to confirm pedestrian identity. WVP outputs enhanced accuracy and overcomes the corresponding drawbacks of each subsystem simultaneously. Experimental results show that WVP can effectively track pedestrians in multi-region, and has great robustness, and the positioning accuracy is decimeter. It also performs well in complex environment.

Among numerous indoor localization systems, WiFi fingerprint-based localization has been one of the most attractive solutions, which is known to be free of extra infrastructure and specialized hardware. To push forward this approach for wide deployment, three crucial goals on delightful deployment ubiquity, high localization accuracy, and low maintenance cost are desirable. However, due to severe challenges about signal variation, device heterogeneity, and database degradation root in environmental dynamics, pioneer works usually make a trade-off among them. In this paper, we propose iToLoc, a deep learning based localization system that achieves all three goals simultaneously. Once trained, iToLoc will provide accurate localization service for everyone using different devices and under diverse network conditions, and automatically update itself to maintain reliable performance anytime. iToLoc is purely based on WiFi fingerprints without relying on specific infrastructures. The core components of iToLoc are a domain adversarial neural network and a co-training based semi-supervised learning framework. Extensive experiments across 7 months with 8 different devices demonstrate that iToLoc achieves remarkable performance with an accuracy of 1.92m and > 95% localization success rate. Even 7 months after the original fingerprint database was established, the rate still maintains > 90%, which significantly outperforms previous works.

Existing smartphone-based Augmented Reality (AR) systems are able to render virtual effects on static anchors. However, today's solutions lack the ability to render follow-up effects attached to moving anchors since they fail to track the 6 degrees of freedom (6-DoF) poses of them. We find an opportunity to accomplish the task by leveraging sensors capable of generating sparse point clouds on smartphones and fusing them with vision-based technologies. However, realizing this vision is non-trivial due to challenges in modeling radar error distributions and fusing heterogeneous sensor data. This study proposes FollowUpAR, a framework that integrates vision and sparse measurements to track object 6-DoF pose on smartphones. We derive a physical-level theoretical radar error distribution model based on an in-depth understanding of its hardware-level working principles and design a novel factor graph competent in fusing heterogeneous data. By doing so, FollowUpAR enables mobile devices to track anchor's pose accurately. We implement FollowUpAR on commodity smartphones and validate its performance with 800,000 frames in a total duration of 15 hours. The results show that FollowUpAR achieves a remarkable rotation tracking accuracy of 2.3° with a translation accuracy of 2.9mm, outperforming most existing tracking systems and comparable to state-of-the-art learning-based solutions. FollowUpAR can be integrated into ARCore and enable smartphones to render follow-up AR effects to moving objects.

Smartphone localization is essential to a wide spectrum of applications in the era of mobile computing. The ubiquity of smartphone mobile cameras and surveillance ambient cameras holds promise for offering sub-meter accuracy localization services thanks to the maturity of computer vision techniques. In general, ambient-camera-based solutions are able to localize pedestrians in video frames at fine-grained, but the tracking performance under dynamic environments remains unreliable. On the contrary, mobile-camera-based solutions are capable of continuously tracking pedestrians; however, they usually involve constructing a large volume of image database, a labor-intensive overhead for practical deployment. We observe an opportunity of integrating these two most promising approaches to overcome above limitations and revisit the problem of smartphone localization with a fresh perspective. However, fusing mobile-camera-based and ambient-camera-based systems is non-trivial due to disparity of camera in terms of perspectives, parameters and incorrespondence of localization results. In this article, we propose iMAC, an integrated mobile cameras and ambient cameras based localization system that achieves sub-meter accuracy and enhanced robustness with zero-human start-up effort. The key innovation of iMAC is a well-designed fusing frame to eliminate disparity of cameras including a construction of projection map function to automatically calibrate ambient cameras, an instant crowd fingerprints model to describe user motion patterns, and a confidence-aware matching algorithm to associate results from two sub-systems. We fully implement iMAC on commodity smartphones and validate its performance in five different scenarios. The results show that iMAC achieves a remarkable localization accuracy of 0.68 m, outperforming the state-of-the-art systems by >75%.

Existing indoor navigation solutions usually require pre-deployed comprehensive location services with precise indoor maps and, more importantly, all rely on dedicatedly installed or existing infrastructure. In this article, we present Pair-Navi, an infrastructure-free indoor navigation system that circumvents all these requirements by reusing a previous traveler’s (i.e., leader) trace experience to navigate future users (i.e., followers) in a Peer-to-Peer mode. Our system leverages the advances of visual simultaneous localization and mapping ( SLAM ) on commercial smartphones. Visual SLAM systems, however, are vulnerable to environmental dynamics in the precision and robustness and involve intensive computation that prohibits real-time applications. To combat environmental changes, we propose to cull non-rigid contexts and keep only the static and rigid contents in use. To enable real-time navigation on mobiles, we decouple and reorganize the highly coupled SLAM modules for leaders and followers. We implement Pair-Navi on commodity smartphones and validate its performance in three diverse buildings and two standard datasets (TUM and KITTI). Our results show that Pair-Navi achieves an immediate navigation success rate of 98.6%, which maintains as 83.4% even after 2 weeks since the leaders’ traces were collected, outperforming the state-of-the-art solutions by >50%. Being truly infrastructure-free, Pair-Navi sheds lights on practical indoor navigations for mobile users.

Smartphone localization is essential to a wide spectrum of applications in the era of mobile computing. The ubiquity of smartphone mobile cameras and surveillance ambient cameras holds promise for offering sub-meter accuracy localization services thanks to the maturity of computer vision techniques. In general, ambient-camera-based solutions are able to localize pedestrians in video frames at fine-grained, but the tracking performance under dynamic environments remains unreliable. On the contrary, mobile-camera-based solutions are capable of continuously tracking pedestrians, however, they usually involve constructing a large volume of image database, a labor-intensive overhead for practical deployment. We observe an opportunity of integrating these two most promising approaches to overcome above limitations and revisit the problem of smartphone localization with a fresh perspective. However, fusing mobile-camera-based and ambient-camera-based systems is non-trivial due to disparity of camera in terms of perspectives, parameters and incorrespondence of localization results. In this paper, we propose iMAC, an integrated mobile cameras and ambient cameras based localization system that achieves sub-meter accuracy and enhanced robustness with zero-human start-up effort. The key innovation of iMAC is a well-designed fusing frame to eliminate disparity of cameras including a construction of projection map function to automatically calibrate ambient cameras, an instant crowd fingerprints model to describe user motion patterns, and a confidence-aware matching algorithm to associate results from two sub-systems. We fully implement iMAC on commodity smart-phones and validate its performance in five different scenarios. The results show that iMAC achieves a remarkable localization accuracy of 0.68m, outperforming the state-of-the-art systems by > 75%.

Visual peer-to-peer navigation is a suitable solution for indoor navigation for it relieves the labor of site-survey and eliminates infrastructure dependence. However, a major drawback hampers its application, as the peer-to-peer mode suffers from a deficiency of paths in large indoor scenarios with multifarious places-of-interest. Nevertheless, we propose one with a profound crowdsourcing scheme that addresses the drawback by merging the paths of different leaders' into a global map. To realize the idea, we further deal with entailed challenges, namely the unidirectional disadvantage, the scale ambiguity, and large computational overhead. We design a navigation strategy to solve the unidirectional problem and turn to VIO to tackle scale ambiguity. We devise a mobile-edge architecture to enable real-time navigation (30fps, 100ms end-to-end delay) and lighten the burden of smartphones (35% battery life for 2h35min) while assuring the accuracy of localization and map construction. Through experimental validations, we show that P2P navigation, previously relying on the abundance of independent paths, can enjoy a sufficiency of navigation paths with a crowdsourced global map. The experiments demonstrate a navigation success rate of 100% and spatial offset of less than 3.2m, better than existing works.

Localization and navigation play a key role in many location-based services and have attracted numerous research efforts from both academic and industrial community. In recent years, visual SLAM has been prevailing for robots and autonomous driving cars. However, the ever-growing computation resource demanded by SLAM impedes its application to resource-constrained mobile devices. In this paper we present the design, implementation, and evaluation of edgeSLAM, an edge assisted real-time semantic visual SLAM service running on mobile devices. edgeSLAM leverages the state-of-the-art semantic segmentation algorithm to enhance localization and mapping accuracy, and speeds up the computation-intensive SLAM and semantic segmentation algorithms by computation offloading. The key innovations of edgeSLAM include an efficient computation offloading strategy, an opportunistic data sharing mechanism, and an adaptive task scheduling algorithm. We fully implement edgeSLAM on an edge server and different types of mobile devices (2 types of smartphones and a development board). Extensive experiments are conducted under 3 data sets, and the results show that edgeSLAM is able to run on mobile devices at 35fps frame rate and achieves a 5cm localization accuracy, outperforming existing solutions by more than 15%. We also demonstrate the usability of edgeSLAM through 2 case studies of pedestrian localization and robot navigation. To the best of our knowledge, edgeSLAM is the first real-time semantic visual SLAM for mobile devices.

Existing indoor navigation solutions usually require pre-deployed comprehensive location services with precise indoor maps and, more importantly, all rely on dedicatedly installed or existed infrastructure. In this paper, we present Pair-Navi, an infrastructure-free indoor navigation system that circumvents all these requirements by reusing a previous traveler's (i.e. leader) trace experience to navigate future users (i.e. followers) in a Peer-to-Peer (P2P) mode. Our system leverages the advances of visual SLAM on commercial smartphones. Visual SLAM systems, however, are vulnerable to environmental dynamics in the precision and robustness and involve intensive computation that prohibits real-time applications. To combat environmental changes, we propose to cull non-rigid contexts and keep only the static and rigid contents in use. To enable real-time navigation on mobiles, we decouple and reorganize the highly coupled SLAM modules for leaders and followers. We implement Pair-Navi on commodity smartphones and validate its performance in three diverse buildings. Our results show that Pair-Navi achieves an immediate navigation success rate of 98.6%, which maintains as 83.4% even after two weeks since the leaders' traces were collected, outperforming the state-of-the-art solutions by >50%. Being truly infrastructure-free, Pair-Navi sheds lights on practical indoor navigations for mobile users.

Indoor localization gains increasingly attentions in the era of Internet of Things. Among various technologies, WiFi-based systems that leverage Received Signal Strengths (RSSs) as location fingerprints become the mainstream solutions. However, RSS fingerprints suffer from critical drawbacks of spatial ambiguity and temporal instability that root in multipath effects and environmental dynamics, which degrade the performance of these systems and therefore impede their wide deployment in real world. Pioneering works overcome these limitations at the costs of ubiquity as they mostly resort to additional information or extra user constraints. In this paper, we present the design and implementation of MatLoc, an indoor localization system purely based on WiFi fingerprints, which jointly mitigates spatial ambiguity and temporal instability and derives reliable performance without impairing the ubiquity. The key idea is to embrace the spatial awareness of RSS values in a novel form of RSS Spatial Gradient (RSG) matrix for enhanced WiFi fingerprints. We devise techniques for the representation, construction, and comparison of the proposed fingerprint form, and integrate them all in a practical system, which follows the classical fingerprinting framework and requires no more inputs than any previous RSS fingerprint based systems. Extensive experiments in different environments demonstrate that MatLoc significantly improves the accuracy in both localization and tracking scenarios by about 30% to 50% compared with five state-of-the-art approaches.