We address the challenge of multi-LiDAR interference, an issue of growing importance as LiDAR sensors are embedded in a growing set of pervasive devices. We introduce a novel approach named D2SR, enabling decentralized interference detection, mitigation, and recovery without explicit coordination among nearby LiDAR devices. D2SR comprises three stages: (a) Detection, which identifies interfered frames, (b) Mitigation, which performs time-shifting of a LiDAR’s active period to reduce interference, and (c) Recovery, which corrects or reconstructs the depth values in interfered regions of a depth frame. Key contributions include a lightweight interference detection algorithm achieving an F1-score of 92%, a simple yet effective decentralized de-synchronization mechanism, and a lightweight depth recovery pipeline that preserves high throughput processing on edge devices. Evaluation on Nvidia Jetson devices demonstrates D2SR’s efficacy: under static settings, D2SR accurately detects interference in 93% of cases (recall=82%) and reduces the depth estimation error by 27% (RMSE= 38.7 cm, compared to RMSE= 60.6 cm for a baseline without D2SR). Furthermore, D2SR is able to reduce the fraction of interfered frames by 75.1% and reduce the depth estimation error (for interfered frames) by 24.9% even for a moving robot scenario.

We present LiLoc , a system for precise 3D localization and tracking of mobile IoT devices (e.g., robots) in indoor environments using multi-perspective LiDAR sensing. LiLoc stands out with two key differentiators. First, unlike traditional localization approaches, our method remains robust in dynamically changing environments, adeptly handling varying crowd levels and object layout changes. Second, LiLoc is independent of pre-built static maps, employing dynamically updated point clouds from infrastructural-mounted LiDARs and LiDARs on individual IoT devices. For fine-grained, near real-time tracking, LiLoc intermittently utilizes complex 3D “global” registration between point clouds for robust spot location estimates. It further complements this with simpler “local” registrations, continuously updating IoT device trajectories. We demonstrate that LiLoc can (a) support accurate location tracking with location and pose estimation error being ≦7.4 cm and ≦3.2°, respectively, for 84% of the time and the median error increasing only marginally (8%), for correctly estimated trajectories, when the ambient environment is dynamic; (b) achieve a 36% reduction in median location estimation error compared to an approach that uses only quasi-static global point cloud; and (c) obtain spot location estimates with a latency of only 973 msec. We also demonstrate how LiLoc efficiently integrates low-power inertial sensing, using a novel integration of inertial-based displacement to accelerate the local registration process, to enhance localization energy efficiency and latency.

This work demonstrates the VGGlass system, which simultaneously interprets human instructions for a target acquisition task and determines the precise 3D positions of both user and the target object. This is achieved by utilizing LiDARs mounted in the infrastructure and a smart glass device worn by the user. Key to our system is the union of LiDAR-based localization termed LiLOC and a multi-modal visual grounding approach termed RealG(2)In-Lite. To demonstrate the system, we use Intel RealSense L515 cameras and a Microsoft HoloLens 2, as the user devices. VGGlass is able to: a) track the user in real-time in a global coordinate system, and b) locate target objects referred by natural language and pointing gestures.

Natural human interactions for Mixed Reality Applications are overwhelmingly multimodal: humans communicate intent and instructions via a combination of visual, aural and gestural cues. However, supporting low-latency and accurate comprehension of such multimodal instructions (MMI), on resource-constrained wearable devices, remains an open challenge, especially as the state-of-the-art comprehension techniques for each individual modality increasingly utilize complex Deep Neural Network models. We demonstrate the possibility of overcoming the core limitation of latency-vs.-accuracy tradeoff by exploiting cross-modal dependencies-i.e., by compensating for the inferior performance of one model with an increased accuracy of more complex model of a different modality. We present a sensor fusion architecture that performs MMI comprehension in a quasi-synchronous fashion, by fusing visual, speech and gestural input. The architecture is reconfigurable and supports dynamic modification of the complexity of the data processing pipeline for each individual modality in response to contextual changes. Using a representative “classroom” context and a set of four common interaction primitives, we then demonstrate how the choices between low and high complexity models for each individual modality are coupled. In particular, we show that (a) a judicious combination of low and high complexity models across modalities can offer a dramatic 3-fold decrease in comprehension latency together with an increase ~10-15% in accuracy, and (b) the right collective choice of models is context dependent, with the performance of some model combinations being significantly more sensitive to changes in scene context or choice of interaction.

We present ERICA, a digital personal trainer for users performing free weights exercises, with two key differentiators: (a) First, unlike prior approaches that either require multiple on-body wearables or specialized infrastructural sensing, ERICA uses a single in-ear "earable" device (piggybacking on a form factor routinely used by millions of gym-goers) and a simple inertial sensor mounted on each weight equipment; (b) Second, unlike prior work that focuses primarily on quantifying a workout, ERICA additionally identifies a variety of fine-grained exercising mistakes and delivers real-time, in-situ corrective instructions. To achieve this, we (a) design a robust approach for user-equipment association that can handle multiple (even 15) concurrently exercising users; (b) develop a suite of statistical models to detect several commonplace repetition-level mistakes; and (c) experimentally study the efficacy of multiple in-situ corrective feedback strategies. Via an end-to-end evaluation of ERICA with 33 participants naturally performing 3 dumbbell-based exercises, we show that (a) ERICA identifies over 94% of mistakes during the first 5 repetitions of a set, (b) the resulting feedback is viewed favorably by 78% of users, and (c) the feedback is effective, reducing mistakes by 10+% during subsequent repetitions.