Early mobilization is a structured protocol designed to facilitate motor recovery in intensive care unit (ICU) patients with ICU-acquired weakness. This process is typically implemented by an interdisciplinary team of nurses, physical therapists, and other healthcare professionals. However, its application is often constrained by the patients' critical conditions, limited mobility, and the challenges of coordinating care within resource-intensive ICU environments. In this study, we developed a patient-centered virtual reality (VR) exergame through an interdisciplinary design process involving clinicians and therapists, tailored to the constraints of critical care. The exergame incorporates progressive mobility levels that mirror early mobilization practices, and includes an embodied avatar to provide guidance and motivation. Using Meta Quest 3 body tracking, the system captures and visualizes patients' movements, thereby providing motivational engagement and quantifiable mobility metrics. We evaluated the exergame in two stages: a dual-user study involving healthy participants and healthcare professionals or students (N = 13), and a subsequent study with cardiothoracic ICU patients (N = 18) to assess feasibility, design validity, and clinical acceptance. Across both studies, participants reported high enjoyment and engagement without discomfort or stress. Furthermore, patients demonstrated increases in movement speed, range of motion, and workspace volume of the upper body across game levels. Physiological monitoring further indicated that the exergame elicited exertion without inducing excessive cardiovascular responses. These findings highlight the feasibility of VR exergames as a clinically acceptable and engaging adjunct to early mobilization in critical care, offering a novel pathway to improve rehabilitation outcomes for ICU patients.

Inaccurate spatial tracking in extended reality (XR) devices leads to virtual object jitter, misalignment, and user discomfort, fundamentally limiting immersive experiences and natural interactions. In this work, we introduce a novel testbed that enables simultaneous, synchronized evaluation of multiple XR devices under identical environmental and kinematic conditions. Leveraging this platform, we present the first comprehensive empirical benchmarking of five state-of-the-art XR devices across 16 diverse scenarios. Our results reveal substantial intra-device performance variation, with individual devices exhibiting up to 101% increases in error when operating in featureless environments. We also demonstrate that tracking accuracy strongly correlates with visual conditions and motion dynamics. We also observe significant inter-device disparities, with performance differences of up to$2.8 \times$, which are closely linked to hardware specifications such as sensor configurations and dedicated processing units. Finally, we explore the feasibility of substituting a motion capture system with the Apple Vision Pro as a practical ground truth reference. While the Apple Vision Pro delivers highly accurate relative pose error estimates ($R^{2}=0.830$), its absolute pose error estimation remains limited ($R^{2}=0.387$), highlighting both its potential and its constraints for rigorous XR evaluation. This work establishes the first standardized framework for comparative XR tracking evaluation, providing the research community with reproducible methodologies, comprehensive benchmark datasets, and open-source tools that enable systematic analysis of tracking performance across devices and conditions, thereby accelerating the development of more robust spatial sensing technologies for XR systems.

Augmented Reality (AR) systems, while enhancing task performance through real-time guidance, pose risks of inducing cognitive tunneling—a hyperfocus on virtual content that compromises situational awareness (SA) in safety-critical scenarios. This paper investigates SA in AR-guided cardiopulmonary resuscitation (CPR), where responders must balance effective compressions with vigilance to unpredictable hazards (e.g., patient vomiting). We developed an AR app on a Magic Leap 2 that overlays real-time CPR feedback (compression depth and rate) and conducted a user study with simulated unexpected incidents (e.g., bleeding) to evaluate SA, in which SA metrics were collected via observation and questionnaires administered during freeze-probe events. Eye tracking analysis revealed that higher SA levels were associated with greater saccadic amplitude and velocity, and with reduced proportion and frequency of fixations on virtual content. To predict SA, we propose FixGraphPool, a graph neural network that structures gaze events (fixations, saccades) into spatiotemporal graphs, effectively capturing dynamic attentional patterns. Our model achieved 83.0 % accuracy ($\mathrm{F} 1=81.0 \%$), outperforming feature-based machine learning and state-of-the-art time-series models by leveraging domain knowledge and spatial-temporal information encoded in ET data. These findings demonstrate the potential of eye tracking for SA modeling in AR and highlight its utility in designing AR systems that ensure user safety and situational awareness.

We present AR-TMT, an AR adaptation of the Trail Making Test designed to evaluate how different types of distractors affect user attention in the AR environment. Built on the Magic Leap 2, AR-TMT captures gaze behavior as users perform a cognitively demanding visual search task under two distraction conditions: bottom-up (visually salient) and top-down (semantically misleading). The system visualizes gaze patterns and performance, revealing distinct attentional responses to the two distraction types. This demo could give insights on cognitive security in XR by offering a empirical findings of visual attention vulnerabilities with regard to distraction types and informing the design of distraction-aware, cognitively resilient AR systems.

Cardiopulmonary Resuscitation (CPR) is a critical procedure where noticing unexpected patient changes is as vital as performing quality compressions. While Augmented Reality (AR) applications have shown promise in guiding CPR, they often focus on metrics such as compression rate and depth, which can negatively impact rescuers by inducing attentional tunneling—a phenomenon where intense focus on virtual elements causes a user to miss critical events in their physical environment. This demonstration presents an AR-guided CPR system to investigate this effect, using a mannequin with embedded LED lights that illuminate randomly during the task. Attendees will experience two variations of our AR application: one with CPR quality feedback in the form of a speed gauge and related text information, and another that adds a dynamic, in-context compression depth visualizer. By recording the reaction time to the LED stimulus and analyzing gaze allocation on virtual elements in our AR app, we will demonstrate how AR can induce attentional tunneling, even with good design intentions.

Augmented reality (AR) enhances user interaction with the real world but also presents vulnerabilities, particularly through Visual Information Manipulation (VIM) attacks. These attacks alter important real-world visual cues, leading to user confusion and misdirected actions. In this demo, we present a hands-on experience using a miniature city setup, where users interact with manipulated AR content via the Meta Quest 3. The demo highlights the impact of VIM attacks on user decision-making and underscores the need for effective security measures in AR systems. Future work includes a user study and cross-platform testing.

Despite the growing use of AR in safety-critical domains, the field lacks a systematic understanding of how different types of distraction affect user behavior in AR environments. To address this gap, we present AR-TMT, an AR adaptation of the Trail Making Test that spatially renders targets for sequential selection on the Magic Leap 2. We implemented distractions in three categories: top-down, bottom-up, and spatial distraction based on Wolfe’s Guided Search model, and captured performance, gaze, motor behavior, and subjective load measures to analyze user attention and behavior. A user study with 34 participants revealed that top-down distraction degraded performance through semantic interference, while bottom-up distraction disrupted initial attentional engagement. Spatial distraction destabilized gaze behavior, leading to more scattered and less structured visual scanning patterns. We also found that performance was correlated with attention control (R2 =.20–.35) under object-based distraction conditions, where distractors possessed task-relevant features. The study offers insights into distraction mechanisms and their impact on users, providing opportunities for generalization to ecologically relevant AR tasks while underscoring the need to address the unique demands of AR environments.

External ventricular drain (EVD) is a common, yet challenging neurosurgical procedure of placing a catheter into the brain ventricular system that requires prolonged training for surgeons to improve the catheter placement accuracy. In this article, we introduce NeuroLens, an Augmented Reality (AR) system that provides neurosurgeons with guidance that aids them in completing an EVD catheter placement. NeuroLens builds on prior work in AR-assisted EVD to present a registered hologram of a patient's ventricles to the surgeons, and uniquely incorporates guidance on the EVD catheter's trajectory, angle of insertion, and distance to the target. The guidance is enabled by tracking the EVD catheter. We evaluate NeuroLens via a study with 33 medical students and 9 neurosurgeons, in which we analyzed participants' EVD catheter insertion accuracy and completion time, eye gaze patterns, and qualitative responses. Our study, in which NeuroLens was used to aid students and surgeons in inserting an EVD catheter into a realistic phantom model of a human head, demonstrated the potential of NeuroLens as a tool that will aid and educate novice neurosurgeons. On average, the use of NeuroLens improved the EVD placement accuracy of the year 1 students by 39.4%, of the year 2$-$-4 students by 45.7%, and of the neurosurgeons by 16.7%. Furthermore, students who focused more on NeuroLens-provided contextual guidance achieved better results, and novice surgeons improved more than the expert surgeons with NeuroLens's assistance.

The virtual content in augmented reality (AR) can introduce misleading or harmful information, leading to semantic misunderstandings or user errors. In this work, we focus on visual information manipulation (VIM) attacks in AR, where virtual content changes the meaning of real-world scenes in subtle but impactful ways. We introduce a taxonomy that categorizes these attacks into three formats: character, phrase, and pattern manipulation, and three purposes: information replacement, information obfuscation, and extra wrong information. Based on the taxonomy, we construct a dataset, AR-VIM, which consists of 452 raw-AR video pairs spanning 202 different scenes, each simulating a real-world AR scenario. To detect the attacks in the dataset, we propose a multimodal semantic reasoning framework, VIM-Sense. It combines the language and visual understanding capabilities of vision-language models (VLMs) with optical character recognition (OCR)-based textual analysis. VIM-Sense achieves an attack detection accuracy of 88.94% on AR-VIM, consistently outperforming vision-only and text-only baselines. The system achieves an average attack detection latency of 7.07 seconds in a simulated video processing framework and 7.17 seconds in a real-world evaluation conducted on a mobile Android AR application.

In Augmented Reality (AR), virtual content enhances user experience by providing additional information. However, improperly positioned or designed virtual content can be detrimental to task performance, as it can impair users' ability to accurately interpret real-world information. In this paper we examine two types of task-detrimental virtual content: obstruction attacks, in which virtual content prevents users from seeing real-world objects, and information manipulation attacks, in which virtual content interferes with users' ability to accurately interpret real-world information. We provide a mathematical framework to characterize these attacks and create a custom open-source dataset for attack evaluation. To address these attacks, we introduce ViDDAR (Vision language model-based Task-Detrimental content Detector for Augmented Reality), a comprehensive full-reference system that leverages Vision Language Models (VLMs) and advanced deep learning techniques to monitor and evaluate virtual content in AR environments, employing a user-edge-cloud architecture to balance performance with low latency. To the best of our knowledge, ViDDAR is the first system to employ VLMs for detecting task-detrimental content in AR settings. Our evaluation results demonstrate that ViDDAR effectively understands complex scenes and detects task-detrimental content, achieving up to 92.15% obstruction detection accuracy with a detection latency of 533 ms, and an 82.46% information manipulation content detection accuracy with a latency of 9.62 s.

3D object detection (OD) is a crucial element in scene understanding. However, most existing 3D OD models have been tailored to work with light detection and ranging (LiDAR) and RGB-D point cloud data, leaving their performance on commonly available visual-inertial simultaneous localization and mapping (VI-SLAM) point clouds unexamined. In this paper, we create and release two datasets: VIP500, 4772 VI-SLAM point clouds covering 500 different object and environment configurations, and VIP500-D, an accompanying set of 20 RGB-D point clouds for the object classes and shapes in VIP500. We then use these datasets to quantify the differences between VI-SLAM point clouds and dense RGB-D point clouds, as well as the discrepancies between VI-SLAM point clouds generated with different object and environment characteristics. Finally, we evaluate the performance of three leading OD models on the diverse data in our VIP500 dataset, revealing the promise of OD models trained on VI-SLAM data; we examine the extent to which both object and environment characteristics impact performance, along with the underlying causes.

Object detection (OD) is crucial for numerous emerging visual sensing applications. As OD models trained on unrepresentative data usually yield poor performance, collecting high-quality data in the local environment is recognized to be essential for improving model accuracy. Yet, the question of how to collect this data is currently largely overlooked; unsupported data collection tends to produce datasets with a significant proportion of redundant or uninformative data, hindering effective model training. To address this challenge, we design a real-time data importance estimation method and integrate it into BiGuide, a bi-level image data acquisition system we create for OD tasks. BiGuide assesses the importance of the captured images in real-time based on informativeness and diversity estimations and dynamically guides users in collecting useful data via image-level and object instance-level guidance. We prototype BiGuide in an edge-based architecture using commodity smartphones as mobile clients, and evaluate its performance via an IRB-approved study with 20 users. Our evaluation demonstrates that OD models trained on the data collected by BiGuide outperform models trained on the data collected by two baseline systems, achieving detection accuracy improvements of up to 33.07% and 14.57%, respectively. Over 85% of the users found BiGuide fast, helpful, and easy to understand and follow.

Virtual reality (VR) simulations have been adopted to provide controllable environments for running augmented reality (AR) experiments in diverse scenarios. However, insufficient research has explored the impact of AR applications on users, especially their attention patterns, and whether VR simulations accurately replicate these effects. In this work, we propose to analyze user attention patterns via eye tracking during XR usage. To represent applications that provide both helpful guidance and irrelevant information, we built a Sudoku Helper app that includes visual hints and potential distractions during the puzzle-solving period. We conducted two user studies with 19 different users each in AR and VR, in which we collected eye tracking data, conducted gaze-based analysis, and trained machine learning (ML) models to predict user attentional states and attention control ability. Our results show that the AR app had a statistically significant impact on enhancing attention by increasing the fixated proportion of time, while the VR app reduced fixated time and made the users less focused. Results indicate that there is a discrepancy between VR simulations and the AR experience. Our ML models achieve 99.3% and 96.3% accuracy in predicting user attention control ability in AR and VR, respectively. A noticeable performance drop when transferring models trained on one medium to the other further highlights the gap between the AR experience and the VR simulation of it.

Inaccurate spatial tracking in extended reality (XR) headsets can cause virtual object jitter, misalignment, and user discomfort, limiting the headsets' potential for immersive content and natural interactions. We develop a modular testbed to evaluate the tracking performance of commercial XR headsets, incorporating system calibration, tracking data acquisition, and result analysis, and allowing the integration of external cameras and IMU sensors for comparison with open-source VI-SLAM algorithms. Using this testbed, we quantitatively assessed spatial tracking accuracy under various user movements and environmental conditions for the latest XR headsets, Apple Vision Pro and Meta Quest 3. The Apple Vision Pro outperformed the Meta Quest 3, reducing relative pose error (RPE) and absolute pose error (APE) by 33.9% and 14.6%, respectively. While both headsets achieved subcentimeter APE in most cases, they exhibited APE exceeding 10 cm in challenging scenarios, highlighting the need for further improvements in reliability and accuracy.

Edge-cloud collaborative inference empowers resource-limited IoT devices to support deep learning applications without disclosing their raw data to the cloud server, thus protecting user's data. Nevertheless, prior research has shown that collaborative inference still results in the exposure of input and predictions from edge devices. To defend against such data leakage in collaborative inference, we introduce InfoScissors, a defense strategy designed to reduce the mutual information between a model's intermediate outcomes and the device's input and predictions. We evaluate our defense on several datasets in the context of diverse attacks. Besides the empirical comparison, we provide a theoretical analysis of the inadequacies of recent defense strategies that also utilize mutual information, particularly focusing on those based on the Variational Information Bottleneck (VIB) approach. We illustrate the superiority of our method and offer a theoretical analysis of it.

In this work, we introduce SEESys, the first system to provide online pose error estimation for Simultaneous Localization and Mapping (SLAM). Unlike prior offline error estimation approaches, the SEESys framework efficiently collects real-time system features and delivers accurate pose error magnitude estimates with low latency. This enables real-time quality-of-service information for downstream applications. To achieve this goal, we develop a SLAM system run-time status monitor (RTS monitor) that performs feature collection with minimal overhead, along with a multi-modality attention-based Deep SLAM Error Estimator (DeepSEE) for error estimation. We train and evaluate SEESys using both public SLAM benchmarks and a diverse set of synthetic datasets, achieving an RMSE of 0.235 cm of pose error estimation, which is 15.8% lower than the baseline. Additionally, we conduct a case study showcasing SEESys in a real-world scenario, where it is applied to a real-time audio error advisory system for human operators of a SLAM-enabled device. The results demonstrate that SEESys provides error estimates with an average end-to-end latency of 37.3 ms, and the audio error advisory reduces pose tracking error by 25%.

In virtual reality (VR), users' headpose, that is, the location and the orientation of users' viewport, determines the view of the virtual world that is shown to the users. The importance of the viewport pose to VR experiences calls for the development of VR viewport pose models. However, no study has obtained a full pose (the position and the orientation) model applicable to modeling the viewport pose in VR experiences. In this paper, informed by our experimental measurements of viewport trajectories across 4 different types of VR interfaces, we first develop a statistical model of viewport poses in VR environments. Based on the developed model, we examine the correlations between pixels in VR frames that correspond to different viewport poses, and obtain an analytical expression for the visibility similarity (ViS) of the pixels across different VR frames. We then propose a lightweight ViS-based algorithm (ALG-ViS) that adaptively splits VR frames into the background and the foreground, reusing the background across different frames. Our implementation of ALG-ViS in two Oculus Quest 2 rendering systems demonstrates ALG-ViS running in real time, supporting the full VR frame rate, and outperforming baselines on measures of frame quality and bandwidth consumption.

Edge computing is increasingly proposed as a solution for reducing resource consumption of mobile devices running simultaneous localization and mapping (SLAM) algorithms, with most edge-assisted SLAM systems assuming the communication resources between the mobile device and the edge server to be unlimited, or relying on heuristics to choose the information to be transmitted to the edge. This paper presents AdaptSLAM, an edge-assisted visual (V) and visual-inertial (VI) SLAM system that adapts to the available communication and computation resources, based on a theoretically grounded method we developed to select the subset of keyframes (the representative frames) for constructing the best local and global maps in the mobile device and the edge server under resource constraints. We implemented AdaptSLAM to work with the state-of-the-art open-source V-and VI-SLAM ORB-SLAM3 framework, and demonstrated that, under constrained network bandwidth, AdaptSLAM reduces the tracking error by 62% compared to the best baseline method.

Real-time object detection (OD) is a key enabling technology for a wide range of emerging mobile system applications. However, deploying an OD model pre-trained on a public dataset (source domain) in a specific local environment (target domain) is known to lead to significant performance degradation because of the so-called domain gap between the dataset and the environment. Collecting local data and fine-tuning the OD model on this data is a commonly used approach for improving the robustness of OD models in real-world deployments. Yet, the question of how to collect this data is currently largely overlooked; unsupported data collection is likely to produce datasets that contain significant proportion of redundant or uninformative data for model training. In this demo, we present BiGuide, a bi-level image data acquisition guidance for OD tasks, to guide users to change their camera locations or angles to different extents (significantly or slightly) to obtain the data which benefits model training via image-level and object instance-level guidance. We showcase an interactive demonstration of collecting data for a lemur species detection application we are developing and deploying at the Duke Lemur Center. Demo participants will take pictures of lemur toys with the mobile phone under the real-time guidance and will observe the real-time display of the metrics that assess the importance of the captured data. They will develop an intuition for how real-time image importance assessment and bi-level guidance improve the quality of collected data.

In ophthalmology, retinal laser therapy is a treatment for retinopathy that requires the use of magnifying lens to treat damaged regions of retinal landmarks, hence creating challenges of inverted magnified images and requiring prolonged training. Augmented Reality (AR) can benefit clinicians during retinal laser therapy by guiding them with retinal landmark holograms and contextual information. Though recent developments in AR magnification show that a direct overlay of the magnified scenes can be achieved, retinal laser therapy requires high precision and visual acuity while maintaining the visual perception of the rest of the environment. Therefore, we demonstrate an AR-based selective magnification system that provides contextual and visualization-based guidance to clinicians. An edge-computing architecture is developed for detecting and matching the feature points between the magnified image and color fundus image of the retina to identify the magnified region of retinal landmarks. We showcase how our AR guidance system can assist clinicians during retinal laser therapy.

Virtual content instability caused by device pose tracking error remains a prevalent issue in markerless augmented reality (AR), especially on smartphones and tablets. However, when examining environments which will host AR experiences, it is challenging to determine where those instability artifacts will occur; we rarely have access to ground truth pose to measure pose error, and even if pose error is available, traditional visualizations do not connect that data with the real environment, limiting their usefulness. To address these issues we present SiTAR (Situated Trajectory Analysis for Augmented Reality), the first situated trajectory analysis system for AR that incorporates estimates of pose tracking error. We start by developing the first uncertainty-based pose error estimation method for visual-inertial simultaneous localization and mapping (VI-SLAM), which allows us to obtain pose error estimates without ground truth; we achieve an average accuracy of up to 96.1% and an average FI score of up to 0.77 in our evaluations on four VI-SLAM datasets. Next, we present our SiTAR system, implemented for ARCore devices, combining a backend that supplies uncertainty-based pose error estimates with a frontend that generates situated trajectory visualizations. Finally, we evaluate the efficacy of SiTAR in realistic conditions by testing three visualization techniques in an in-the-wild study with 15 users and 13 diverse environments; this study reveals the impact both environment scale and the properties of surfaces present can have on user experience and task performance.

Simultaneous localization and mapping (SLAM) takes in sensor data, e.g., camera frames, and estimates the user's trajectory while creating a map of the surrounding environment. However, existing SLAM evaluation methods are not reference-free, requiring ground-truth trajectories collected from external systems that are infeasible for most scenarios. In this demo, we present Deep SLAM Error Estimator (DeepSEE), a framework that collects features from a standard visual SLAM pipeline as multivariate time series and uses an attention-based neural network to estimate the tracking error at run time. We evaluate DeepSEE in a game engine-based virtual environment, which generates the visual input for DeepSEE and provides the ground-truth trajectory. Demo participants can navigate the virtual environment to create their own trajectories and view the online pose error estimation. This demo showcases how DeepSEE can act as a quality-of-service indicator for downstream applications.

Today’s Internet must support applications with increasingly dynamic and heterogeneous connectivity requirements, such as video streaming and the Internet of Things. Yet current network management practices generally rely on pre-specified network configurations, which may not be able to cope with dynamic application needs. Moreover, even the best-specified policies will find it difficult to cover all possible scenarios, given applications’ increasing heterogeneity and dynamic network conditions, e.g., on volatile wireless links. In this work, we instead propose a model-free learning approach to find the optimal network policies for current network flow requirements. This approach is attractive as comprehensive models do not exist for how different policy choices affect flow performance under changing network conditions. However, it can raise new challenges for online learning algorithms: policy configurations can affect the performance of multiple flows sharing the same network resources, and this performance coupling limits the scalability and optimality of existing online learning algorithms. In this work, we extend multi-armed bandit frameworks to propose new online learning algorithms for protocol selection with provably sublinear regret under certain conditions. We validate the optimality and scalability of our algorithms through data-driven simulations and testbed experiments. (An extended abstract of this work was accepted by IEEE ICNP as a short paper Zhanget al. (2019)).

We investigate training machine learning (ML) models across a set of geo-distributed, resource-constrained clusters of devices through unmanned aerial vehicles (UAV) swarms. The presence of time-varying data heterogeneity and computational resource inadequacy among device clusters motivate four key parts of our methodology: (i)stratified UAV swarmsof leader, worker, and coordinator UAVs, (ii)hierarchical nested personalized federated learning(HN-PFL), a distributed ML framework for personalized model training across the worker-leader-core network hierarchy, (iii)cooperative UAV resource poolingto address computational inadequacy of devices by conducting model training among the UAV swarms, and (iv)model/concept driftto model time-varying data distributions. In doing so, we consider bothmicro(i.e., UAV-level) andmacro(i.e., swarm-level) system design. At the micro-level, we propose network-awareHN-PFL, where we distributively orchestrate UAVs inside swarms to optimize energy consumption and ML model performance with performance guarantees. At the macro-level, we focus on swarm trajectory and learning duration design, which we formulate as a sequential decision making problem tackled via deep reinforcement learning. Our simulations demonstrate the improvements achieved by our methodology in terms of ML performance, network resource savings, and swarm trajectory efficiency.

The importance of the dynamics of the viewport pose, i.e., the location and the orientation of users’ points of view, for virtual reality (VR) experiences calls for the development of VR viewport pose models. In this paper, informed by our experimental measurements of viewport trajectories across 3 different types of VR interfaces, we first develop a statistical model of viewport poses in VR environments. Based on the developed model, we examine the correlations between pixels in VR frames that correspond to different viewport poses, and obtain an analytical expression for the visibility similarity (ViS) of the pixels across different VR frames. We then propose a lightweight ViS-based ALG-ViS algorithm that adaptively splits VR frames into the background and the foreground, reusing the background across different frames. Our implementation of ALG-ViS in two Oculus Quest 2 rendering systems demonstrates ALG-ViS running in real time, supporting the full VR frame rate, and outperforming baselines on measures of frame quality and bandwidth consumption.

Recent advances in eye tracking have given birth to a new genre of gaze-based context sensing applications, ranging from cognitive load estimation to emotion recognition. To achieve state-of-the-art recognition accuracy, a large-scale, labeled eye movement dataset is needed to train deep learning-based classifiers. However, due to the heterogeneity in human visual behavior, as well as the labor-intensive and privacy-compromising data collection process, datasets for gaze-based activity recognition are scarce and hard to collect. To alleviate the sparse gaze data problem, we present EyeSyn, a novel suite of psychology-inspired generative models that leverages only publicly available images and videos to synthesize a realistic and arbitrarily large eye movement dataset. Taking gaze-based museum activity recognition as a case study, our evaluation demonstrates that EyeSyn can not only replicate the distinct pat-terns in the actual gaze signals that are captured by an eye tracking device, but also simulate the signal diversity that results from dif-ferent measurement setups and subject heterogeneity. Moreover, in the few-shot learning scenario, EyeSyn can be readily incorpo-rated with either transfer learning or meta-learning to achieve 90% accuracy, without the need for a large-scale dataset for training.

The personalization of augmented reality (AR) experiences based on environmental and user context is key to unlocking their full potential. The recent addition of eye tracking to AR headsets provides a convenient method for detecting user context, but complex analysis of raw gaze data is required to detect where a user's attention and thoughts truly lie. In this demo we present Catch My Eye, the first system to incorporate deep neural network (DNN)-based activity recognition from user gaze into a realistic mobile AR app. We develop an edge computing-based architecture to offload context computation from resource-constrained AR devices, and present a working example of content adaptation based on user context, for the scenario of a virtual art gallery. It shows that user activities can be accurately recognized and employed with sufficiently low latency for practical AR applications.

External ventricular drain (EVD) is a common, yet challenging neurosurgical procedure of placing a catheter into the brain ventricular system that requires prolonged training for surgeons to improve the catheter placement accuracy. In this paper, we introduce NeuroLens, an Augmented Reality (AR) system that provides neurosurgeons with guidance that aides them in completing an EVD catheter placement. NeuroLens builds on prior work in AR-assisted EVD to present a registered hologram of a patient’s ventricles to the surgeons, and uniquely incorporates guidance on the EVD catheter’s trajectory, angle of insertion, and distance to the target. The guidance is enabled by tracking the EVD catheter. We evaluate NeuroLens via a study with 33 medical students, in which we analyzed students’ EVD catheter insertion accuracy and completion time, eye gaze patterns, and qualitative responses. Our study, in which NeuroLens was used to aid students in inserting an EVD catheter into a realistic phantom model of a human head, demonstrated the potential of NeuroLens as a tool that will aid and educate novice neurosurgeons. On average, the use of NeuroLens improved the EVD placement accuracy of year 1 students by 39.4% and of the year 2–4 students by 45.7%. Furthermore, students who focused more on NeuroLens-provided contextual guidance achieved better results.

Demand is growing for markerless augmented reality (AR) experiences, but designers of the real-world spaces that host them still have to rely on inexact, qualitative guidelines on the visual environment to try and facilitate accurate pose tracking. Furthermore, the need for visual texture to support markerless AR is often at odds with human aesthetic preferences, and understanding how to balance these competing requirements is challenging due to the siloed nature of the relevant research areas. To address this, we present an integrated design methodology for AR spaces, that incorporates both tracking and human factors into the design process. On the tracking side, we develop the first VI-SLAM evaluation technique that combines the flexibility and control of virtual environments with real inertial data. We use it to perform systematic, quantitative experiments on the effect of visual texture on pose estimation accuracy; through 2000 trials in 20 environments, we reveal the impact of both texture complexity and edge strength. On the human side, we show how virtual reality (VR) can be used to evaluate user satisfaction with environments, and highlight how this can be tailored to AR research and use cases. Finally, we demonstrate our integrated design methodology with a case study on AR museum design, in which we conduct both VI-SLAM evaluations and a VR-based user study of four different museum environments.

Federated learning (FL) has attracted increasing attention as a promising technique to drive a vast number of edge devices with artificial intelligence. However, it is very challenging to guarantee the efficiency of a FL system in practice due to the heterogeneous computation resources on different devices. To improve the efficiency of FL systems in the real world, asynchronous FL (AFL) and semi-asynchronous FL (SAFL) methods are proposed such that the server does not need to wait for stragglers. However, existing AFL and SAFL systems suffer from poor accuracy and low efficiency in realistic settings where the data is non-IID distributed across devices and the on-device resources are extremely heterogeneous. In this work, we propose FedSEA - a semi-asynchronous FL framework for extremely heterogeneous devices. We theoretically disclose that the unbalanced aggregation frequency is a root cause of accuracy drop in SAFL. Based on this analysis, we design a training configuration scheduler to balance the aggregation frequency of devices such that the accuracy can be improved. To improve the efficiency of the system in realistic settings where the devices have dynamic on-device resource availability, we design a scheduler that can efficiently predict the arriving time of local updates from devices and adjust the synchronization time point according to the devices' predicted arriving time. We also consider the extremely heterogeneous settings where there exist extremely lagging devices that take hundreds of times as long as the training time of the other devices. In the real world, there might be even some extreme stragglers which are not capable of training the global model. To enable these devices to join in training without impairing the systematic efficiency, Fed-SEA enables these extreme stragglers to conduct local training on much smaller models. Our experiments show that compared with status quo approaches, FedSEA improves the inference accuracy by 44.34% and reduces the systematic time cost and local training time cost by 87.02× and 792.9×. FedSEA also reduces the energy consumption of the devices with extremely limited resources by 752.9×.

Sensing and Augmented Reality (AR) can benefit a wide range of applications that involve the use of magnifying lenses. Recent developments in AR magnification provide a direct overlay of the magnified scenes in AR. However, instrumentation tasks that require high precision and visual acuity need to selectively magnify a region of interest while maintaining the visual perception of the rest of the environment. In this demo, we present AR-Magnifier, an AR magnification system through feature detection and matching. We propose a general framework based on an edge-computing architecture that can be applied to various types of instrumentation tasks. A pipeline is developed for detecting feature points and computing the homography matching to identify the magnified region of an object. We showcase how selective magnification in AR through sensing can assist the user in complex instrumentation tasks by providing visualization-based guidance.

Mobile augmented reality (AR), which integrates virtual objects (i.e., holographic contents) with 3-D real environments in real time, has been rapidly gaining popularity in the last five years. The delivery mechanisms of these holographic contents to mobile AR devices, however, are rarely investigated. To combat bandwidth limitations that preclude providing holographic contents to user devices on-demand, in this article, we propose the intelligent AR (Intelli-AR) preloading algorithm to improve transmission efficiency in the edge-assisted network, in which edge servers proactively transmit holographic contents to the devices. Without user devices’ future motion trajectories, the Intelli-AR preloading algorithm models the user devices’ motion trajectories as Markov decision process (MDP) and adaptively learns the optimal preloading policy. The Intelli-AR preloading is decomposed into two parts and separately deployed on the edge server and the user devices to reduce the computation complexity. The Intelli-AR solution improves the ratio of successful preloading by 11.52% compared to the best baseline in the practical data set when the users’ motion trajectories tend to be more random, and by 21.97% compared to the best baseline in the data set which is synthesized from a real-life mobile AR environment.

Kinetic energy harvesting technologies have been progressively used to power wearable devices and to sense the context through energy generation patterns. However, detecting human activities with signals from kinetic harvesters still needs improvement due to the use of approaches based on handcrafted features and the overfitting to device location or subjects. Hence, in this article, we present a deep learning architecture that leverages the feature extraction capability of the convolutional neural networks and the construction of the temporal sequences of recurrent neural networks to improve existing classification results. To provide sufficient data for the deep learning classifier, we propose three data augmentation methods to increase intraclass variance simulating new users performing the same activities. The proposed architecture outperforms existing approaches of kinetic harvesting-based human activity recognition by 13% of accuracy when the training data are augmented with the proposed methods. Finally, given the dependency of kinetic harvesting signals on device location and subjects, we employ transfer learning to improve the classification performance when the system is exposed to new subjects and locations. Transfer learning helps to increase classification performance by 30% when the device location is changed and 35% when the data come from a new subject.

Mobile Augmented Reality (AR), which overlays digital content on the real-world scenes surrounding a user, is bringing immersive interactive experiences where the real and virtual worlds are tightly coupled. To enable seamless and precise AR experiences, an image recognition system that can accurately recognize the object in the camera view with low system latency is required. However, due to the pervasiveness and severity of image distortions, an effective and robust image recognition solution for “in the wild” mobile AR is still elusive. In this article, we present CollabAR, an edge-assisted system that provides distortion-tolerant image recognition for mobile AR with imperceptible system latency . CollabAR incorporates both distortion-tolerant and collaborative image recognition modules in its design. The former enables distortion-adaptive image recognition to improve the robustness against image distortions, while the latter exploits the spatial-temporal correlation among mobile AR users to improve recognition accuracy. Moreover, as it is difficult to collect a large-scale image distortion dataset, we propose a Cycle-Consistent Generative Adversarial Network-based data augmentation method to synthesize realistic image distortion. Our evaluation demonstrates that CollabAR achieves over 85% recognition accuracy for “in the wild” images with severe distortions, while reducing the end-to-end system latency to as low as 18.2 ms.

• Understand biomolecular systems through their motions •Numerical integration of Newton's laws of motion — Model atoms as point masses — Compute forces on every atom based on current positions — Update atom velocities and positions in discrete time steps of a few femtoseconds • Force computation described by a model: the force field

Anton 3 is the newest member in a family of supercomputers specially designed for atomic-level simulation of molecules relevant to biology (e.g., DNA, proteins, and drug molecules). Anton 3 achieves order-of-magnitude improvements in time-to-solution over its predecessor, Anton 2 (the current state of the art), and is over 100-fold faster than any other currently available supercomputer, thereby enabling broad new avenues of research on critical questions in biology and drug discovery. This speedup means that a 512-node Anton 3 simulates a million atoms at over 100 microseconds per day. Furthermore, Anton 3 attains this performance while consuming an order of magnitude less energy per simulated microsecond than any other machine. Like its predecessors, Anton 3 was designed from the ground up around a new custom chip to best exploit the capabilities offered by new technologies. We present here the main architectural and algorithmic developments that were necessary to achieve such significant advances.

Conventional radio-frequency (RF) sensing systems rely on either frequency diversity or spatial diversity to ensure high sensing accuracy. Such reliance introduces several practical limitations that hinder the pervasive deployment of existing solutions. To circumvent this prevalent reliance, we present MetaSense, a system that leverages antenna pattern diversity for fine-grained RF sensing. MetaSense incorporates the dynamic metasurface antenna (DMA) and the auxiliary-assisted ensemble multimask learning (AEMML) framework in its design. The DMA is a novel type of antenna that can provide a diverse set of uncorrelated radiation patterns in a low-cost and low-complexity manner. The AEMML is a quality-aware learning framework that can dynamically assess and aggregate the heterogeneous channel measurements from different antenna patterns to ensure high sensing accuracy. It also incorporates a transfer learning model that allows it to generalize to new sensing conditions with few training instances required. We prototype MetaSense and demonstrate its effectiveness on a writing motion recognition task using a custom-designed 2-D DMA. The results show that MetaSense achieves 92% to 98% accuracy in classifying ten miniature writing motions, outperforming a nontunable antenna by 20% in all scenarios. Moreover, when deployed in new sensing positions where limited training instances are available, MetaSense requires as few as five training instances per class to achieve over 90% accuracy.

Multi-user augmented reality (AR), where multiple co-located users view a common set of virtual objects, is becoming increasingly popular. For example, Google Just a Line allows multiple users to draw virtual graffiti in the same physical space. Multi-user AR requires network communications in order to coordinate the positions of the virtual objects on each user's display, yet there is currently little understanding of how such apps communicate. In this work, we address this key gap in knowledge by showing that the communicated data directly impacts the latency and positioning of the virtual objects rendered on the users' displays. We develop solutions to these problems that we find along three facets: (1) efficient communication strategies that trade off communication latency for spatial consistency of the virtual objects; (2) a new metric that enables mobile AR devices to update their virtual objects as they move around and observe more of the scene; and (3) a tool to automatically quantify how much the virtual objects' positions inadvertently change in time and space. Our evaluation is performed on Android smartphones running open-source AR. The results show that our system, SPAR, can decrease the latency by up to 55%, while decreasing the spatial inconsistency by up to 60%, compared to baseline methods.

Mobile Augmented Reality (AR), which overlays digital content on the real-world scenes surrounding a user, is bringing immersive interactive experiences where the real and virtual worlds are tightly coupled. To enable seamless and precise AR experiences, an image recognition system that can accurately recognize the object in the camera view with low system latency is required. However, due to the pervasiveness and severity of image distortions, an effective and robust image recognition solution for mobile AR is still elusive. In this paper, we present CollabAR, an edge-assisted system that provides distortion-tolerant image recognition for mobile AR with imperceptible system latency. CollabAR incorporates both distortion-tolerant and collaborative image recognition modules in its design. The former enables distortion-adaptive image recognition to improve the robustness against image distortions, while the latter exploits the ‘spatial-temporal’ correlation among mobile AR users to improve recognition accuracy. We implement CollabAR on four different commodity devices, and evaluate its performance on two multi-view image datasets. Our evaluation demonstrates that CollabAR achieves over 96% recognition accuracy for images with severe distortions, while reducing the end-to-end system latency to as low as 17.8ms for commodity mobile devices.

In this work, we present GazeGraph, a system that leverages human gazes as the sensing modality for cognitive context sensing. GazeGraph is a generalized framework that is compatible with different eye trackers and supports various gaze-based sensing applications. It ensures high sensing performance in the presence of heterogeneity of human visual behavior, and enables quick system adaptation to unseen sensing scenarios with few-shot instances. To achieve these capabilities, we introduce the spatial-temporal gaze graphs and the deep learning-based representation learning method to extract powerful and generalized features from the eye movements for context sensing. Furthermore, we develop a few-shot gaze graph learning module that adapts the `learning to learn' concept from meta-learning to enable quick system adaptation in a data-efficient manner. Our evaluation demonstrates that GazeGraph outperforms the existing solutions in recognition accuracy by 45% on average over three datasets. Moreover, in few-shot learning scenarios, GazeGraph outperforms the transfer learning-based approach by 19% to 30%, while reducing the system adaptation time by 80%.

With the emergence of the Internet of Things (IoT), billions of heterogeneous physical objects are connected through a network for collecting and sharing information, which can improve various aspects of daily lives, including smart living and transportation, and smart ambient environment, including smart city, smart home, smart agriculture, smart water, waste management, etc. The main objective of the IoT devices is to provide seamless services to the users without their intervention. The all-connected paradigm (i.e., connecting people, things, processes, and data in the network) is based on near Internet ubiquity and includes three types of communication: 1) machine-to-machine; 2) person-to-machine; and 3) person-to-person, and consists as the base for reliable services provision to the end devices at the edge. Fog paradigm implements effectively and efficiently the all-data requests to be shared in a reliable manner as fog implementations complement the cloud computing paradigm by extending computing and caching capabilities to the edges of the network, and it facilitates smart localization decisions and rapid responses. The wide range of IoT services calls for a disruptive, highly efficient, scalable, and flexible communication network able to cope with the increasing demands and the number of connected devices, as well as the diverse and stringent application requirements.

Plant Spike is an in situ low-cost sensor system that is wireless, miniature, and low powered. It can be seamlessly implanted in subsurface locations across major cities to measure urban soil health. Plant Spike incorporates noncontact soil moisture monitoring, temperature monitoring, light intensity monitoring, advanced power management, and Bluetooth low energy transmit-only communication for transmitting information to a client device. With a novel combination of aggressive power reduction techniques, the system's lifetime is over two years with a 500-mAh battery. By connecting on-board sensors to a single-chip microcontroller, the total component and assembly cost of each module is less than $10. The sensor system has been tested within an urban soil testbed located on Columbia University's Morningside Campus in New York City as well as street tree pits located in Morningside Heights, proving the functionality and robustness of the system. Plant Spike is able to measure temperature and light ranges that are comparable to the fluctuations experienced by soils located within the climate zone of New York City.

Augmented reality is an emerging application on mobile devices. However, there is a lack of understanding of the communication requirements and challenges of multi-user AR scenarios. In this position paper, we propose several important research issues that need to be addressed for low-latency, accurate shared AR experiences: (a) Systems tradeoffs of AR communication architectures used today in mobile AR platforms; (b) Understanding AR communication patterns and adapting the AR application layer to dynamically changing network conditions; and (c) Tools and methodologies to evaluate AR quality of experience in real time on mobile devices. We present preliminary measurements of off-the-shelf mobile AR platforms as well as results from our AR system, ShareAR, illustrating performance tradeoffs and indicating promising new research directions.

The integration of microfluidics and biosensor technology is transforming microbiology research by providing new capabilities for clinical diagnostics, cancer research, and pharmacology studies. This integration enables new approaches for biochemistry automation and cyber-physical adaptation. Similarly, recent years have witnessed the rapid growth of the Internet of Things (IoT) paradigm, where different types of real-world elements such as wearable sensors are connected and allowed to autonomously interact with each other. Combining the advances of both cyber-physical microfluidics and IoT domains can generate new opportunities for knowledge fusion by transforming distributed local microfluidic elements into a global network of coordinated microfluidic systems. This paper aims to streamline this transformation and it presents a research vision for enabling the Internet of Microfluidic Things (IoMT). To leverage advances in connected Microfluidic Things, we highlight new perspectives on system architecture, and describe technical challenges related to design automation, temporal flexibility, security, and service assignment. This vision is supported by case studies from cancer research and pharmacology studies to explain the significance of the proposed framework.

Today’s Internet must support applications with increasingly dynamic and heterogeneous connectivity requirements, such as video streaming and the Internet of Things. Yet current network management practices generally rely on pre-specified flow configurations, which cannot cover all possible scenarios. In this work, we instead propose a model-free learning approach to automatically optimize the policies for heterogeneous network flows. This approach is attractive as no existing comprehensive models quantify how different policy choices affect flow performance under dynamically changing network conditions. We extend multi-armed bandit frameworks to propose new online learning algorithms for protocol selection, addressing the challenge of policy configurations affecting the performance of multiple flows sharing the same network resources. This performance coupling limits the scalability and optimality of existing online learning algorithms. We theoretically prove that our algorithm achieves a sublinear regret and demonstrate its optimality and scalability through data-driven simulations.

We study a multi-agent partially observable environment in which autonomous agents aim to coordinate their actions, while also learning the parameters of the unknown environment through repeated interactions. In particular, we focus on the role of communication in a multi-agent reinforcement learning problem. We consider a learning algorithm in which agents make decisions based on their own observations of the environment, as well as the observations of other agents, which are collected through communication between agents. We first identify two potential benefits of this type of information sharing when agents' observation quality is heterogeneous: (1) it can facilitate coordination among agents, and (2) it can enhance the learning of all participants, including the better informed agents. We show however that these benefits of communication depend in general on its timing, so that delayed information sharing may be preferred in certain scenarios.

Augmented reality (AR) technologies have seen significant improvement in recent years with several consumer and commercial solutions being developed. New security challenges arise as AR becomes increasingly ubiquitous. Previous work has proposed techniques for securing the output of AR devices and used reinforcement learning (RL) to train security policies which can be difficult to define manually. However, whether such systems and policies can be deployed on a physical AR device without degrading performance was left an open question. We develop a visual output security application using a RL trained policy and deploy it on a Magic Leap One head-mounted AR device. The demonstration illustrates that RL based visual output security systems are feasible.

Mobile Augmented Reality (AR), which overlays digital information with real-world scenes surrounding a user, provides an enhanced mode of interaction with the ambient world. Contextual AR applications rely on image recognition to identify objects in the view of the mobile device. In practice, due to image distortions and device resource constraints, achieving high performance image recognition for AR is challenging. Recent advances in edge computing offer opportunities for designing collaborative image recognition frameworks for AR. In this demonstration, we present CollabAR, an edge-assisted collaborative image recognition framework. CollabAR allows AR devices that are facing the same scene to collaborate on the recognition task. Demo participants develop an intuition for different image distortions and their impact on image recognition accuracy. We showcase how heterogeneous images taken by different users can be aggregated to improve recognition accuracy and provide a better user experience in AR.

This paper introduces an analytical framework to investigate optimal design choices for the placement of virtual controllers along the cloud-to-things continuum. The main application scenarios include low-latency cyber-physical systems in which real-time control actions are required in response to the changes in states of an Internet of Things (IoT) node. In such cases, deploying controller software on a cloud server is often not tolerable due to delay from the network edge to the cloud. Hence, it is desirable to trade reliability with latency by moving controller logic closer to the network edge. Modeling the IoT node as a dynamical system that evolves linearly in time with quadratic penalty for state deviations, recursive expressions for the optimum control policy and the resulting minimum cost value are obtained by taking virtual fog controller reliability and response time latency into account. Our results indicate that latency is more critical than reliability in provisioning virtualized control services over fog endpoints, as it determines the swiftness of the fog control system as well as the timeliness of state measurements. Based on a drone trajectory tracking model, an extensive simulation study is also performed to illustrate the influence of reliability and latency on the control of autonomous vehicles over fog.

Fog computing, the distribution of computing resources closer to the end devices along the cloud-to-things continuum, is recently emerging as an architecture for scaling of the Internet of Things (IoT) sensor networking applications. Fog computing requires novel computing program decompositions for heterogeneous hierarchical settings. To evaluate these new decompositions, we designed, developed, and instrumented a fog computing testbed that includes cloud computing and computing gateway execution points collaborating to finish complex data analytics operations. In this interactive demonstration we present one fog-specific algorithmic decomposition we recently examined and adapted for fog computing: a multi-execution point linear regression decomposition that jointly optimizes operation latency, quality, and costs. The demonstration highlights the role fog computing can play in future sensor networking architectures, and highlights some of the challenges of creating computing program decompositions for these architectures. An annotated video of the demonstration is available at [5].

Numerous energy harvesting wireless devices that will serve as building blocks for the Internet of Things (IoT) are currently under development. However, there is still only limited understanding of the properties of various energy sources and their impact on energy harvesting adaptive algorithms. Hence, we focus on characterizing the kinetic (motion) energy that can be harvested by a wireless node with an IoT form factor and on developing energy allocation algorithms for such nodes. In this paper, we describe methods for estimating harvested energy from acceleration traces. To characterize the energy availability associated with specific human activities (e.g., relaxing, walking, cycling), we analyze a motion dataset with over 40 participants. Based on acceleration measurements that we collected for over 200 hours, we study energy generation processes associated with day-long human routines. We also briefly summarize our experiments with moving objects. We develop energy allocation algorithms that take into account practical IoT node design considerations, and evaluate the algorithms using the collected measurements. Our observations provide insights into the design of motion energy harvesters, IoT nodes, and energy harvesting adaptive algorithms.

This article focuses on a new type of wireless devices in the domain between RFIDs and sensor networks—Energy-Harvesting Active Networked Tags (EnHANTs). Future EnHANTs will be small, flexible, and self-powered devices that can be attached to objects that are traditionally not networked (e.g., books, furniture, toys, produce, and clothing). Therefore, they will provide the infrastructure for various tracking applications and can serve as one of the enablers for the Internet of Things. We present the design considerations for the EnHANT prototypes, developed over the past 4 years. The prototypes harvest indoor light energy using custom organic solar cells, communicate and form multihop networks using ultra-low-power Ultra-Wideband Impulse Radio (UWB-IR) transceivers, and dynamically adapt their communications and networking patterns to the energy harvesting and battery states. We describe a small-scale testbed that uniquely allows evaluating different algorithms with trace-based light energy inputs. Then, we experimentally evaluate the performance of different energy-harvesting adaptive policies with organic solar cells and UWB-IR transceivers. Finally, we discuss the lessons learned during the prototype and testbed design process.

Numerous energy harvesting wireless devices that will serve as building blocks for the Internet of Things (IoT) are currently under development. However, there is still only limited understanding of the properties of various energy sources and their impact on energy harvesting adaptive algorithms. Hence, we focus on characterizing the kinetic (motion) energy that can be harvested by a wireless node with an IoT form factor and on developing energy allocation algorithms for such nodes. In this paper, we describe methods for estimating harvested energy from acceleration traces. To characterize the energy availability associated with specific human activities (e.g., relaxing, walking, cycling), we analyze a motion dataset with over 40 participants. Based on acceleration measurements that we collected for over 200 hours, we study energy generation processes associated with day-long human routines. We also briefly summarize our experiments with moving objects. We develop energy allocation algorithms that take into account practical IoT node design considerations, and evaluate the algorithms using the collected measurements. Our observations provide insights into the design of motion energy harvesters, IoT nodes, and energy harvesting adaptive algorithms.

This paper focuses on a new type of wireless devices in the domain between RFIDs and sensor networks - Energy Harvesting Active Networked Tags (EnHANTs). Future EnHANTs will be small, flexible, and self-powered devices that can be attached to objects that are traditionally not networked (e.g., books, toys, clothing), thereby providing the infrastructure for novel tracking applications. We present the design considerations for the EnHANT prototypes, developed over the past 3 years. The prototypes harvest indoor light energy using custom organic solar cells, communicate and form multihop networks using ultralow-power Ultra-Wideband Impulse Radio (UWB-IR) transceivers, and adapt their communications and networking patterns to the energy harvesting and battery states. We also describe a small scale EnHANTs testbed that uniquely allows evaluating different algorithms with trace-based light energy inputs.

The modern computing landscape increasingly requires a range of skills to successfully integrate complex systems. Project-based learning is used to help students build professional skills. However, it is typically applied to small teams and small efforts. In this paper, we describe our experience in engaging a large number of students in research projects within a multi-year interdisciplinary research effort. The projects expose the students to various disciplines in Electrical Engineering (circuit design, wireless communications, hardware prototyping), Computer Science (embedded systems, algorithm design, networking) and Applied Physics (thin-film battery design, solar cell fabrication). While a student project is usually focused on one discipline area, it requires interaction with at least two other areas. Over 4 years, 115 semester-long projects have been completed. The students were a diverse group of high school, undergraduate, and M.S. Computer Science, Computer Engineering, and Electrical Engineering students. Some of the approaches we have taken to facilitate student learning are real-world system development constraints, regular cross-group meetings, and extensive involvement of Ph.D. students in student mentorship and knowledge transfer. To assess our approaches, we conducted a survey among the participating students. The results demonstrate the effectiveness of our methods. For example, 70% of the students surveyed indicated that working on their research project improved their ability to function on multidisciplinary teams more than coursework, internships, or any other activity.

Recent advances in energy harvesting materials and ultra-low-power communications will soon enable the realization of networks composed of energy harvesting devices. These devices will operate using very low ambient energy, such as energy harvested from indoor lights. We focus on characterizing the light energy availability in indoor environments and on developing energy allocation algorithms for energy harvesting devices. First, we present results of our long-term indoor radiant energy measurements, which provide important inputs required for algorithm and system design (e.g., determining the required battery sizes). Then, we focus on algorithm development, which requires nontraditional approaches, since energy harvesting shifts the nature of energy-aware protocols from minimizing energy expenditure to optimizing it. Moreover, in many cases, different energy storage types (rechargeable battery and a capacitor) require different algorithms. We develop algorithms for calculating time fair energy allocation in systems with deterministic energy inputs, as well as in systems where energy inputs are stochastic.

Recent advances in energy harvesting materials and ultra-low-power communications will soon enable the realization of networks composed of energy harvesting devices. These devices will operate using very low ambient energy, such as indoor light energy. We focus on characterizing the energy availability in indoor environments and on developing energy allocation algorithms for energy harvesting devices. First, we present results of our long-term indoor radiant energy measurements, which provide important inputs required for algorithm and system design (e.g., determining the required battery sizes). Then, we focus on algorithm development, which requires nontraditional approaches, since energy harvesting shifts the nature of energy-aware protocols from minimizing energy expenditure to optimizing it. Moreover, in many cases, different energy storage types (rechargeable battery and a capacitor) require different algorithms. We develop algorithms for determining time fair energy allocation in systems with predictable energy inputs, as well as in systems where energy inputs are stochastic.

Energy Harvesting Active Networked Tags (EnHANTs) are a new class of devices in the domain between RFIDs and sensor networks. EnHANTs will be small, flexible, and energetically self-reliant. Their development is enabled by advances in ultra-low-power ultra-wideband (UWB) communications and in organic semiconductor-based energy harvesting materials. In this demo, we present UWB-enabled EnHANT prototypes. Each prototype is based on a MICA2 mote integrated with a UWB Transceiver and an energy harvesting module (EHM) that allows demonstrating energy harvesting-adaptive communications. Additional information about EnHANTs is available at [2] and http://enhants.ee.columbia.edu.

Energy Harvesting Active Networked Tags (EnHANTs) will be a new class of devices in the domain between RFIDs and sensor networks. Small, flexible, and energetically self-reliant, EnHANTs will be attached to objects that are traditionally not networked, such as books, furniture, toys, produce, and clothing. More information about the EnHANTs project is available at http://enhants.ee.columbia.edu. In this demo we present a small network of EnHANT prototypes. The current EnHANT prototypes are integrated with novel custom in-house-developed energy harvesting and communications hardware, namely organic solar cells and ultra-wide-band impulse radio (UWB-IR) transceivers. The demo showcases prototypes communicating using the novel UWB-IR transceivers and adapting their communications and networking parameters to the available environmental energy harvested by the organic solar cells.

Femtocell deployments allow for high precision in localizing mobile devices. Many of today's location based services have long been mapping the placements of wireless base stations and using the obtained maps to localize mobile devices. Allowing unauthorized third parties to obtain the locations of femtocell base stations may not be desired by network operators. Localizing mobile devices using the information about femtocell base stations' locations is a service that an investor in a femtocell deployment may want to exploit exclusively. In this work we present a station identity management system that enables preserving femtocell base stations location privacy. Through the use of dynamic base station identifiers, the system ensures that unauthorized third parties are not able to map the locations of the base stations for use in their localization services. We analyze the design tradeoffs of the presented approach for different femtocell technologies. Results indicate that complexity will be limited, and that the presented system creates network dynamics smaller than the existing dynamics due to mobility. Additionally, we present an approach for providing location information to authorized systems at different resolution levels.

We focus on resource allocation for energy harvesting devices. We analytically and numerically evaluate the performance of algorithms that determine time fair energy allocation in systems with predictable and stochastic energy inputs. To gain insight into the performance of networks of devices, we obtain results for the simple cases of a single node and a link. Due to the need for low complexity algorithms, we focus on simple policies (some of which proposed in the past as heuristics) and analytically derive performance guarantees. We also evaluate the performance via simulation, using real-world energy traces that we collected for over a year, and in a testbed of energy harvesting devices developed within the EnHANTs project.

With the convergence of ultra-low-power communications and energy-harvesting technologies, networking self-sustainable ubiquitous devices is becoming feasible. Hence, we have been recently developing new devices, referred to as Energy Harvesting Active Networked Tags (EnHANTs). These small, flexible, and energetically self-reliant tags can be seen as a new class of devices in the domain between RFIDs and sensor networks. EnHANTs are made possible by advances in ultra-lowpower ultra-wideband (UWB) communications and in organic semiconductor-based energy harvesting materials. They will enable novel tracking applications, such as continuous monitoring of objects and locating misplaced items. In this demo, we present phase I EnHANT prototypes. These prototypes are much larger than the envisioned EnHANTs and do not include custom-made UWB and organic electronic components. Yet, they serve as platforms for preliminary experiments and allow demonstrating energy harvesting-adaptive EnHANT communications. Each prototype is based on a MICA2 mote and includes a custom-designed sensor board with a light sensor and a solar cell, which are used to determine the light energy received from the environment. We have also designed a monitoring system which is used in the demo to show how the EnHANT prototypes adjust their communications patterns based on their energy harvesting parameters.

This paper presents the design challenges posed by a new class of ultra-low-power devices referred to as Energy-Harvesting Active Networked Tags (EnHANTs). EnHANTs are small, flexible, and self-reliant (in terms of energy devices that can be attached to objects that are traditionally not networked (e.g., books, clothing, and produce), thereby providing the infrastructure for various novel tracking applications. Examples of these applications include locating misplaced items, continuous monitoring of objects (items in a store, boxes in transit), and determining locations of disaster survivors. Recent advances in ultra-low-power wireless communications, ultra-wideband (UWB) circuit design, and organic electronic harvesting techniques will enable the realization of EnHANTs in the near future. In order for EnHANTs to rely on harvested energy, they have to spend significantly less energy than Bluetooth, Zigbee, and IEEE 802.15.4a devices. Moreover, the harvesting components and the ultra-low-power physical layer have special characteristics whose implications on the higher layers have yet to be studied (e.g., when using ultra-low-power circuits, the energy required to receive a bit is an order of magnitude higher than the energy required to transmit a bit). These special characteristics pose several new cross-layer research problems. In this paper, we describe the design challenges at the layers above the physical layer, point out relevant research directions, and outline possible starting points for solutions.