Plastics recycling is a critical ecological and economic solution to manage plastic waste, yet a staggering proportion of plastics from daily use is landfilled or incinerated. A critical step to recycling plastics is our ability to sort plastics by type (HDPE, LDPE, PET, PP, PS, and PVC) at a mixed recycling facility. However, challenges such as sensor system cost, difficulty in data collection, and the dense sampling required for model fine-tuning continue to hinder reliable large-scale deployment and limit progress toward a sustainable circular plastic economy. In this work, we propose a novel physics-informed plastics classification system based on active thermal imaging. Additionally, we present a two-stage training strategy that uses a large quantity of easily generated PDE-based simulated samples for pretraining and fine-tunes the model to real-world data distributions using only a sparse set of samples. We validate the efficacy of the proposed approach on real-world plastic samples. Thus, we introduce Thermal Imaging based Plastic Sorting (TIPS), a system that achieves up to 100% and 94.7% accuracy in plastic type classification for black and white plastics, respectively.

Volumetric learning underpins many 3D vision tasks such as completion, reconstruction, and mesh generation, yet training objectives still rely on Chamfer Distance (CD) or Earth Mover’s Distance (EMD), which fail to balance recall and precision. We propose Quality-Aware Loss (QAL), a drop-in replacement for CD/EMD that combines a coverage-weighted nearest-neighbor term with an uncovered-ground-truth attraction term, explicitly decoupling recall and precision into tunable components. Across diverse pipelines, QAL achieves consistent coverage gains, improving by an average of +4.3 pts over CD and +2.8 pts over the best alternatives. These improvements reliably recover thin structures and under-represented regions that CD/EMD overlook. Extensive ablations confirm stable performance across hyperparameters and output resolutions, while full retraining on PCN and ShapeNet demonstrates generalization across datasets and backbones. Moreover, QAL-trained completions yield higher grasp scores under GraspNet evaluation, showing that improved coverage translates directly into more reliable robotic manipulation. QAL thus offers a principled, interpretable, and practical objective for robust 3D vision and safety-critical robotics pipelines.

Plastics detection using mobile apps can greatly assist in plastics sorting at the source and allow great improvements in the percentage of plastics that are recycled. Previous work such as DeepWaste and MWaste has tackled general waste classification, including plastics, but few efforts focus on real-time plastic type identification on mobile devices. CLIPS addresses this gap by developing a mobile app in combination with a cloud service that enables plastic-material classification. Further, CLIPS utilizes a continual learning architecture to adapt the model to the local stream of plastics, allowing for greater detection accuracy over time. We demonstrate that our approach can improve performance by 37% through continual learning compared to a pre-trained model. We also demonstrate a positive backward transfer of +27% and a forward transfer of +19.63% with continual learning over time.

The tracking module of a visual-inertial SLAM system processes incoming image frames and IMU data to estimate the position of the frame in relation to the map. It is important for the tracking to complete in a timely manner for each frame to avoid poor localization or tracking loss. We therefore present a new approach which leverages GPU computing power to accelerate time-consuming components of tracking in order to improve its performance. These components include stereo feature matching and local map tracking. We implement our design inside the ORB-SLAM3 tracking process using CUDA. Our evaluation demonstrates an overall improvement in tracking performance of up to 2.8× on a desktop and Jetson Xavier NX board in stereo-inertial mode, using the well-known SLAM datasets EuRoC and TUM-VI.

Visual SLAM systems are concurrent, performance-critical systems that respond to real-time environmental conditions and are frequently deployed on resource-constrained hardware. Previous work has identified three interconnected systems challenges to building consistent, accurate, and robust SLAM systems— timeliness , concurrency , and context awareness . In this article, we analyze three popular, state-of-the-art frameworks with varying system designs and optimization techniques, and we quantify the extent to which they are affected by the aforementioned system challenges. We find that all SLAM systems must balance the interconnected nature of timeliness and accuracy, and different system designs and optimization techniques uniquely address this tension. Global-map-based SLAM systems typically achieve the best performance but suffer in resource-constrained scenarios with increased concurrency . Across all SLAM systems, incorporating context awareness into decision-making would mitigate the impact of timeliness and concurrency on accuracy in resource-constrained scenarios.

Recently, there has been growing interest in the identification and re-identification of individuals from long distances using rooftop cameras, UAV cameras, street cams, and similar devices. This type of recognition extends beyond facial recognition, utilizing whole-body markers such as gait. However, datasets to train and test such recognition algorithms are scarce and often lack labeling. This paper introduces DIOR—a comprehensive framework for data collection, semi-automated annotation, and a dataset comprising 1.649 million RGB frames labeled with 3D/2D skeleton gait markers across 14 subjects. This dataset includes 200,000 RGB frames captured from long-range cam-eras. Our approach employs advanced 3D computer vision techniques to achieve pixel-level accuracy in indoor environments using motion capture systems. For outdoor, long-range environments, we eliminate the reliance on motion capture systems and implement a cost-effective, hybrid 3D computer vision and learning pipeline using only four inexpensive RGB cameras. This method successfully achieves precise skeleton labeling of distant subjects, even when their visual size is as small as 20-25 pixels within an RGB frame. We benchmark models trained on existing datasets such as CASIA-B, on our proposed dataset for the task of Gait recognition. Our pipeline and the accompanying dataset will be made publicly available following acceptance.

In this paper, we introduce JacobiGPU, a technique that uses a GPU to improve the efficiency of loop closure in visual-inertial SLAM systems, particularly when approximating Jacobians using the Finite Difference Method (FDM). Traditional FDM techniques often face computational overhead due to repeated perturbations in pose graphs. We address this overhead with a novel methodology, leveraging strategic graph partitioning and an optimized approach to Jacobian approximation. By integrating JacobiGPU into ORB-SLAM3’s g2o, we enhance the linearization process. Our evaluation, conducted on 12 sequences of varying lengths from the EuRoC and TUM-VI datasets, demonstrated a speedup of up to 4.23x in the linearization stage and an overall enhancement of up to 2.08x in the overall optimization process.

Historically, feature-based approaches have been used extensively for camera-based robot perception tasks such as localization, mapping, tracking, and others. Several of these approaches also combine other sensors (inertial sensing, for example) to perform combined state estimation. Our work rethinks this approach; we present a representation learning mechanism that identifies visual features that best correspond to robot motion as estimated by an external signal. Specifically, we utilize the robot’s transformations through an external signal (inertial sensing, for example) and give attention to image space that is most consistent with the external signal. We use a pairwise consistency metric as a representation to keep the visual features consistent through a sequence with the robot’s relative pose transformations. This approach enables us to incorporate information from the robot’s perspective instead of solely relying on the image attributes. We evaluate our approach on real-world datasets such as KITTI & EuRoC and compare the refined features with existing feature descriptors. We also evaluate our method using our real robot experiment. We notice an average of 49% reduction in the image search space without compromising the trajectory estimation accuracy. Our method reduces the execution time of visual odometry by 4.3% and also reduces reprojection errors. We demonstrate the need to select only the most important features and show the competitiveness using various feature detection baselines.

This paper presents an innovative approach to archaeological and geological exploration, combining Synthetic Aperture Radar (SAR) imagery, Ground Penetrating Radar (GPR), and advanced robotic algorithms. Utilizing SAR data captured by Capella, the study identifies areas of interest (AOIs) through supervised classification methods. These AOIs are then surveyed by a UAV equipped with GPR, optimized for efficient pathfinding and maximal coverage using robotic exploration algorithms. The survey generates high-resolution radar images, detailed digital elevation models, and orthomosaic images through photogrammetry, providing a comprehensive view of both surface and subsurface features.

Autonomous vehicles and human drivers are prone to line-of-sight limitations. Road-side mounted 3D sensors like LiDARs can augment a vehicle's on-board perception. However, this entails fusing 3D frames at low latency and high accuracy. Road-side and vehicle 3D frames are captured from different viewpoints. This adversely affects alignment accuracy and can be computationally expensive. To this end, VRF optimizes for both latency and accuracy by decoupling the alignment process into indirect and direct alignments. First, VRF indirectly aligns the 3D frames by aligning them to a common reference point i.e., a vehicle's on-board 3D map. Then, it directly aligns the two point clouds to refine this alignment. To ensure high accuracy, it incorporates novel offline registration and alignment accuracy forecasting modules. To ensure low latency, it uses a fast fusion pipeline that caches previous and offline computations. To our knowledge, VRF is the first vehicle road-side cooperative system to ensure cm-level accuracy and end-to-end latency less than 20 ms. Most importantly, its latency is below the 100 ms threshold required for autonomous vehicles to react to external events. Finally, VRF can improve reaction time to external events by as much as 5 seconds1.

A fundamental challenge in robot perception is the coupling of the sensor pose and robot pose. This has led to research in active vision where robot pose is changed to reorient the sensor to areas of interest for perception. Further, egomotion such as jitter, and external effects such as wind and others affect perception requiring additional effort in software such as image stabilization. This effect is particularly pronounced in micro-air vehicles and micro-robots who typically are lighter and subject to larger jitter but do not have the computational capability to perform stabilization in real-time. We present a novel microelectromechanical (MEMS) mirror LiDAR system to change the field of view of the LiDAR independent of the robot motion. Our design has the potential for use on small, low-power systems where the expensive components of the LiDAR can be placed external to the small robot. We show the utility of our approach in simulation and on prototype hardware mounted on a UAV. We believe that this LiDAR and its compact movable scanning design provide mechanisms to decouple robot and sensor geometry allowing us to simplify robot perception. We also demonstrate examples of motion compensation using IMU and external odometry feedback in hardware.

Deep learning has had remarkable success in robotic perception, but its data-centric nature suffers when it comes to generalizing to ever-changing environments. By contrast, physics-based optimization generalizes better, but it does not perform as well in complicated tasks due to the lack of high-level semantic information and reliance on manual parametric tuning. To take advantage of these two complementary worlds, we present PyPose: a robotics-oriented, PyTorch-based library that combines deep perceptual models with physics-based optimization. PyPose's architecture is tidy and well-organized, it has an imperative style interface and is efficient and user-friendly, making it easy to integrate into real-world robotic applications. Besides, it supports parallel computing of any order gradients of Lie groups and Lie algebras and 2nd-order optimizers, such as trust region methods. Experiments show that PyPose achieves more than 10× speedup in computation compared to the state-of-the-art libraries. To boost future research, we provide concrete examples for several fields of robot learning, including SLAM, planning, control, and inertial navigation.

In this paper, we present our approach to efficiently leveraging GPU resources to improve the performance of local bundle adjustment for visual-inertial SLAM. We observe that for local bundle adjustment (i) the Schur complement method, a technique often used to speed up bundle adjustment, has the largest overhead when solving for the parameter update, and (ii) the workload consists of operations on small- to medium-sized matrices. Based on these observations, we develop and combine several techniques that efficiently handle small- to medium-sized matrices. We then implement these techniques as a drop-in replacement block solver for g2o, a library frequently used for bundle adjustment, and integrate it with ORB-SLAM3, a well-known open-source visual-inertial SLAM system. Our evaluation done with two popular datasets, EuRoC and TUM-VI, shows that we can reduce the time taken by local bundle adjustment by 13.81%-33.79% with our techniques across an embedded device and a desktop machine.

As many as three million school age children between the ages of 5 and 14 years, live with severe to profound hearing loss in Nigeria. Many of these Deaf or Hard of Hearing (DHH) children developed their hearing loss later in life, non-congenitally, hence their parents are hearing. While their teachers in the Deaf schools they attend can often communicate effectively with them in "dialects" of American Sign Language (ASL), the unofficial sign lingua franca in Nigeria, communication at home with other family members is challenging and sometimes non-existent. This results in adverse social consequences including stigmatization, for the students. With the recent successes of AI in natural language understanding, the goal of automated sign language understanding is becoming more realistic using neural deep learning technologies. To this effect, the proposed project aims at co-designing and developing an ongoing AI-driven two-way sign language interpretation tool that can be deployed in homes, to improve language accessibility and communication between the DHH students and other family members. This ensures inclusive and equitable social interactions and can promote lifelong learning opportunities for them outside of the school environment.

In autonomous racing, accurately tracking the race line at the limits of handling is essential to guarantee competitiveness. In this study, we show the effectiveness of Differential Flatness based control for high-speed trajectory tracking for car-like robots. We compare the tracking performance of our controller against Nonlinear Model Predictive Control and resource use while running on embedded hardware and show that on average KFC reduces the computation resource usage by 50 % while performing on par with NMPC. Our implementation of the proposed controller, the simulation environment and detailed results is open-sourced on https://github.com/droneslab/.

Urban Air Mobility (UAM) promises a new dimension to decongested, safe, and fast travel in urban and suburban hubs. These UAM aircraft are conceived to operate from small airports called vertiports each comprising multiple take-offllanding and battery-recharging spots. Since they might be situated in dense urban areas and need to handle many aircraft landings and take-offs each hour, managing this schedule in real-time becomes challenging for a traditional air-traffic controller but instead calls for an automated solution. This paper provides a novel approach to this problem of Urban Air Mobility - Vertiport Schedule Management (UAM-VSM), which leverages graph reinforcement learning to generate decision-support policies. Here the designated physical spots within the vertiport's airspace and the vehicles being managed are represented as two separate graphs, with feature extraction performed through a graph convolutional network (GCN). Extracted features are passed onto perceptron layers to decide actions such as continue to hover or cruise, continue idling or take-off, or land on an allocated vertiport spot. Performance is measured based on delays, safety (no. of collisions) and battery consumption. Through realistic simulations in AirSim applied to scaled down multi-rotor vehicles, our results demonstrate the suitability of using graph reinforcement learning to solve the UAM-VSM problem and its superiority to basic reinforcement learning (with graph embed dings) or random choice baselines.

Mapping 3D airflow fields is important for many HVAC, industrial, medical, and home applications. However, current approaches are expensive and time-consuming. We present Anemoi, a sub-$100 drone-based system for autonomously mapping 3D airflow fields in indoor environments. Anemoi leverages the effects of airflow on motor control signals to estimate the magnitude and direction of wind at any given point in space. We introduce an exploration algorithm for selecting optimal waypoints that minimize overall airflow estimation uncertainty. We demonstrate through microbenchmarks and real deployments that Anemoi is able to estimate wind speed and direction with errors up to 0.41 m/s and 25.1° lower than the existing state of the art and map 3D airflow fields with an average RMS error of 0.73 m/s.

Localization in urban environments is becoming increasingly important and used in tools such as ARCore [ 18 ], ARKit [ 34 ] and others. One popular mechanism to achieve accurate indoor localization and a map of the space is using Visual Simultaneous Localization and Mapping (Visual-SLAM). However, Visual-SLAM is known to be resource-intensive in memory and processing time. Furthermore, some of the operations grow in complexity over time, making it challenging to run on mobile devices continuously. Edge computing provides additional compute and memory resources to mobile devices to allow offloading tasks without the large latencies seen when offloading to the cloud. In this article, we present Edge-SLAM, a system that uses edge computing resources to offload parts of Visual-SLAM. We use ORB-SLAM2 [ 50 ] as a prototypical Visual-SLAM system and modify it to a split architecture between the edge and the mobile device. We keep the tracking computation on the mobile device and move the rest of the computation, i.e., local mapping and loop closing, to the edge. We describe the design choices in this effort and implement them in our prototype. Our results show that our split architecture can allow the functioning of the Visual-SLAM system long-term with limited resources without affecting the accuracy of operation. It also keeps the computation and memory cost on the mobile device constant, which would allow for the deployment of other end applications that use Visual-SLAM. We perform a detailed performance and resources use (CPU, memory, network, and power) analysis to fully understand the effect of our proposed split architecture.

Visual SLAM is a long-standing research area with many significant advances over the years. New systems typically build on previous contributions, but this requires significant development overhead, a highly detailed understanding of previous system implementations, and is rife with programming pitfalls. To enable fast experimentation and reduce the need for researchers to re-invent the wheel, we propose an extensible Visual SLAM framework with three features: modularity, seamless edge offloading, and safe concurrency.

This paper presents a novel multi-robot coverage path planning (CPP) algorithm - aka SCoPP - that provides a time-efficient solution, with workload balanced plans for each robot in a multi-robot system, based on their initial states. This algorithm accounts for discontinuities (e.g., no-fly zones) in a specified area of interest, and provides an optimized ordered list of way-points per robot using a discrete, computationally efficient, nearest neighbor path planning algorithm. This algorithm involves five main stages, which include the transformation of the user’s input as a set of vertices in geographical coordinates, discretization, load-balanced partitioning, auctioning of conflict cells in a discretized space, and a path planning procedure. To evaluate the effectiveness of the primary algorithm, a multi-unmanned aerial vehicle (UAV) post-flood assessment application is considered, and the performance of the algorithm is tested on three test maps of varying sizes. Additionally, our method is compared with a state-of-the-art method created by Guasella et al. Further analyses on scalability and computational time of SCoPP are conducted. The results show that SCoPP is superior in terms of mission completion time; its computing time is found to be under 2 mins for a large map covered by a 150-robot team, thereby demonstrating its computationally scalability.

Unmanned Aerial Vehicles (UAVs) are an emerging computation platform known for their safety-critical need. In this paper, we conduct an empirical study on a widely used open-source UAV software framework, Paparazzi, with the goal of understanding the safety-critical concerns of UAV software from a bottom-updeveloper-in-the-fieldperspective. We set our focus on the use of Bounding Functions (BFs), the runtime checks injected by Paparazzi developers on the range of variables. Through an in-depth analysis on BFs in the Paparazzi autopilot software, we found a large number of them (109 instances) are used to bound safety-critical variables essential to the cyber-physical nature of the UAV, such as its thrust, its speed, and its sensor values. The novel contributions of this study are two fold. First, we take a static approach to classify all BF instances, presenting a noveldatatype-based5-category taxonomy with fine-grained insight on the role of BFs in ensuring the safety of UAV systems. Second, we dynamically evaluate the impact of the BF uses through adifferentialapproach, establishing the UAV behavioral difference with and without BFs. The two-pronged static and dynamic approach together illuminates a rarely studied design space of safety-critical UAV software systems.

UAVs are deployed in various applications including disaster search-and-rescue, precision agriculture, law enforcement and first response. As UAV software systems grow more complex, the drawbacks of developing them in low-level languages become more pronounced. For example, the lack of memory safety in C implies poor isolation between the UAV autopilot and other concurrent tasks. As a result, the most crucial aspect of UAV reliability-timely control of the flight-could be adversely impacted by other tasks such as perception or planning. We introduce JCopter, an autopilot framework for UAVs developed in a managed language, i.e., a high-level language with built-in safe memory and timing management. Through detailed simulation as well as flight testing, we demonstrate how JCopter retains the timeliness of C-based autopilots while also providing the reliability of managed languages.

We present Rushmore, a system that securely displays static or animated images using TrustZone. The core functionality of Rushmore is to securely decrypt and display encrypted images (sent by a trusted party) on a mobile device. Although previous approaches have shown that it is possible to securely display encrypted images using TrustZone, they exhibit a critical limitation that significantly hampers the applicability of using TrustZone for display security. The limitation is that, when the trusted domain of TrustZone (the secure world) takes control of the display, the untrusted domain (the normal world) cannot display anything simultaneously. This limitation comes from the fact that previous approaches give the secure world exclusive access to the display hardware to preserve security. With Rushmore, we overcome this limitation by leveraging a well-known, yet overlooked hardware feature called an IPU (Image Processing Unit) that provides multiple display channels. By partitioning these channels across the normal world and the secure world, we enable the two worlds to simultaneously display pixels on the screen without sacrificing security. Furthermore, we show that with the right type of cryptographic method, we can decrypt and display encrypted animated images at 30 FPS or higher for medium-to-small images and at around 30 FPS for large images. One notable cryptographic method we adapt for Rushmore is visual cryptography, and we demonstrate that it is a light-weight alternative to other cryptographic methods for certain use cases. Our evaluation shows that in addition to providing usable frame rates, Rushmore incurs less than 5% overhead to the applications running in the normal world.

Human-swarm interaction has recently gained attention due to its plethora of new applications in disaster relief, surveillance, rescue, and exploration. However, if the task difficulty increases, the performance of the human operator decreases, thereby decreasing the overall efficacy of the human-swarm team. Thus, it is critical to identify the task difficulty and adaptively allocate the task to the human operator to maintain optimal performance. In this direction, we study the classification of task difficulty in a human-swarm interaction experiment performing a target search mission. The human may control platoons of unmanned aerial vehicles (UAVs) and unmanned ground vehicles (UGVs) to search a partially observable environment during the target search mission. The mission complexity is increased by introducing adversarial teams that humans may only see when the environment is explored. While the human is completing the mission, their brain activity is recorded using an electroencephalogram (EEG), which is used to classify the task difficulty. We have used two different approaches for classification: A feature-based approach using coherence values as input and a deep learning-based approach using raw EEG as input. Both approaches can classify the task difficulty well above the chance. The results showed the importance of the occipital lobe (O1 and O2) coherence feature with the other brain regions. Moreover, we also study individual differences (expert vs. novice) in the classification results. The analysis revealed that the temporal lobe in experts (T4 and T3) is predominant for task difficulty classification compared with novices.

Many existing SLAM approaches rely on the assumption of static environments for accurate performance. However, several robot applications require them to traverse repeatedly in semi-static or dynamic environments. There has been some recent research interest in designing persistence filters to reason about persistence in such scenarios. Our goal in this work is to incorporate such persistence reasoning in visual SLAM. To this end, we incorporate persistence filters [1] into ORB-SLAM, a well-known visual SLAM algorithm. We observe that the simple integration of their proposal results in inefficient persistence reasoning. Through a series of modifications and using two locally collected datasets, we demonstrate the utility of such persistence filtering as well as our customizations in ORB-SLAM. Overall, incorporating persistence filtering could result in a significant reduction in map size (about 30% in the best case) and a corresponding reduction in run-time while retaining similar accuracy to methods that use much larger maps.

The recent advances in mobile devices have allowed them to run spatial sensing algorithms such as Visual Simultaneous Localization and Mapping (Visual-SLAM). However, the resource requirements of Visual-SLAM prevents long-operation of such algorithm on mobile devices. We demonstrate Edge-SLAM [2], a system that adapts edge computing into Visual-SLAM through a split architecture. Edge-SLAM offloads the compute-intensive modules of Visual-SLAM to the edge without losing accuracy. Our experiments show that Edge-SLAM architecture reduces the use of computation and memory resources on mobile devices and keeps it constant. Thus, enabling long-operation of Visual-SLAM along with other applications services on mobile devices.

Localization in urban environments is becoming increasingly important and used in tools such as ARCore [11], ARKit [27] and others. One popular mechanism to achieve accurate indoor localization as well as a map of the space is using Visual Simultaneous Localization and Mapping (Visual-SLAM). However, Visual-SLAM is known to be resource-intensive in memory and processing time. Further, some of the operations grow in complexity over time, making it challenging to run on mobile devices continuously. Edge computing provides additional compute and memory resources to mobile devices to allow offloading of some tasks without the large latencies seen when offloading to the cloud. In this paper, we present Edge-SLAM, a system that uses edge computing resources to offload parts of Visual-SLAM. We use ORB-SLAM2 as a prototypical Visual-SLAM system and modify it to a split architecture between the edge and the mobile device. We keep the tracking computation on the mobile device and move the rest of the computation, i.e., local mapping and loop closure, to the edge. We describe the design choices in this effort and implement them in our prototype. Our results show that our split architecture can allow the functioning of the Visual-SLAM system long-term with limited resources without affecting the accuracy of operation. It also keeps the computation and memory cost on the mobile device constant which would allow for deployment of other end applications that use Visual-SLAM.

Human-Swarm interaction has attracted a lot of attention for their applications in areas such as exploration, rescue, surveillance, and interplanetary exploration. When humans assume a supervisory or tactician role in managing the robot swarm, the humans' (physiological) state significantly affects the mission performance. In this work, we explore the physiological correlates with the user's tactical decisions in a simulated search and rescue mission. The mission consists of supervising three groups of unmanned aerial vehicles and three groups of unmanned ground vehicles to search for a target building. The mission complexity is increased by introducing static adversarial teams.Due to the adversarial team's presence, the user should employ different tactics to search for a target. While the user interacts with the swarm, brain activity in forms of electroencephalogram (EEG) and eye movements are recorded. 20 participants, with prior experience in playing real-time strategy games, took part in the study. A linear mixed effect model is used to study the correlated physiological features and tactical decisions. Six features are extracted from the physiological data: engagement level, mental workload, Fz-Pz coherence, Fz-O1 coherence, pupil size, and the number of gaze fixations. The results show that mental engagement and Fz-O1 coherence are the important factors in predicting the tactical decisions. Specifically, Fz-O1 coherence in Beta (22.5-30 Hz) and Gamma (38-42 Hz) band is found to be significant.

Spatial sensing is a fundamental requirement for applications in robotics and augmented reality. In urban spaces such as malls, airports, apartments, and others, it is quite challenging for a single robot to map the whole environment. So, we employ a swarm of robots to perform the mapping. One challenge with this approach is merging sub-maps built by each robot. In this work, we use wireless access points, which are ubiquitous in most urban spaces, to provide us with coarse orientation between sub-maps, and use a custom ICP algorithm to refine this orientation to merge them. We demonstrate our approach with maps from a building on campus and evaluate it using two metrics. Our results show that, in the building we studied, we can achieve an average Absolute Trajectory error of 0.2m in comparison to a map created by a single robot and average Root Mean Square mapping error of 1.3m from ground truth landmark locations.

This paper proposes Mimic, an automated UI compatibility testing system for Android apps. Mimic is designed specifically for comparing the UI behavior of an app across different devices, different Android versions, and different app versions. This design choice stems from a common problem that Android developers and researchers face-how to test whether or not an app behaves consistently across different environments or internal changes. Mimic allows Android app developers to easily perform backward and forward compatibility testing for their apps. It also enables a clear comparison between a stable version of app and a newer version of app. In doing so, Mimic allows multiple testing strategies to be used, such as randomized or sequential testing. Finally, Mimic programming model allows such tests to be scripted with much less developer effort than other comparable systems. Additionally, Mimic allows parallel testing with multiple testing devices and thereby speeds up testing time. To demonstrate these capabilities, we perform extensive tests for each of the scenarios described above. Our results show that Mimic is effective in detecting forward and backward compatibility issues, and verify runtime behavior of apps. Our evaluation also shows that Mimic significantly reduces the development burden for developers.

Trusted Applications (TAs) written for Trusted Execution Environments (TEEs) using ARM TrustZone are currently written in C; there is limited support for higher-level languages. This leads to common manual memory management problems like buffer overflow and use-after-free. Higher-level languages, which have managed runtimes, allow for automated memory management, the benefits of which are widely accepted. To allow for automated memory management of TAs, we need to have a runtime that handles allocation and garbage collection (GC). However, having the entire allocator and GC in the secure world would increase the Trusted Computing Base (TCB) of the secure world. We propose TrustGC, a mechanism to partition garbage collection and allocation between the secure world and the normal world. TrustGC allows for automated memory management of TAs by leveraging the help of a GC partly running in the normal world.

Advances in hardware, software and sensing are bringing swarms of robots to daily life. A major challenge in enabling such applications is multi-robot coordination. Most multi-robot coordination algorithms are developed under the assumption of perfect communication, which does not hold in practical wireless networks. To understand the consequences of this, we investigate the performance of a representative task allocation algorithm for multi-robot systems, namely, the Asynchronous Consensus Based Bundle Algorithm (ACBBA), in realistic network conditions. We show that the ACBBA deviates from its desired theoretical behavior when deployed in a realistic network. This manifests in the form of redundant task assignments across agents, which violates the algorithm's "conflict-free" assignment constraint and degrades the task allocation efficiency. We explore several network-based mitigations to this problem.

Rapid advancements in computing, communication, sensing, and actuation have seen the growth of Internet of Things (IoT) devices in our daily life. One of the fundamental constraints of a typical IoT device is energy as IoT devices rely on a battery. Therefore, it is crucial for their operating system (OS) to be able to accurately account for system-wide energy usage. Specifically, the OS should be able to attribute such accounted energy to the running applications accurately. Traditional OSs have limited capability when it comes to tracking components such as sensors, actuators and network interfaces, as they are often used in an asynchronous fashion. This would make it difficult to conduct energy accounting accurately. This paper proposes a new mechanism to accurately account for the asynchronous energy usage of resources in mobile systems and IoT devices. Our insight is that by accurately relating the application requests with kernel requests to device and corresponding device responses, we can accurately attribute time of use to the requesting process. However, resources such as WiFi reception violate this assumption. In such cases, we can measure usage by the number of bytes in each individual transaction. Using such a hybrid approach, we can account for energy usage with 94% accuracy and perform much better than using each of these models individually.

Gesto is a system that enables task automation for Android apps using gestures and voice commands. Using Gesto, a user can record a UI action sequence for an app, choose a gesture or a voice command to activate the UI action sequence, and later trigger the UI action sequence by the corresponding gesture/voice command. Gesto enables this for existing Android apps without requiring their source code or any help from their developers. In order to make such capability possible, Gesto combines bytecode instrumentation and UI action record-and-replay. To show the applicability of Gesto, we develop four use cases using real apps downloaded from Google Play-Bing, Yelp, AVG Cleaner, and Spotify. For each of these apps, we map a gesture or a voice command to a sequence of UI actions. According to our measurement, Gesto incurs modest overhead for these apps in terms of memory usage, energy usage, and code size increase. We evaluate our instrumentation capability and overhead using 1,000 popular apps downloaded from Google Play. Our result shows that Gesto is able to instrument 94.9% of the apps without any significant overhead. In addition, since our prototype currently supports 6 main UI elements of Android, we evaluate our coverage and measure what percentage of UI element uses we can cover. Our result shows that our 6 UI elements can cover 96.4% of all statically-declared UI element uses in the 1,000 Google Play apps.

Time predictability is difficult to achieve in the complex, layered execution environments that are common in modern embedded devices such as smartphones. We explore adopting the Android programming model for a range of embedded applications that extends beyond mobile devices, under the constraint that changes to widely used libraries should be minimized. The challenges we explore include the interplay between real-time activities and the rest of the system, how to express the timeliness requirements of components, and how well those requirements can be met on stock embedded platforms. We detail the design and implementation of our modifications to the Android framework along with a real-time VM and OS, and we provide experimental data validating feasibility over five applications.

Mobile systems face privacy challenges including coarsegrained privacy control and the inability to distinguish private and public files. We propose System-E, a novel system which can enhance the user privacy on mobile systems (e.g., Android) by (1) enabling users to set finer grained permissions for apps accessing data, and (2) enabling automatic classification of data (e.g., photos) at the storage layer (e.g., by using deep learning) to prevent potentially sensitive data from being stored/accessed with open permissions.

ARTY is a week-long program for middle school students to teach them programming of robots and allow them to express themselves artistically. It was started in 2013 and ran its fourth edition in 2016. We describe the ideas behind the inception of this program, its curriculum, our experiences during the 2016 workshop and challenges/future directions for the program. Our primary intent in this paper is to convey the program curriculum and its design, including the way in which robots can be viewed as vehicles for artistic expression. Some results from a brief attitudinal survey that was administered before and after the workshop are also included along with a discussion of outcomes assessment and issues.

Advancements in the design of drones have led to their use in varied environments and applications such as battle field surveillance. In such scenarios, swarms of drones can coordinate to survey a given area. We consider the problem of covering an arbitrary area containing obstacles using multiple drones, i.e., the so-called coverage path planning (CPP) problem. The goal of the CPP problem is to find paths for each drone such that the entire area is covered. However, a major limitation in such deployments is drone flight time. To most efficiently use a swarm, we propose to minimize the maximum energy consumption among all drones' flight paths. We perform measurements to understand energy consumption of a drone. Using these measurements, we formulate an Energy Efficient Coverage Path Planning (EECPP) problem. We solve this problem in two steps: a load-balanced allocation of the given area to individual drones, and a minimum energy path planning (MEPP) problem for each drone. We conjecture that MEPP is NP-hard as it is similar to the Traveling Salesman Problem (TSP). We propose an adaptation of the well-known Lin-Kernighan heuristic for the TSP to efficiently solve the problem. We compare our solution to the recently proposed depth-limited search with back tracking algorithm, the optimal solution, and rastering as a baseline. Results show that our algorithm is more computationally efficient and provides more energy-efficient solutions compared to the other heuristics.

We are developing an extensible, open-source framework that can localize and track rigid bodies using a network of cameras (both RGB and depth). This system is motivated by two design goals - (i) ease of setup, and (ii) ability to be agnostic of individual cameras and recognition algorithms. The goal of this implementation is to be a poor man's motion capture system that can be quickly set up for experimentation and provide accurate 3-D pose of the rigid body and scalable across cameras and the volume of coverage.

Simultaneous Localization and Mapping (SLAM) is the process of learning about both the environment and about a robot's location with respect to the environment and is essential for robots to autonomously navigate. A variety of algorithms using many different sensors such as RGB-D cameras, laser range finders, ultrasonic sensors and others have been proposed to perform SLAM. However, these algorithms face common challenges are that of computational complexity, wrong loop closure detection and failure to localize correctly when robot loses state (kidnapped robot problem). In this work, we utilize Wi-Fi signal strength sensing to aid the SLAM process in indoor environments and address the challenges mentioned above.

The RoboBee is a novel insect-scale flapping wing Micro Aerial Vehicle that is envisioned to enable exciting applications. While recent results have demonstrated full control as well as biomimetic behaviors such as perching, more complex challenges such as perception and navigation still exist. Typically, challenges in perception-based control can only be solved by experimentation. However, such fly-size MAVs are not widely available to researchers at large due to its intricate manufacturing process and limited mechanical lifetime. To facilitate the development of perception and control algorithms of insect-scale MAVs, we explore an approach of simulating flapping wing aerial vehicle dynamics on a quad-rotor. This work performs detailed analysis of the transformation of control inputs, and demonstrates feasibility by numerically simulating basic flight patterns of models of a RoboBee as well as that of a scaled quad-rotor.

In this paper, we design a pluggable data management solution for modern mobile platforms (e.g., Android). Our goal is to allow data management mechanisms and policies to be implemented independently of core app logic. Our design allows a user to install data management solutions as apps, install multiple such solutions on a single device, and choose a suitable solution each for one or more apps. It allows app developers to focus their effort on app logic and helps the developers of data management solutions to achieve wider deployability. It also gives increased control of data management to end users and allows them to use different solutions for different apps. We present a prototype implementation of our design called BlueMountain, and implement several data management solutions for file and database management to demonstrate the utility and ease of using our design. We perform detailed microbenchmarks as well as end-to-end measurements for files and databases to demonstrate the performance overhead incurred by our implementation.

BlueMountain is a system that enables building pluggable data management solutions which can be linked with any Android app at runtime, without requiring any modifications to the Android platform. BlueMountain simplifies the app development, provides flexibility to end users, and works with existing apps.

This poster presents Pixelsior, a new mobile platform service for photo data management in mobile apps.

This paper proposes a new technique that enables open innovation in mobile platforms. Our technique allows third-party developers to modify, instrument, or extend platform API calls and deploy their modifications seamlessly. The uniqueness of our technique is that it enables modifications completely at the app layer without requiring any platform-level changes. This allows practical openness---third parties can easily distribute their modifications for a platform without the need to update the entire platform. To demonstrate the benefits of our technique, we have developed a prototype on Android called Reptor and used it to instrument real-world apps with novel functionality. Our evaluation in realistic scenarios shows that Reptor has little overhead in performance and energy, and only modest overhead in memory usage that ranges from 0.6% to 10% for the observed worst cases.

We demonstrate AutoClicker, a fully automated UI testing system for large-scale Android apps using multiple devices. It provides a way to quickly and easily verify that a large number of Android apps behave correctly at runtime in a repeatable manner.

We demonstrate Reptor, a bytecode instrumentation tool enabling API virtualization on Android. It provides a general way to alter functionality of platform APIs on Android. With Reptor, third-party developers can modify the behavior of platform APIs according to their needs. All modifications are completely at the app layer without modifying the underlying platform. This allows practical openness---third-party developers can easily distribute their modifications for a platform without the need to update the entire platform.

We demonstrate Gesto, a dynamic gesture mapping tool. It provides users to map any gesture to a certain UI event that the users need. Also, the mapping can be easily changed by users.

Since the introduction of the smartphone, mobile computing has become pervasive in our society. Meanwhile, Mobile devices have evolved far beyond the stereotypical personal devices and been employed in various traditional real-time embedded domains. Of the currently available mobile systems, Android has seen the most widespread deployment outside of the consumer electronics market. Its open source nature has prompted its ubiquitous adoption in sensing, medical, robotics, and autopilot applications. However, it is not surprising that Android does not provide any real-time guarantee since it is designed as a mobile system and optimised for mobility, user experience, and energy efficiency. Although there has been much interest in adopting Android in real-time contexts, surprisingly little work has been done to examine the suitability of Android for real-time systems. Existing work only provides solutions to traditional problems, including real-time garbage collection at the virtual machine layer, real-time OS scheduling and resource management. While it is critical to address these issues, it is by no means sufficient. After all, Android is a vast system that is more than a Java virtual machine and a kernel.

Time predictability is difficult to achieve in the complex, layered execution environments that are common in modern embedded devices. We consider the possibility of adopting the Android programming model for a range of embedded applications that extends beyond mobile devices, under the constraint that changes to widely used libraries should be minimized. The challenges we explore include: the interplay between real-time activities and the rest of the system, how to express the timeliness requirements of components, and how well those requirements can be met on stock embedded platforms. We report on the design and implementation of an Android virtual machine with soft-real-time support, and provides experimental data validating feasibility over three case studies.

One essential functionality of a modern operating system is to accurately account for the resource usage of the underlying hardware. This is especially important for computing systems that operate on battery power, since energy management requires accurately attributing resource uses to processes. However, components such as sensors, actuators and specialized network interfaces are often used in an asynchronous fashion, and makes it difficult to conduct accurate resource accounting. For example, a process that makes a request to a sensor may not be running on the processor for the full duration of the resource usage; and current mechanisms of resource accounting fail to provide accurate accounting for such asynchronous uses. This paper proposes a new mechanism to accurately account for the asynchronous usage of resources in mobile systems. Our insight is that by accurately relating the user requests with kernel requests to device and corresponding device responses, we can accurately attribute resource use to the requesting process. Our prototype implemented in Linux demonstrates that we can account for the usage of asynchronous resources such as GPS and WiFi accurately.

This study describes the design and implementation of several bioinspired algorithms for providing guidance to an ultra-lightweight micro-aerial vehicle (MAV) using a 2.6 g omnidirectional vision sensor. Using this visual guidance system we demonstrate autonomous speed control, centring, and heading stabilisation on board a 30 g MAV flying in a corridor-like environment. In addition to the computation of wide-field optic flow, the comparatively high-resolution omnidirectional imagery provided by this sensor also offers the potential for image-based algorithms such as landmark recognition to be implemented in the future.

Today's mobile applications operate in a diverse set of environments, where it is difficult for a developer to know beforehand what conditions his or her application will be put under. For example, once deployed on an online application store, an application can be downloaded on different types of hardware, ranging from budget smartphones to high-end tablets. In addition, network conditions can vary widely from Wi-Fi to 3G to 4G. Mobile applications also need to co-exist with other applications that compete for resources at different times.

Micro-aerial vehicle (MAV) swarms are an emerging class of mobile sensing systems. Simulation and staged deployment to prototype testbeds are useful in the early stages of large-scale system design, when hardware is unavailable or deployment at scale is impractical. To faithfully represent the problem domain, a MAV swarm simulator must be able to model the key aspects of the system: actuation, sensing, and communication. We present Simbeeotic, a simulation framework geared toward modeling swarms of MAVs. Simbeeotic enables algorithm development and rapid MAV prototyping through pure simulation and hardware-in-the-loop experimentation. We demonstrate that Simbeeotic provides the appropriate level of fidelity to evaluate prototype systems while maintaining the ability to test at scale.

Micro-aerial vehicle (MAV) swarms are an emerging class of mobile sensing systems. Designing the next generation of such swarms requires the ability to rapidly test algorithms, sensors, and support infrastructure at scale. Simulation is useful in the early stages of such large-scale system design, when hardware is unavailable or deployment at scale is impractical. To faithfully represent the problem domain, an MAV swarm simulator must be able to model all key aspects of the system: actuation, sensing, and communication. Further, it is important to be able to quickly test swarm behavior using different control algorithms in a varied set of environments, and with a variety of sensors.

We compare our previously developed deterministic [7] and stochastic [3], [4] strategies for allocating tasks in robotic swarms1 consisting of very large populations of highly resource-constrained robots. We study our two task allocation approaches in a simulated scenario in which a collective of insect-inspired micro-aerial vehicles (MAVs) must produce a specified spatial distribution of pollination activity over a crop field. We investigate the approaches' requirements, advantages, and disadvantages under realistic conditions of error in robot localization, navigation, and sensing in simulation. Our results show that the deterministic approach, which requires region-based robot navigation, yields higher task progress in all cases. For robots without such navigation capabilities, the stochastic approach is a feasible alternative, and its resulting task progress is less sensitive to error in localization, error in navigation, and a combination of high error in localization, navigation, and sensing.

Research in micro-aerial vehicle (MAV) construction, control, and high-density power sources is enabling swarms of MAVs as a new class of mobile sensing systems. For efficient operation, such systems must adapt to dynamic environments, cope with uncertainty in sensing and control, and operate with limited resources. We propose a novel system architecture based on a hive-drone model that simplifies the functionality of an individual MAV to a sequence of sensing and actuation commands with no in-field communication. This decision simplifies the hardware and software complexity of individual MAVs and moves the complexity of coordination entirely to a central hive computer. We present Karma, a system for programming and managing MAV swarms. Through simulation and testbed experiments we demonstrate how applications in Karma can run on limited resources, are robust to individual MAV failure, and adapt to changes in the environment.

Relative bearing between robots is important in applications like pursuit-evasion [11] and SLAM [7]. This is also true in sensor networks, where the bearing of one sensor node relative to another has been used for localization [5], [18], [20] and topology control [14], [21], [6]. Most systems use dedicated sensors like an IR array or a camera to obtain relative bearing. We study the use of radio signal strength (RSS) in commodity radios for obtaining relative bearing. We show that by using the robot's mobility, commodity radios can be used to obtain coarse relative bearing. This measurement can be used for a suite of applications that do not require very precise bearing measurement. We analyze signal strength variations in simulation and experiment and also show an algorithm that uses this coarse bearing computation in a practical setting.

We introduce an algorithm which detects and traces a specified level set of a scalar field (a contour) on a plane. A network of static sensor nodes with limited communication and processing are deployed in a planar environment along with a mobile node which can both sense and move. As the mobile node moves through the environment, it computes the local spatial gradient of the field by communicating with its immediate neighbors in the static sensor network. The algorithm causes the mobile node to perform gradient descent on the scalar field till it arrives at a location on the desired contour. From this point onwards, the algorithm drives the mobile node to trace the desired contour without departing from it. Experiments in simulation indicate that the required contour is found with reasonable accuracy (between 80-90%) for networks with node degree of greater than or equal to six. Our results also indicate that the paths generated by our algorithm are near-optimal in terms of the distance traversed by the mobile node. Our preliminary experimental results with a physical robot show that our algorithm is feasible.

Severe energy limitations, and a paucity of computation pose a set of difficult design challenges for sensor networks. Recent progress in two seemingly disparate research areas namely, distributed robotics and low power embedded systems has led to the creation of mobile (or robotic) sensor networks. Autonomous node mobility brings with it its own challenges, but also alleviates some of the traditional problems associated with static sensor networks. We illustrate this by presenting the design of the robomote, a robot platform that functions as a single mobile node in a mobile sensor network. We briefly describe two case studies where the robomote has been used for table top experiments with a mobile sensor network.

Structural health monitoring (SHM) is a well-established multi-disciplinary research field. The goal of SHM is to develop technologies and techniques to automatically detect, localize, and classify damages in large structures (ships, bridges, aircraft and buildings). The state of the art in SHM relies on collecting response of these structures to ambient phenomena such as wind, passing vehicles or earthquakes at various points in the structure (either via manual inspections or expensive wired data acquisition systems) to be analyzed centrally. In our demonstration we will show a proof of concept working model of an automated distributed damage detection system using a sensor-actuator network.

No abstract available.

One of the main design constraints in mobile ad hoc networks (MANETs) is that they are power constrained. Hence, every effort is to be channeled towards reducing power. More precisely, network lifetime is a key design metric in MANETs. Since every node has to perform the functions of a router, if some nodes die early due to lack of energy, it will not be possible for other nodes to communicate with each other. Hence, the network will get disconnected and the network lifetime will be adversely affected. This paper presents a lifetime prediction routing protocol for MANETs that maximizes the network lifetime by finding routing solutions that minimize the variance of the remaining energies of the nodes in the network. Although this scheme introduces some additional traffic, simulations show that it improves the network lifetime by about 20-30%.

This paper describes a dynamic voltage and frequency scaling (DVFS) technique for MPEG decoding to reduce the energy consumption while maintaining a quality of servic(QoS) constraint. The computational workload for an incoming frame is predicted using a frame-based history so that the processor voltage and frequency can be scaled to provide the exact amount of computing power needed to decode the frame. More precisely, the required decoding time for each frame is separated into two parts: a frame-dependent (FD) part and a frame-independent (FI) part. The FD part varies greatly according to the type of the incoming frame whereas the FI part remains constant regardless of the frame type. In the DVFS scheme presented in this paper the FI part is used to compensate for the prediction error that may occur during the FD part such that a significant amount of energy can be saved while meeting the frame rate requirement. The proposed DVFS algorithm has been implemented on a StrongArm-1110 based evaluation board. Measurement results demonstrate a higher than 50% CPU energy saving as a result of DVFS.

Ad hoc wireless networks are power constrained since nodes operate with limited battery energy. To maximize the lifetime of these networks (defined by the condition that a fixed percentage of the nodes in the network "die out" due to lack of energy), network-related transactions through each mobile node must be controlled such that the power dissipation rates of all nodes are nearly the same. Assuming that all nodes start with a finite amount of battery capacity and that the energy dissipation per bit of data and control packet transmission or reception is known, this paper presents a new source-initiated (on-demand) routing protocol for mobile ad hoc networks that increases the network lifetime. Simulation results show that the proposed power-aware source routing protocol has a higher performance than other source initiated routing protocols in terms of the network lifetime.