Reconstructing realistic digital twins has become crucial as advances in mixed reality, metaverse, and robotics demand more accurate simulations for the physical world. Despite technical progress, building high-fidelity digital twins from a systematic and human-centered perspective remains underexplored. Drawing from the human processing model, we decompose human-centric reality into perception, motion, and cognition, and define a reality-preserving digital twin (RPDT) as a reconstruction integrating these dimensions. We present RealTwin, an attribute-graph-based representation and inference framework for RPDT. Leveraging the grounding capabilities of Multimodal Large Language Models (MLLMs), RealTwin chains AI tools to construct attribute graphs that faithfully encode real-world properties. We validate RealTwin through both technical evaluation, showing promising success in graph parsing and attribute inference, and a user study, assessing its applicability across diverse user groups. Enlightened by RealTwin, we discuss critical issues, including ecology, interaction space, and real-world adoption, for future end-to-end, fine-grained, and scalable digital twin reconstruction.

Neural fields (NFs) map continuous coordinates to signals such as color or density, but fast high-quality reconstruction from sparse observations remains difficult. Classical Neural Tangent Kernel (NTK) regression gives closed-form fits, yet it is fundamentally linear and cannot accumulate reusable task priors. We develop three algorithms that address these gaps. NTK-KIP learns a distilled support set of coordinates (and optional labels) so that a finite NTK can inpaint large missing regions from little observed data, yielding a compact non-linear representation instead of a raw kernel solve. MetaQuill meta-learns a shared initialization for an INR so that new scenes can be adapted by updating only a small task-specific weight offset, which provides true feature learning and a reusable prior. Finally, MetaQuill-KIP fuses both ideas: it seeds the task with a KIP-style non-linear warm start, then refines only that small offset around the meta-learned initialization. MetaQuill-KIP achieves high-PSNR reconstructions and semantically plausible inpainting under very sparse observations, while requiring only lightweight per-instance adaptation, whereas diffusion-style baselines typically depend on large pretrained generative priors and costly per-image tuning. This shows that NTK-driven neural fields can be made both non-linear and meta-learnable, narrowing the gap between analytic kernels and practical few-shot reconstruction.

This paper challenges the prevailing view that convolutional neural network (CNN) filters become increasingly specialized in deeper layers. Motivated by recent observations of clusterable repeating patterns in depthwise separable CNNs (DS-CNNs) trained on ImageNet, we extend this investigation across various domains and datasets. Our analysis of DS-CNNs reveals that deep filters maintain generality, contradicting the expected transition to class-specific features. We demonstrate the generalizability of these filters through transfer learning experiments, showing that frozen filters from models trained on different datasets perform well and can be further improved when sourced from larger, better-performing models. Our findings indicate that spatial features learned by depthwise separable convolutions remain generic across all layers, domains, and architectures. This research provides new insights into the nature of generalization in neural networks, particularly in DS-CNNs, and has significant implications for transfer learning and model design.

Zhaoxuan Wu, Zijian Zhou, Arun Verma, Alok Prakash, Daniela Rus, Bryan Kian Hsiang Low. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.

CHI ’25, Yokohama, Japan

Diffusion models have shown promise in data-driven planning. While these planners are commonly employed in applications where decisions are critical, they still lack established safety guarantees. In this paper, we address this limitation by introducing SafeDiffuser, a method to equip diffusion models with safety guarantees via control barrier functions. The key idea of our approach is to embed finite-time diffusion invariance, i.e., a form of specification consisting of safety constraints, into the denoising diffusion procedure. This way we enable data generation under safety constraints. We show that SafeDiffusers maintain the generative performance of diffusion models while also providing robustness in safe data generation. We evaluate our method on a series of tasks, including maze path generation, legged robot locomotion, and 3D space manipulation, and demonstrate the advantages of robustness over vanilla diffusion models.

Simulation plays a key role in scaling robot learning and validating policies, but constructing simulations remains labor-intensive. In this paper, we introduce ReGen, a generative simulation framework that automates this process using inverse design. Given an agent's behavior (such as a motion trajectory or objective function) and its textual description, we infer the underlying scenarios and environments that could have caused the behavior. Our approach leverages large language models to construct and expand a graph that captures cause-and-effect relationships and relevant entities with properties in the environment, which is then processed to configure a robot simulation environment. Our approach supports (i) augmenting simulations based on ego-agent behaviors, (ii) controllable, counterfactual scenario generation, (iii) reasoning about agent cognition and mental states, and (iv) reasoning with distinct sensing modalities, such as braking due to faulty GPS signals. We demonstrate our method in autonomous driving and robot manipulation tasks, generating more diverse, complex simulated environments compared to existing simulations with high success rates, and enabling controllable generation for corner cases. This approach enhances the validation of robot policies and supports data or simulation augmentation, advancing scalable robot learning for improved generalization and robustness.

We propose Linear Oscillatory State-Space models (LinOSS) for efficiently learning on long sequences. Inspired by cortical dynamics of biological neural networks, we base our proposed LinOSS model on a system of forced harmonic oscillators. A stable discretization, integrated over time using fast associative parallel scans, yields the proposed state-space model. We prove that LinOSS produces stable dynamics only requiring nonnegative diagonal state matrix. This is in stark contrast to many previous state-space models relying heavily on restrictive parameterizations. Moreover, we rigorously show that LinOSS is universal, i.e., it can approximate any continuous and causal operator mapping between time-varying functions, to desired accuracy. In addition, we show that an implicit-explicit discretization of LinOSS perfectly conserves the symmetry of time reversibility of the underlying dynamics. Together, these properties enable efficient modeling of long-range interactions, while ensuring stable and accurate long-horizon forecasting. Finally, our empirical results, spanning a wide range of time-series tasks from mid-range to very long-range classification and regression, as well as long-horizon forecasting, demonstrate that our proposed LinOSS model consistently outperforms state-of-the-art sequence models. Notably, LinOSS outperforms Mamba and LRU by nearly 2x on a sequence modeling task with sequences of length 50k.

Safe learning is central to AI-enabled robots where a single failure may lead to catastrophic results. Existing safe learning methods are not scalable, inefficient and hard to train, and tend to generate unstable signals under noisy inputs that are challenging to be deployed for robots. To address these challenges, we propose Adaptive explicit-Barrier Net (ABNet) in which barriers explicitly show up in the closed-form model that guarantees safety. The ABNet has the potential to incrementally scale toward larger safe foundation models. Each head of ABNet could learn safe control policies from different features and focuses on specific part of the observation. In this way, we do not need to directly construct a large model for complex tasks, which significantly facilitates the training of the model while ensuring its stable output. Most importantly, we can still formally prove the safety guarantees of the ABNet. We demonstrate the efficiency and strength of ABNet in 2D robot obstacle avoidance, safe robot manipulation, and vision-based end-to-end autonomous driving, with results showing much better robustness and guarantees over existing models.

Recent advancements in multimodal large language models have driven breakthroughs in visual question answering. Yet, a critical gap persists, `conceptualization'—the ability to recognize and reason about the same concept despite variations in visual form, a basic ability of human reasoning. To address this challenge, we introduce the Visual Graph Arena (VGA), a dataset featuring six graph-based tasks designed to evaluate and improve AI systems’ capacity for visual abstraction. VGA uses diverse graph layouts (e.g., Kamada-Kawai vs. planar) to test reasoning independent of visual form. Experiments with state-of-the-art vision models and multimodal LLMs reveal a striking divide: humans achieved near-perfect accuracy across tasks, while models totally failed on isomorphism detection and showed limited success in path/cycle tasks. We further identify behavioral anomalies suggesting pseudo-intelligent pattern matching rather than genuine understanding. These findings underscore fundamental limitations in current AI models for visual understanding. By isolating the challenge of representation-invariant reasoning, the VGA provides a framework to drive progress toward human-like conceptualization in AI visual models. The Visual Graph Arena is available at: \href{https://vga.csail.mit.edu/}{vga.csail.mit.edu}.

The deployment of autonomous vehicles controlled by machine learning techniques requires extensive testing in diverse real-world environments, robust handling of edge cases and out-of-distribution scenarios, and comprehensive safety validation to ensure that these systems can navigate safely and effectively under unpredictable conditions. Addressing Out-OfDistribution (OOD) driving scenarios is essential for enhancing safety, as OOD scenarios help validate the reliability of the models within the vehicle's autonomy stack. However, generating OOD scenarios is challenging due to their long-tailed distribution and rarity in urban driving datasets. Recently, Large Language Models (LLMs) have shown promise in autonomous driving, particularly for their zero-shot generalization and common-sense reasoning capabilities. In this paper, we leverage these LLM strengths to introduce a framework for generating diverse OOD driving scenarios. Our approach uses LLMs to construct a branching tree, where each branch represents a unique OOD scenario. These scenarios are then simulated in the CARLA simulator using an automated framework that aligns scene augmentation with the corresponding textual descriptions. We evaluate our framework through extensive simulations, and assess its performance via a diversity metric that measures the richness of the scenarios. Additionally, we introduce a new “OOD-ness” metric, which quantifies how much the generated scenarios deviate from typical urban driving conditions. Furthermore, we explore the capacity of modern Vision-Language Models (VLMs) to interpret and safely navigate through the simulated OOD scenarios. Our findings offer valuable insights into the reliability of language models in addressing OOD scenarios within the context of urban driving.

Swarmalators move as a function of their pairwise phase interactions, and control their phase as a function of their relative position or motion to other agents. This enables dual sync and swarm behaviors that mimic those exhibited by diverse natural and artificial swarms; these behaviors have almost entirely been explored only through computational simulations. Here, we realize through a 15-robot collective many of the predicted swarmalator behaviors when agents are chiral and non-chiral, when there is frequency coupling, and when the natural frequency distribution is homogeneous and heterogeneous. This work presents an experimental platform that can realize many theoretically predicted collective behaviors, it sheds light on the differences between the simulations and experiments, and it will serve in future studies to realize swarmalator and active matter collective behaviors.

Differentiable simulation has become a powerful tool for system identification. While prior work has focused on identifying robot properties using robot-specific data or object properties using object-specific data, our approach calibrates object properties by using information from the robot, without relying on data from the object itself. Specifically, we utilize robot joint encoder information, which is commonly available in standard robotic systems. Our key observation is that by analyzing the robot's reactions to manipulated objects, we can infer properties of those objects, such as inertia and softness. Leveraging this insight, we develop differentiable simulations of robot-object interactions to inversely identify the properties of the manipulated objects. Our approach relies solely on proprioception – the robot's internal sensing capabilities – and does not require external measurement tools or vision-based tracking systems. This general method is applicable to any articulated robot and requires only joint position information. We demonstrate the effectiveness of our method on a low-cost robotic platform, achieving accurate mass and elastic modulus estimations of manipulated objects with just a few seconds of computation on a laptop.

The development of novel autonomous underwater gliders has been hindered by limited shape diversity, primarily due to the reliance on traditional design tools that depend heavily on manual trial and error. Building an automated design framework is challenging due to the complexities of representing glider shapes and the high computational costs associated with modeling complex solid-fluid interactions. In this work, we introduce an AI-enhanced automated computational framework designed to overcome these limitations by enabling the creation of underwater robots with non-trivial hull shapes. Our approach involves an algorithm that cooptimizes both shape and control signals, utilizing a reducedorder geometry representation and a differentiable neural-network-based fluid surrogate model. This end-to-end design workflow facilitates rapid iteration and evaluation of hydrodynamic performance, leading to the discovery of optimal and complex hull shapes across various control settings. We validate our method through wind tunnel experiments and swimming pool gliding tests, demonstrating that our computationally designed gliders surpass manually designed counterparts in terms of energy efficiency. By addressing challenges in efficient shape representation and neural fluid surrogate models, our work paves the way for the development of highly efficient underwater gliders, with implications for long-range ocean exploration and environmental monitoring.

There is significant interest in creating compliant modular robots that can change their volume. Inspired by how biological cells move, these systems can potentially combine the resilience of modular robotics with the increased environmental interactions of soft robotics. However, current versions have limited speed, expansion, and portability. In this paper, we address these concerns through AuxSwarm, a compliant system composed of auxetic-based robotic voxels. These voxels control their volume through a scissor-like bi-layer auxetic design, growing up to 1.57 times their original size in 0.2 seconds. This combination of speed and expansion is unique across modular soft robots, enabling dynamic locomotion capabilities. We characterize the voxels and demonstrate the versatility of this approach through case studies of 2D bending and 3D cube flipping. AuxSwarm provides a first step towards addressable voxel-based smart materials, while simultaneously addressing the robustness and actuation challenges faced by soft robots.

In this paper, we present an adaptive walker system designed to address limitations in current intelligent walker technologies. While recent advancements have been made in this field, existing systems often struggle to seamlessly interpret user intent for speed control and lack adaptability across diverse scenarios and terrain. Our proposed solution incorporates high-resolution tactile sensors, deep learning algorithms, IMU sensors, and linear motors to dynamically adjust to the user's intentions and terrain changes. The system is capable of predicting the user's desired speed with an error margin of only 20.99%, relying solely on tactile input from hand and arm contact points. Additionally, it maintains the walker's horizontal stability with an error of less than 1 degree by adjusting leg lengths in response to variations in ground angle. This adaptive walker enhances user safety and comfort, particularly for individuals with reduced strength or cognitive abilities, and offers reliable assistance on uneven terrain such as uphill and downhill paths.

Astract-Recent advances in foundation models are significantly expanding the capabilities of AI models. As part of this progress, this paper introduces a robot design framework that uses a diffusion model approach for generating 3D mesh structures. Specifically, we focus on generating directly fabri-cable robot structures that require no post-processing guided by human-imposed design constraints. Our approach can find the optimal design of the robot by optimizing or composing embedding vectors of the model. The efficacy of the framework is validated through an application to design, fabricate, and evaluate a jumping robot. Our solution is an optimized jumping robot with a 41% increase in jump height compared to the state-of-the-art design. Additionally, when the robot is augmented with an optimized foot, it can land reliably with a success ratio of 88% in contrast to the 4% success ratio of the base robot.

Understanding and anticipating events and actions is critical for intraoperative assistance and decision-making during minimally invasive surgery. We propose a predictive neural network that is capable of understanding and predicting critical interaction aspects of surgical workflow based on endoscopic, intracorporeal video data, while flexibly leveraging surgical knowledge graphs. The approach incorporates a hypergraph-transformer (HGT) structure that encodes expert knowledge into the network design and predicts the hidden embedding of the graph. We verify our approach on established surgical datasets and applications, including the prediction of action-triplets, and the achievement of the Critical View of Safety (CVS), which is a critical safety measure. Moreover, we address specific, safety-related forecasts of surgical processes, such as predicting the clipping of the cystic duct or artery without prior achievement of the CVS. Our results demonstrate improvement in prediction of interactive event when incorporating with our approach compared to unstructured alternatives.

Safe motion planning is essential for autonomous vessel operations, especially in challenging spaces such as narrow inland waterways. However, conventional motion planning approaches are often computationally intensive or overly conservative. This paper proposes a safe motion planning strategy combining Model Predictive Control (MPC) and Control Barrier Functions (CBFs). We introduce a time-varying inflated ellipse obstacle representation, where the inflation radius is adjusted depending on the relative position and attitude between the vessel and the obstacle. The proposed adaptive inflation reduces the conservativeness of the controller compared to traditional fixed-ellipsoid obstacle formulations. The MPC solution provides an approximate motion plan, and high-order CBFs ensure the vessel’s safety using the varying inflation radius. Simulation and real-world experiments demonstrate that the proposed strategy enables the fully-actuated autonomous robot vessel to navigate through narrow spaces in real time and resolve potential deadlocks, all while ensuring safety.

A recent study has proposed the ``Master Key Filters Hypothesis" for convolutional neural network filters. This paper extends this hypothesis by radically constraining its scope to a single set of just 8 universal filters that depthwise separable convolutional networks inherently converge to. While conventional DS-CNNs employ thousands of distinct trained filters, our analysis reveals these filters are predominantly linear shifts (ax+b) of our discovered universal set. Through systematic unsupervised search, we extracted these fundamental patterns across different architectures and datasets. Remarkably, networks initialized with these 8 unique frozen filters achieve over 80\% ImageNet accuracy, and even outperform models with thousands of trainable parameters when applied to smaller datasets. The identified master key filters closely match Difference of Gaussians (DoGs), Gaussians, and their derivatives, structures that are not only fundamental to classical image processing but also strikingly similar to receptive fields in mammalian visual systems. Our findings provide compelling evidence that depthwise convolutional layers naturally gravitate toward this fundamental set of spatial operators regardless of task or architecture. This work offers new insights for understanding generalization and transfer learning through the universal language of these master key filters.

Recently, Sharma et al. (2024) suggested a method called LAyer- SElective-Rank reduction (LASER) which demonstrated that pruning high‑order components of carefully chosen LLM’s weight matrices can boost downstream accuracy—without any gradient‑based fine‑tuning. Yet LASER’s exhaustive, per‑matrix search (each requiring full‑dataset forward passes) makes it impractical for rapid deployment. We demonstrate that this overhead can be removed and find that: (i) Only a small, carefully chosen subset of matrices needs to be inspected—eliminating the layer‑by‑layer sweep, (ii) The gradient of each matrix’s singular values pinpoints which matrices merit reduction, (iii) Increasing the factorization search space by allowing matrices rows to cluster around multiple subspaces and then decomposing each cluster separately further reduces overfitting on the original training data and further lifts accuracy by up to 24.6 percentage points, and finally, (iv) we discover that evaluating on just 100 samples rather than the full training data—both for computing the indicative gradients and for measuring the final accuracy—suffices to further reduce the search time; we explain that as adaptation to downstream tasks is dominated by prompting style, not dataset size. As a results, we show that combining these findings yields a fast and robust adaptation algorithm for downstream tasks. Overall, with a single gradient step on 100 examples and a quick scan of the top candidate layers and factorization techniques, we can adapt LLMs to new datasets—entirely without fine‑tuning.

This paper investigates theresilientmulti-robot efficient search problem (R-MuRES), which aims at coordinating multiple robots to detect a ‘non-adversarial’ moving target with the minimal expected time. One unique characteristic of R-MuRES among others is the possibility of individual robot's malfunction and withdrawal from the team during task execution, which results in avariablenumber of searchers in the deployment phase and entails that the possibility of team member failures must be considered during the planning stage, particularly in the training phase. We propose a resilient value function factorization (R-FAC) paradigm, which constructs the central value function from individual ones in a resilient manner, taking into account individual robots' failures, and ensures that the constructed central value function has the minimal mean squared temporal difference error across various team compositions. R-FAC stipulates that the individual global maximum (IGM) principle is satisfied for whichever team configuration and thus any functioning robot contributes positively to the remaining team, as long as it executes the greedy policy with respect to the factorized individual value function. Subsequently, we introduce thevariationalvalue decomposition network (V2DN) as one of the instantiated R-FAC algorithms. V2DN employs the$\log$-sum-$\exp$mechanism to construct the central value function from individual ones, enabling it to take a varying number of robots' individual value functions as inputs. Then, we explain why, specifically for the multi-robot search task, the$\log$-sum-$\exp$mechanism is superior to the brute-force summation operation used in the canonical value decomposition network (VDN), and compare V2DN with state-of-the-art MuRES solutions as well as the vanilla VDN algorithm in two canonical MuRES testing environments and show that it achieves the best resiliency score when one or several individual robots quit the team during task execution. Furthermore, we validate V2DN with a real multi-robot system in a self-constructed indoor environment as the proof of concept.

The paper considers the route planning problem for a vehicle with limited capacity operating in a road network. The vehicle is assigned a set of transportation requests that are more complex than traveling between two locations, may involve dependencies between their sub-tasks, and include deadlines and priorities. The requests arrive gradually over the deployment time-horizon, and thus replanning is needed for new requests. We address cases when not all requests can be serviced by their deadlines despite car sharing. We introduce multiple quality measures for plans that account for requests' delays with respect to deadlines and priorities. We formalize the problem as planning in a weighted transition system under syntactically co-safe LTL formulas. We develop an online planning and replanning algorithm based on the automata-based approach to least-violating plan synthesis and on translation to a Mixed Integer Linear Program (MILP). Furthermore, we show that the MILP reduces to graph search for a subclass of quality measures that satisfy a monotonicity property. We show the approach in simulations, including a case study on the mid-Manhattan road network over the span of 24 hours.

Recent advances in depthwise-separable convolutional neural networks (DS-CNNs) have led to novel architectures, that surpass the performance of classical CNNs, by a considerable scalability and accuracy margin. This paper reveals another striking property of DS-CNN architectures: discernible and explainable patterns emerge in their trained depthwise convolutional kernels in all layers. Through an extensive analysis of millions of trained filters, with different sizes and from various models, we employed unsupervised clustering with autoencoders, to categorize these filters. Astonishingly, the patterns converged into a few main clusters, each resembling the difference of Gaussian (DoG) functions, and their first and second-order derivatives. Notably, we classify over 95\% and 90\% of the filters from state-of-the-art ConvNeXtV2 and ConvNeXt models, respectively. This finding is not merely a technological curiosity; it echoes the foundational models neuroscientists have long proposed for the vision systems of mammals. Our results thus deepen our understanding of the emergent properties of trained DS-CNNs and provide a bridge between artificial and biological visual processing systems. More broadly, they pave the way for more interpretable and biologically-inspired neural network designs in the future.

Modern deep learning requires large volumes of data, which could contain sensitive or private information that cannot be leaked. Recent work has shown for homogeneous neural networks a large portion of this training data could be reconstructed with only access to the trained network parameters. While the attack was shown to work empirically, there exists little formal understanding of its effective regime and which datapoints are susceptible to reconstruction. In this work, we first build a stronger version of the dataset reconstruction attack and show how it can provably recover the \emph{entire training set} in the infinite width regime. We then empirically study the characteristics of this attack on two-layer networks and reveal that its success heavily depends on deviations from the frozen infinite-width Neural Tangent Kernel limit. Next, we study the nature of easily-reconstructed images. We show that both theoretically and empirically, reconstructed images tend to ``outliers'' in the dataset, and that these reconstruction attacks can be used for \textit{dataset distillation}, that is, we can retrain on reconstructed images and obtain high predictive accuracy.

Developing autonomous agents that can interact with changing environments is an open challenge in machine learning. Robustness is particularly important in these settings as agents are often fit offline on expert demonstrations but deployed online where they must generalize to the closed feedback loop within the environment. In this work, we explore the application of recurrent neural networks to tasks of this nature and understand how a parameterization of their recurrent connectivity influences robustness in closed-loop settings. Specifically, we represent the recurrent connectivity as a function of rank and sparsity and show both theoretically and empirically that modulating these two variables has desirable effects on network dynamics. The proposed low-rank, sparse connectivity induces an interpretable prior on the network that proves to be most amenable for a class of models known as closed-form continuous-time neural networks (CfCs). We find that CfCs with fewer parameters can outperform their full-rank, fully-connected counterparts in the online setting under distribution shift. This yields memory-efficient and robust agents while opening a new perspective on how we can modulate network dynamics through connectivity.

Dataset Distillation seeks to summarize a large dataset by generating a reduced set of synthetic samples. While there has been much success at distilling small datasets such as CIFAR-10 on smaller neural architectures, Dataset Distillation methods fail to scale to larger high-resolution datasets and architectures. In this work, we introduce **D**ataset **D**istillation with **D**omain **S**hift (**D3S**), a scalable distillation algorithm, made by reframing the dataset distillation problem as a *domain shift* one. In doing so, we derive a universal bound on the distillation loss, and provide a method for efficiently approximately optimizing it. We achieve state-of-the-art results on Tiny-ImageNet, ImageNet-1k, and ImageNet-21K over a variety of recently proposed baselines, including high cross-architecture generalization. Additionally, our ablation studies provide lessons on the importance of validation-time hyperparameters on distillation performance, motivating the need for standardization.

Large Language Models have recently gained significant attention in scientific discovery for their extensive knowledge and advanced reasoning capabilities. However, they encounter challenges in effectively simulating observational feedback and grounding it with language to propel advancements in physical scientific discovery. Conversely, human scientists undertake scientific discovery by formulating hypotheses, conducting experiments, and revising theories through observational analysis. Inspired by this, we propose to enhance the knowledge-driven, abstract reasoning abilities of LLMs with the computational strength of simulations. We introduce Scientific Generative Agent (SGA), a bilevel optimization framework: LLMs act as knowledgeable and versatile thinkers, proposing scientific hypotheses and reason about discrete components, such as physics equations or molecule structures; meanwhile, simulations function as experimental platforms, providing observational feedback and optimizing via differentiability for continuous parts, such as physical parameters. We conduct extensive experiments to demonstrate our framework’s efficacy in constitutive law discovery and molecular design, unveiling novel solutions that differ from conventional human expectations yet remain coherent upon analysis.

Despite recent exponential advancements in computer vision and reinforcement learning, it remains challenging for robots to interact with liquids. These challenges are particularly pronounced due to the limitations imposed by opaque containers, transparent liquids, fine-grained splashes, and visual obstructions arising from the robot’s own manipulation activities. Yet, there exists a substantial opportunity for robotics to excel in liquid identification and manipulation, given its potential role in chemical handling in laboratories and various manufacturing sectors such as pharmaceuticals or beverages. In this work, we present a novel approach for liquid class identification and state estimation leveraging electrical impedance sensing. We design and mount a digitally embroidered electrode array to a commercial robot gripper. Coupled with a customized impedance sensing board, we collect data on liquid manipulation with a swept frequency sensing mode and a frequency-specific impedance measuring mode. Our developed learning-based model achieves an accuracy of 93.33% in classifying 9 different types of liquids (8 liquids + air), and 97.65% in estimating the liquid state. We investigate the effectiveness of our system with a series of ablation studies. These findings highlight our work as a promising solution for enhancing robotic manipulation in liquid-related tasks.

In autonomous robot navigation, the trajectories from path planners are considered to be safe regions, and deviations could endanger vessels. Model Predictive Control (MPC) stands as a popular choice for trajectory tracking problems as it naturally addresses operational constraints, such as dynamics and control constraints. Nevertheless, achieving robustness in changing environments like oceans and rivers, which are constantly subject to significant external disturbances, remains an ongoing challenge for MPC. It must consistently keep the system within a predefined safe region (such as a reference trajectory) even in the presence of model inaccuracies and perturbations. To address this challenge, we present a robust model predictive control strategy utilizing Control Barrier Functions (CBFs), which increases the disturbance-rejection abilities. We verify our method on an autonomous surface vessel in simulation and natural waters, both with external disturbances. Specifically, compared with the traditional MPC method, our proposed MPC-CBF strategy reduces tracking errors by 17.82% and 40.26% in simulations and field experiments, respectively. Although the control effort slightly increases by 7.78% and 4.20%, respectively, these results clearly demonstrate the enhanced resilience of MPC-CBF to disturbances.

Natural and robotic swarms often exhibit nonreciprocal interactions; agents do not exhibit equal and opposite forces on each other. By studying the effects of reciprocal and non-reciprocal interactions we are better able to design emergent behaviors in robot collectives composed of agents that exert attractive and repulsive forces on each other. Moreover, by controlling agent-specific coupling forces on-demand, we can enable a collective to exhibit desired behaviors previously not possible. We use a general form of the swarming oscillator, swarmalator, model to study reciprocal and non-reciprocal interactions among agents that affect each other’s motions over long and short distances, we use non-reciprocal coupling to elicit collective locomotion toward or away from target sites, and we use the control barrier function method to optimize the non-reciprocal interactions for a desired spatial formation. This work addresses the interests of the active matter, swarm robotics, and control barrier functions communities and demonstrates various collective behaviors with strong potential to be realized in macro- and micro- length scale robot swarms.

Bio-inspired neural networks have the potential to advance our understanding of neural computation and improve the state-of-the-art of AI systems. Bio-electrical synapses directly transmit neural signals, by enabling fast current flow between neurons. In contrast, bio-chemical synapses transmit neural signals indirectly, through neurotransmitters. Prior work showed that interpretable dynamics for complex robotic control, can be achieved by using chemical synapses, within a sparse, bio-inspired architecture, called Neural Circuit Policies (NCPs). However, a comparison of these two synaptic models, within the same architecture, remains an unexplored area. In this work we aim to determine the impact of using chemical synapses compared to electrical synapses, in both sparse and all-to-all connected networks. We conduct experiments with autonomous lane-keeping through a photorealistic autonomous driving simulator to evaluate their performance under diverse conditions and in the presence of noise. The experiments highlight the substantial influence of the architectural and synaptic-model choices, respectively. Our results show that employing chemical synapses yields noticeable improvements compared to electrical synapses, and that NCPs lead to better results in both synaptic models.

There is an ever-growing zoo of modern neural network models that can efficiently learn end-to-end control from visual observations. These advanced deep models, ranging from convolutional to Vision Transformers, from small to gigantic networks, have been extensively tested on offline image classification tasks. In this paper, we study these vision models with respect to the open-loop training to closed-loop generalization abilities, i.e., deployment realizes a causal feedback loop that is not present during training. This causality gap typically emerges in robotics applications such as autonomous driving, where a network is trained to imitate the control commands of a human. In this setting, two situations arise: 1) Closed-loop testing in-distribution, where the test environment shares properties with those of offline training data. 2) Closed-loop testing under distribution shifts and out-of-distribution. Contrary to recently reported results, we show that under proper training guidelines, all vision architectures perform indistinguishably well on in-distribution deployment, resolving the causality gap. In situation 2, We observe that scale is the strongest factor in improving closed-loop generalization regardless of the choice of the model architecture. Our results predict the trend that in the future we will see larger and larger models being used in offline-training-online-deployment imitation learning tasks in robotic applications.

Soft robotic manipulators with many degrees of freedom can carry out complex tasks safely around humans. However, manufacturing of soft robotic hands with several degrees of freedom requires a complex multi-step manual process, which significantly increases their cost. We present a design of a multi-material 15 DoF robotic hand with five fingers including an opposable thumb. Our design has 15 pneumatic actuators based on a series of hollow chambers that are driven by an external pressure system. The thumb utilizes rigid joints and the palm features internal rigid structure and soft skin. The design can be directly 3D printed using a multi-material additive manufacturing process without any assembly process and therefore our hand can be manufactured for less than 300 dollars. We test the hand in conjunction with a low-cost vision-based teleoperation system on different tasks.

While much work has been done recently in the realm of model-based control of soft robots and soft-rigid hybrids, most works examine robots that have an inherently serial structure. While these systems have been prevalent in the literature, there is an increasing trend toward designing soft-rigid hybrids with intrinsically coupled elasticity between various degrees of freedom. In this work, we seek to address the issues of modeling and controlling such structures, particularly when underactuated. We introduce several simple models for elastic coupling, typical of those seen in these systems. We then propose a controller that compensates for the elasticity, and we prove its stability with Lyapunov methods without relying on the elastic dominance assumption. This controller is applicable to the general class of underactuated soft robots. After evaluating the controller in simulated cases, we then develop a simple hardware platform to evaluate both the models and the controller. Finally, using the hardware, we demonstrate a novel use case for underactuated, elastically coupled systems in "sensorless" force control.

Underwater robots are indispensable for aquatic exploration, yet their size and complexity often limit broader application. This research presents a pioneering micro autonomous underwater vehicle (µAUV) design. This robot is distinguished by its utilization of mass-produced drone components, novel jet propulsion mechanisms, and multifunctional spherical shell. Its architecture is modular, appendage-free, and largely seal-free. Preliminary tests highlight its motion capabilities and set new benchmarks for centimeter-scale µAUV advancements.

As autonomous driving technology matures, end-to-end methodologies have emerged as a leading strategy, promising seamless integration from perception to control via deep learning. However, existing systems grapple with challenges such as unexpected open set environments and the complexity of black-box models. At the same time, the evolution of deep learning introduces larger, multimodal foundational models, offering multi-modal visual and textual understanding. In this paper, we harness these multimodal foundation models to enhance the robustness and adaptability of autonomous driving systems. We introduce a method to extract nuanced spatial features from transformers and the incorporation of latent space simulation for improved training and policy debugging. We use pixel/patch-aligned feature descriptors to expand foundational model capabilities to create an end-to-end multimodal driving model, demonstrating unparalleled results in diverse tests. Our solution combines language with visual perception and achieves significantly greater robustness on out-of-distribution situations.

Many computations in robotics can be dramatically accelerated if the robot configuration space is described as a collection of simple sets. For example, recently developed motion planners rely on a convex decomposition of the free space to design collision-free trajectories using fast convex optimization. In this work, we present an efficient method for approximately covering complex configuration spaces with a small number of polytopes. The approach constructs a visibility graph using sampling and generates a clique cover of this graph to find clusters of samples that have mutual line of sight. These clusters are then inflated into large, full-dimensional, polytopes. We evaluate our method on a variety of robotic systems and show that it consistently covers larger portions of free configuration space, with fewer polytopes, and in a fraction of the time compared to previous methods.

Sensing plays a pivotal role in robotic manipulation, dictating the accuracy and versatility with which objects are handled. Vision-based sensing methods often suffer from fabrication complexity and low durability, while approaches that rely on direct measurements on the gripper often have limited resolution and are difficult to scale. Here, we present a soft robotic gripper made out of two cubic lattices that are sensorized by embedding air channels within the structure. The lattices are 3D printed from a single build material, simplifying the fabrication process. The flexibility of this approach offers significant control over sensor and lattice design, while the pressure-based internal sensing provides measurements with minimal disruption to the grasping surface. With only 12 sensors, 6 per lattice, this gripper can estimate an object’s weight and location and offer new insights into grasp parameters like friction coefficients and grasp force.

Autonomous driving presents a complex challenge, which is usually addressed with artificial intelligence models that are end-to-end or modular in nature. Within the landscape of modular approaches, a bio-inspired neural circuit policy model has emerged as an innovative control module, offering a compact and inherently interpretable system to infer a steering wheel command from abstract visual features. Here, we take a leap forward by integrating a variational autoencoder with the neural circuit policy controller, forming a solution that directly generates steering commands from input camera images. By substituting the traditional convolutional neural network approach to feature extraction with a variational autoencoder, we enhance the system’s interpretability, enabling a more transparent and understandable decision-making process.In addition to the architectural shift toward a variational autoencoder, this study introduces the automatic latent perturbation tool, a novel contribution designed to probe and elucidate the latent features within the variational autoencoder. The automatic latent perturbation tool automates the interpretability process, offering granular insights into how specific latent variables influence the overall model’s behavior. Through a series of numerical experiments, we demonstrate the interpretative power of the variational autoencoder-neural circuit policy model and the utility of the automatic latent perturbation tool in making the inner workings of autonomous driving systems more transparent.

This paper addresses the challenge of actuating millimetre-sized motors, which are wirelessly driven by external magnetic fields. Traditional approaches, relying on rotating magnetic fields, often inadvertently cause the entire robot – especially if it is small and lightweight – to rotate, instead of a specified shaft in the motor. To overcome this issue, our study introduces a novel mechanism that leverages symmetrically configured magnetic motors to cancel out the torques, thus preventing unwanted rotation of the robot. This is achieved by utilizing a magnetic field along a single axis to induce rotational movement. The design features two millimetre-sized rotating magnets that interact to achieve a 90◦rotation, complemented by an external magnetic field that accomplishes the remaining 270◦, thus completing a full rotation. Furthermore, we demonstrate that applying a perpendicularly oriented magnetic field can inversely affect the motor’s rotation direction. A proof-of-concept experiment employing this mechanism successfully actuated a gripper in a water tank while it is free-floating, showcasing its potential for enhancing robotic applications at the sub-centimeter scale, where the small net torque of a miniature motor is essential.

In this work, we consider the problem of learning end to end perception to control for ground vehicles solely from aerial imagery. Photogrammetric simulators allow the synthesis of novel views through the transformation of pre-generated assets into novel views. However, they have a large setup cost, require careful collection of data and often human effort to create usable simulators. We use a Neural Radiance Field (NeRF) as an intermediate representation to synthesize novel views from the point of view of a ground vehicle. These novel viewpoints can then be used for several downstream autonomous navigation applications. In this work, we demonstrate the utility of novel view synthesis though the application of training a policy for end to end learning from images and depth data. In a traditional real to sim to real framework, the collected data would be transformed into a visual simulator which could then be used to generate novel views. In contrast, using a NeRF allows a compact representation and the ability to optimize over the parameters of the visual simulator as more data is gathered in the environment. We demonstrate the efficacy of our method in a custom built mini-city environment through the deployment of imitation policies on robotic cars. We additionally consider the task of place localization and demonstrate that our method is able to relocalize the car in the real world.

Generating varied scenarios through simulation is crucial for training and evaluating safety-critical systems, such as autonomous vehicles. Yet, the task of modeling the trajectories of other vehicles to simulate diverse and meaningful close interactions remains prohibitively costly. Adopting language descriptions to generate driving behaviors emerges as a promising strategy, offering a scalable and intuitive method for human operators to simulate a wide range of driving interactions. However, the scarcity of large-scale annotated language-trajectory data makes this approach challenging. To address this gap, we propose Text-to-Drive (T2D) to synthesize diverse driving behaviors via Large Language Models (LLMs). We introduce a knowledge-driven approach that operates in two stages. In the first stage, we employ the embedded knowledge of LLMs to generate diverse language descriptions of driving behaviors for a scene. Then, we leverage LLM’s reasoning capabilities to synthesize these behaviors in simulation. At its core, T2D employs an LLM to construct a state chart that maps low-level states to high-level abstractions. This strategy aids in downstream tasks such as summarizing low-level observations, assessing policy alignment with behavior description, and shaping the auxiliary reward, all without needing human supervision. With our knowledge-driven approach, we demonstrate that T2D generates more diverse trajectories compared to other baselines and offers a natural language interface that allows for interactive incorporation of human preference. Please check our website for more examples: here

Efficiently learning strategic multi-agent behavior remains a challenge for robotic systems deployed in real-world scenarios, especially when considering underactuated or dynamically unstable systems. Such systems demand an integrated approach that informs long-term strategic planning with constraints imposed by reactive control, and vice versa, to effectively accomplish task objectives in competitive scenarios. In this paper, we introduce a hierarchical control model to address this: a high-level controller synthesizes strategic guidance from aggregated team experiences, while a low-level controller formulates corresponding task-specific continuous controls. We apply this concept to coordination of competitive multi-team behavior in dynamic flight scenarios with F-16 aircraft. This work introduces a hierarchical reinforcement learning approach for multi-agent coordination, leveraging decoupled distributional value representations at the high-level together with goal-conditioned policy learning at the low-level, providing a control structure that integrates long-horizon strategic planning with short-horizon dynamic control. We further provide a parallel simulator for efficient learning with multi-agent F-16 dynamics.

The natural flexibility of soft robotic grippers allows for versatile and compliant grasping. However, this same flexibility can restrict the gripper’s strength. Striking a balance between compliance and strength is essential for effective soft grippers. In this work, we present Flexible Robust Observant Gripper (FROG), a soft gripper that is both compliant and strong. We describe the mechanical design of the gripper, characterize the soft flexures used in the design, and analyze the grasp forces generated by the gripper. Utilizing the structure of the gripper, we develop feedforward grasp controllers and a classifier to distinguish between grasp types. Grasping experiments show that FROG can effectively grasp a variety of objects, including very soft or delicate items. Holding force tests show that our gripper can conform to the grasped object and exert large grasp forces.

In-context learning (ICL) allows transformer-based language models that are pre-trained on general text to quickly learn a specific task with a few "task demonstrations" without updating their parameters, significantly boosting their flexibility and generality. ICL possesses many distinct characteristics from conventional machine learning, thereby requiring new approaches to interpret this learning paradigm. Taking the viewpoint of recent works showing that transformers learn in context by formulating an internal optimizer, we propose an influence function-based attribution technique, DETAIL, that addresses the specific characteristics of ICL. We empirically verify the effectiveness of our approach for demonstration attribution while being computationally efficient. Leveraging the results, we then show how DETAIL can help improve model performance in real-world scenarios through demonstration reordering and curation. Finally, we experimentally prove the wide applicability of DETAIL by showing our attribution scores obtained on white-box models are transferable to black-box models in improving model performance.

In this study, we present evidence suggesting that depthwise convolutional kernels are effectively replicating the structural intricacies of the biological receptive fields observed in the mammalian retina. We provide analytics of trained kernels from various state-of-the-art models substantiating this evidence. Inspired by this intriguing discovery, we propose an initialization scheme that draws inspiration from the biological receptive fields. Experimental analysis of the ImageNet dataset with multiple CNN architectures featuring depthwise convolutions reveals a marked enhancement in the accuracy of the learned model when initialized with biologically derived weights. This underlies the potential for biologically inspired computational models to further our understanding of vision processing systems and to improve the efficacy of convolutional networks.

Analysis of relations between objects and comprehension of abstract concepts in the surgical video is important in AI-augmented surgery. However, building models that integrate our knowledge and understanding of surgery remains a challenging endeavor. In this paper, we propose a novel way to integrate conceptual knowledge into temporal analysis tasks using temporal concept graph networks. In the proposed networks, a knowledge graph is incorporated into the temporal video analysis of surgical notions, learning the meaning of concepts and relations as they apply to the data. We demonstrate results in surgical video data for tasks such as verification of the critical view of safety, estimation of the Parkland grading scale as well as recognizing instrument-action-tissue triplets. The results show that our method improves the recognition and detection of complex benchmarks as well as enables other analytic applications of interest.

We study the problem of training and certifying adversarially robust quantized neural networks (QNNs). Quantization is a technique for making neural networks more efficient by running them using low-bit integer arithmetic and is therefore commonly adopted in industry. Recent work has shown that floating-point neural networks that have been verified to be robust can become vulnerable to adversarial attacks after quantization, and certification of the quantized representation is necessary to guarantee robustness. In this work, we present quantization-aware interval bound propagation (QA-IBP), a novel method for training robust QNNs. Inspired by advances in robust learning of non-quantized networks, our training algorithm computes the gradient of an abstract representation of the actual network. Unlike existing approaches, our method can handle the discrete semantics of QNNs. Based on QA-IBP, we also develop a complete verification procedure for verifying the adversarial robustness of QNNs, which is guaranteed to terminate and produce a correct answer. Compared to existing approaches, the key advantage of our verification procedure is that it runs entirely on GPU or other accelerator devices. We demonstrate experimentally that our approach significantly outperforms existing methods and establish the new state-of-the-art for training and certifying the robustness of QNNs.

We propose a new method to ensure neural ordinary differential equations (ODEs) satisfy output specifications by using invariance set propagation. Our approach uses a class of control barrier functions to transform output specifications into constraints on the parameters and inputs of the learning system. This setup allows us to achieve output specification guarantees simply by changing the constrained parameters/inputs both during training and inference. Moreover, we demonstrate that our invariance set propagation through data-controlled neural ODEs not only maintains generalization performance but also creates an additional degree of robustness by enabling causal manipulation of the system’s parameters/inputs. We test our method on a series of representation learning tasks, including modeling physical dynamics and convexity portraits, as well as safe collision avoidance for autonomous vehicles.

We propose a new dataset distillation algorithm using reparameterization and convexification of implicit gradients (RCIG), that substantially improves the state-of-the-art. To this end, we first formulate dataset distillation as a bi-level optimization problem. Then, we show how implicit gradients can be effectively used to compute meta-gradient updates. We further equip the algorithm with a convexified approximation that corresponds to learning on top of a frozen finite-width neural tangent kernel. Finally, we improve bias in implicit gradients by parameterizing the neural network to enable analytical computation of final-layer parameters given the body parameters. RCIG establishes the new state-of-the-art on a diverse series of dataset distillation tasks. Notably, with one image per class, on resized ImageNet, RCIG sees on average a 108% improvement over the previous state-of-the-art distillation algorithm. Similarly, we observed a 66% gain over SOTA on Tiny-ImageNet and 37% on CIFAR-100.

A coreset is a small weighted subset of an input set that approximates its loss function, for a given set of queries. Coresets became prevalent in machine learning as they have shown to be advantageous for many applications. Unfortunately, coresets are constructed in a problem-dependent manner, where for each problem, a new coreset construction algorithm is suggested, taking years to prove its correctness. Even the generic frameworks require additional (problem-dependent) computations or proofs to be done by the user. Besides, many problems do not have (provable) small coresets, limiting their applicability. To this end, we suggest an automatic practical framework for constructing coresets, which requires (only) the input data and the desired cost function from the user, without the need for any other task-related computation to be done by the user. To do so, we reduce the problem of approximating a loss function to an instance of vector summation approximation, where the vectors we aim to sum are loss vectors of a specific subset of the queries, such that we aim to approximate the image of the function on this subset. We show that while this set is limited, the coreset is quite general. An extensive experimental study on various machine learning applications is also conducted. Finally, we provide a “plug and play" style implementation, proposing a user-friendly system that can be easily used to apply coresets for many problems. We believe that these contributions enable future research and easier use and applications of coresets.

Radial basis function neural networks (RBFNN) are well-known for their capability to approximate any continuous function on a closed bounded set with arbitrary precision given enough hidden neurons. In this paper, we introduce the first algorithm to construct coresets for RBFNNs, i.e., small weighted subsets that approximate the loss of the input data on any radial basis function network and thus approximate any function defined by an RBFNN on the larger input data. In particular, we construct coresets for radial basis and Laplacian loss functions. We then use our coresets to obtain a provable data subset selection algorithm for training deep neural networks. Since our coresets approximate every function, they also approximate the gradient of each weight in a neural network, which is a particular function on the input. We then perform empirical evaluations on function approximation and dataset subset selection on popular network architectures and data sets, demonstrating the efficacy and accuracy of our coreset construction.

We present Learned Risk Metric Maps (LRMM) for real-time estimation of coherent risk metrics of high-dimensional dynamical systems operating in unstructured, partially observed environments. LRMM models are simple to design and train-requiring only procedural generation of obstacle sets, state and control sampling, and supervised training of a function approximator-which makes them broadly applicable to arbitrary system dynamics and obstacle sets. In a parallel autonomy setting, we demonstrate the model's ability to rapidly infer collision probabilities of a fast-moving car-like robot driving recklessly in an obstructed environment; allowing the LRMM agent to intervene, take control of the vehicle, and avoid collisions. In this time-critical scenario, we show that LRMMs can evaluate risk metrics 20-100x times faster than alternative safety algorithms based on control barrier functions (CBFs) and Hamilton-Jacobi reachability (HJ-reach), leading to 5–15 % fewer obstacle collisions by the LRMM agent than CBFs and HJ-reach. This performance improvement comes in spite of the fact that the LRMM model only has access to local/partial observation of obstacles, whereas the CBF and HJ-reach agents are granted privileged/global information. We also show that our model can be equally well trained on a 12-dimensional quadrotor system operating in an obstructed indoor environment. The LRMM codebase is provided at https://github.com/mit-drl/pyrmm.

Intelligent intersection managers can improve safety by detecting dangerous drivers or failure modes in autonomous vehicles, warning oncoming vehicles as they approach an intersection. In this work, we present FailureNet, a recurrent neural network trained end-to-end on trajectories of both nominal and reckless drivers in a scaled miniature city. FailureNet observes the poses of vehicles as they approach an intersection and detects whether a failure is present in the autonomy stack, warning cross-traffic of potentially dangerous drivers. FailureNet can accurately identify control failures, upstream perception errors, and speeding drivers, distinguishing them from nominal driving. The network is trained and deployed with autonomous vehicles in the MiniCity. Compared to speed or frequency-based predictors, FailureNet's recurrent neural network structure provides improved predictive power, yielding upwards of 84% accuracy when deployed on hardware.

Safety has been a key goal for autonomous driving since its inception, and we believe recognizing and responding to risk is a key component of safety. In this work, we aim to answer the question, “How can explainable risk representations be generated and used to produce risk-averse trajectories?” To answer this question, previous work uses risk metrics to formulate an optimization problem. In contrast, our work is based on research showing the usefulness of grids as a representation to generate image-based risk maps through a trained neural network. We propose a method of determining risk from a bird's eye view (BEV) of an autonomous vehicle's surroundings. Our method consists of (1) a risk map generator, which is trained to recognize risk associated with nearby agents and the map, (2) differentiable value iteration using the risk map to learn a policy, and (3) a trajectory sampler, which samples from this policy to generate a trajectory. We evaluate our planner in a close-loop manner and find improvements in its overall ability to mimic human driving while maintaining comparable safety statistics. Self-ablation also reveals the potential for fine-tuning the behavior of the planner given a designer's needs.

Multi-sensor fusion is essential for an accurate and reliable autonomous driving system. Recent approaches are based on point-level fusion: augmenting the LiDAR point cloud with camera features. However, the camera-to-LiDAR projection throws away the semantic density of camera features, hindering the effectiveness of such methods, especially for semantic-oriented tasks (such as 3D scene segmentation). In this paper, we propose BEVFusion, an efficient and generic multi-task multi-sensor fusion framework. It unifies multi-modal features in the shared bird's-eye view (BEV) representation space, which nicely preserves both geometric and semantic information. To achieve this, we diagnose and lift the key efficiency bottlenecks in the view transformation with optimized BEV pooling, reducing latency by more than$\mathbf{40}\times$. BEVFusion is fundamentally task-agnostic and seamlessly supports different 3D perception tasks with almost no architectural changes. It establishes the new state of the art on the nuScenes benchmark, achieving 1.3% higher mAP and NDS on 3D object detection and 13.6% higher mIoU on BEV map segmentation, with 1.9× lower computation cost. Code to reproduce our results is available at https://github.com/mit-han-lab/bevfusion.

We suggest the first system that runs real-time semantic segmentation via deep learning on the weak microcomputer Raspberry Pi Zero v2 (whose price was $15) attached to a toy drone. In particular, since the Raspberry Pi weighs less than 16 grams, and its size is half of a credit card, we could easily attach it to the common commercial DJI Tello toy-drone ($\times 92.5\times 41$mm). The result is an autonomous drone (no laptop nor human in the loop) that can detect and classify objects in real-time from a video stream of an onboard monocular RGB camera (no GPS or LIDAR sensors). The companion videos demonstrate how this Tello drone scans the lab for people (e.g. for the use of firefighters or security forces) and for an empty parking slot outside the lab. Existing deep learning solutions are either much too slow for real-time computation on such IoT devices, or provide results of impractical quality. Our main challenge was to design a system that takes the best of all worlds among numerous combinations of networks, deep learning platforms/frameworks, compression techniques, and compression ratios. To this end, we provide an efficient searching algorithm that aims to find the optimal combination which results in the best tradeoff between the network running time and its accuracy/performance.

Accurate control of autonomous marine robots still poses challenges due to the complex dynamics of the environment. In this paper, we propose a Deep Reinforcement Learning (DRL) approach to train a controller for autonomous surface vessel (ASV) trajectory tracking and compare its performance with an advanced nonlinear model predictive controller (NMPC) in real environments. Taking into account environmental disturbances (e.g., wind, waves, and currents), noisy measurements, and non-ideal actuators presented in the physical ASV, several effective reward functions for DRL tracking control policies are carefully designed. The control policies were trained in a simulation environment with diverse tracking trajectories and disturbances. The performance of the DRL controller has been verified and compared with the NMPC in both simulations with model-based environmental disturbances and in natural waters. Simulations show that the DRL controller has 53.33% lower tracking error than that of NMPC. Experimental results further show that, compared to NMPC, the DRL controller has 35.51% lower tracking error, indicating that DRL controllers offer better disturbance rejection in river environments than NMPC.

Adverse weather has raised a big challenge for autonomous vehicles. Unreliable measurements due to sensor degradation could seriously affect the performance of autonomous driving tasks, such as perception and localization. In this work, we study sensor degradation in rainy weather and present a novel method that evaluates the uncertainty for each laser measurement from a 3D LiDAR. With uncertainty estimation, downstream tasks that rely on LiDAR input (e.g., perception or localization) can increase their reliability by adjusting their reliance on laser measurements with varying fidelity. Alternatively, uncertainty estimation can be used for sensor performance evaluation. Our proposed method, SmartRainNet, uses an attention-based Mixture Density Network to model the dependence between neighboring laser measurements and then calculate the probability density for each laser measurement as an uncertainty score. We evaluate SmartRainNet on synthetic and naturalistic sensor degradation datasets and provide qualitative and quantitative results to demonstrate the effectiveness of our method in evaluating uncertainty. Finally, we demonstrate three practical applications of uncertainty estimation to address autonomous driving challenges in rainy weather.

We explore the idea of robotic mechanisms that can shift between soft and rigid states, with the long-term goal of creating robots that marry the flexibility and robustness of soft robots with the strength and precision of rigid robots. We present a simple yet effective method to achieve large and rapid stiffness variations by compressing and relaxing a flexure using cables. Next, we provide a differentiable modeling framework that can be used for motion planning, which simultaneously reasons about the modulated stiffness joints, tendons, rigid joints, and basic hydrodynamics. We apply this stiffness tuning and simulation recipe to create SoRiTu, an untethered soft-rigid robotic sea turtle capable of various swimming maneuvers.

Game-theoretic motion planners are a powerful tool for the control of interactive multi-agent robot systems. Indeed, contrary to predict-then-plan paradigms, game-theoretic planners do not ignore the interactive nature of the problem, and simultaneously predict the behaviour of other agents while considering change in one's policy. This, however, comes at the expense of computational complexity, especially as the number of agents considered grows. In fact, planning with more than a handful of agents can quickly become intractable, disqualifying game-theoretic planners as possible candidates for large scale planning. In this paper, we propose a planning algorithm enabling the use of game-theoretic planners in robot systems with a large number of agents. Our planner is based on the reality of locality of information and thus deploys local games with a selected subset of agents in a receding horizon fashion to plan collision avoiding trajectories. We propose five different principled schemes for selecting game participants and compare their collision avoidance performance. We observe that the use of Control Barrier Functions for priority ranking is a potent solution to the player selection problem for motion planning.

Inspired by human proteins that are synthesized from only 20 types of amino acids, the development of self-assembly methods that allow robots to be built simply by randomly stirring the parts has been explored for many years. The key challenges include how to synthesize parts in pieces into a three-dimensional functional structure in a practical time, and subsequently, achieve a controlled robotic motion, all with minimal human intervention. This study proposes a method of self-assembling a 3D robot by first self-assembling random parts into a 2D structure and then self-folding it into a 3D shape. Once self-folded, the robot, whose compositional parts contain magnets, becomes capable of performing basic tasks such as block-pushing upon an application of an external magnetic field. Self-assembly from parts into a two-dimensional structure was performed by repeatedly colliding the parts with each other, and combining them with complementary-shaped parts, like matching jigsaw puzzle pieces. Self-folding was performed by shrinking a heat-responsive film attached across the hinge of each assembly part in hot water, causing the entire 2D structure to self-fold. The experiment demonstrated a series of 13 parts self-assembling into the shape of a 3D beetle, then walking and pushing an object in 13 minutes. The self-assembly process is programmed (mechanically) to generate the same geometry even if the number of parts is greater than the necessary number for the structure, thus is capable of generating multiple structures simultaneously.

The cooperation of a human pilot with an autonomous agent during flight control realizes parallel autonomy. We propose an air-guardian system that facilitates cooperation between a pilot with eye tracking and a parallel end-to-end neural control system. Our vision-based air-guardian system combines a causal continuous-depth neural network model with a cooperation layer to enable parallel autonomy between a pilot and a control system based on perceived differences in their attention profiles. The attention profiles for neural networks are obtained by computing the networks' saliency maps (feature importance) through the VisualBackProp algorithm, while the attention profiles for humans are either obtained by eye tracking of human pilots or saliency maps of networks trained to imitate human pilots. When the attention profile of the pilot and guardian agents align, the pilot makes control decisions. Otherwise, the air-guardian makes interventions and takes over the control of the aircraft. We show that our attention-based air-guardian system can balance the trade-off between its level of involvement in the flight and the pilot's expertise and attention. The guardian system is particularly effective in situations where the pilot was distracted due to information overload. We demonstrate the effectiveness of our method for navigating flight scenarios in simulation with a fixed-wing aircraft and on hardware with a quadrotor platform.

Autonomous driving in rain remains challenging. Rain causes sensor performance degradation that can affect sensor measurement quality. During the rain, lasers may suffer from energy loss due to raindrop absorption. As a result, some laser measurements reflected from obstacles may not be recognized by the LiDAR sensor, thus raising potential risks for autonomous vehicles. This work investigates a novel task that aims to detect those missing measurements. Our solution uses a two-stage learning method to generate an anomaly score for each missing measurement, representing the likelihood of being caused by rain. We evaluate our method with real-world data and demonstrate its effectiveness in identifying anomalous missing measurements through qualitative and quantitative experiments.

Autonomous driving in the rain remains a challenge. One main problem is performance degradation caused by rain. This work introduces a new dataset to study this problem. Our dataset is collected from a full-scale vehicle equipped with a 3D LiDAR sensor and multiple forward-facing cameras under various rainy conditions. In addition, rainfall intensity is recorded in real-time from a rain sensor. The combination of sensor and rainfall intensity measurement is designed for studying algorithm performance under different levels of rainfall. In this work, in addition to presenting dataset creation details, we also introduce three degradation evaluation tasks with baseline results, including rainfall intensity estimation, LiDAR degradation estimation, and 2D object detection evaluation. This dataset, development kit, and baseline codes will be made available at https://smart-rain-dataset.github.io/

Sensor degradation is one of the major challenges for autonomous driving. During the rain, the interference from raindrops can negatively influence LiDAR measurements. For example, valid measurements could be reduced during the rain, and some measurements may become noisy. Unreliable measurements can lead to potential safety issues if autonomous driving systems are unaware of these changes. In this work, we will release a naturalistic driving dataset to advance the research in studying LiDAR degradation. Our dataset consists of 3D LiDAR scans collected by a data collection vehicle under various rainy conditions. Besides these raw scans, we also release LiDAR scan pairs (each pair consists of one scan from rainy weather and one scan from clear weather at the same location). These LiDAR pairs are developed to help researchers identify LiDAR degradation. Finally, we will release a toolbox integrated with mapping, localization, and scan synthesis functions used to create this dataset. This toolbox can facilitate dataset creation for studying degradation in other harsh weather conditions. More information can be found at https://smart-rain-dataset.github.io/.

Machine learning-based approaches for soft robot proprioception have recently gained popularity, in part due to the difficulties in modeling the relationship between sensor signals and robot shape. However, to date, there exists no systematic analysis of the required design choices to set up a machine learning pipeline for soft robot proprioception. Here, we present the first study examining how design choices on different levels of the machine learning pipeline affect the performance of a neural network for predicting the state of a soft robot. We address the most frequent questions researchers face, such as how to choose the appropriate sensor and actuator signals, process input and output data, deal with time series, and pick the best neural network architecture. By testing our hypotheses on data collected from two vastly different systems–an electrically actuated robotic platform and a pneumatically actuated soft trunk–we seek conclusions that may generalize beyond one specific type of soft robot and hope to provide insights for researchers to use machine learning for soft robot proprioception.

Semi-cooperative behaviors are intrinsic properties of human drivers and should be considered for autonomous driving. In addition, new autonomous planners can consider the social value orientation (SVO) of human drivers to generate socially-compliant trajectories. Yet the overall impact on traffic flow for this new class of planners remain to be understood. In this work, we present study of implicit semi-cooperative driving where agents deploy a game-theoretic version of iterative best response assuming knowledge of the SVOs of other agents. We simulate nominal traffic flow and investigate whether the proportion of prosocial agents on the road impact individual or system-wide driving performance. Experiments show that the proportion of prosocial agents has a minor impact on overall traffic flow and that benefits of semi-cooperation disproportionally affect egoistic and high-speed drivers.

Multi-agent reinforcement learning (MARL) research is faced with a trade-off: it either uses complex environments requiring large compute resources, which makes it inaccessible to researchers with limited resources, or relies on simpler dynamics for faster execution, which makes the transferability of the results to more realistic tasks challenging. Motivated by these challenges, we present Gigastep, a fully vectorizable, MARL environment implemented in JAX, capable of executing up to one billion environment steps per second on consumer-grade hardware. Its design allows for comprehensive MARL experimentation, including a complex, high-dimensional space defined by 3D dynamics, stochasticity, and partial observations. Gigastep supports both collaborative and adversarial tasks, continuous and discrete action spaces, and provides RGB image and feature vector observations, allowing the evaluation of a wide range of MARL algorithms. We validate Gigastep's usability through an extensive set of experiments, underscoring its role in widening participation and promoting inclusivity in the MARL research community.

Dataset Distillation is the task of synthesizing small datasets from large ones while still retaining comparable predictive accuracy to the original uncompressed dataset. Despite significant empirical progress in recent years, there is little understanding of the theoretical limitations/guarantees of dataset distillation, specifically, what excess risk is achieved by distillation compared to the original dataset, and how large are distilled datasets? In this work, we take a theoretical view on kernel ridge regression (KRR) based methods of dataset distillation such as Kernel Inducing Points. By transforming ridge regression in random Fourier features (RFF) space, we provide the first proof of the existence of small (size) distilled datasets and their corresponding excess risk for shift-invariant kernels. We prove that a small set of instances exists in the original input space such that its solution in the RFF space coincides with the solution of the original data. We further show that a KRR solution can be generated using this distilled set of instances which gives an approximation towards the KRR solution optimized on the full input data. The size of this set is linear in the dimension of the RFF space of the input set or alternatively near linear in the number of effective degrees of freedom, which is a function of the kernel, number of data points, and the regularization parameter $\lambda$. The error bound of this distilled set is also a function of $\lambda$. We verify our bounds analytically and empirically.

Nature evolves creatures with a high complexity of morphological and behavioral intelligence, meanwhile computational methods lag in approaching that diversity and efficacy. Co-optimization of artificial creatures' morphology and control in silico shows promise for applications in physical soft robotics and virtual character creation; such approaches, however, require developing new learning algorithms that can reason about function atop pure structure. In this paper, we present DiffuseBot, a physics-augmented diffusion model that generates soft robot morphologies capable of excelling in a wide spectrum of tasks. \name bridges the gap between virtually generated content and physical utility by (i) augmenting the diffusion process with a physical dynamical simulation which provides a certificate of performance, and (ii) introducing a co-design procedure that jointly optimizes physical design and control by leveraging information about physical sensitivities from differentiable simulation. We showcase a range of simulated and fabricated robots along with their capabilities. Check our website: https://diffusebot.github.io/

Transformers have proven superior performance for a wide variety of tasks since they were introduced. In recent years, they have drawn attention from the vision community in tasks such as image classification and object detection. Despite this wave, an accurate and efficient multiple-object tracking (MOT) method based on transformers is yet to be designed. We argue that the direct application of a transformer architecture with quadratic complexity and insufficient noise-initialized sparse queries - is not optimal for MOT. We propose TransCenter, a transformer-based MOT architecture with dense representations for accurately tracking all the objects while keeping a reasonable runtime. Methodologically, we propose the use of image-related dense detection queries and efficient sparse tracking queries produced by our carefully designed query learning networks (QLN). On one hand, the dense image-related detection queries allow us to infer targets' locations globally and robustly through dense heatmap outputs. On the other hand, the set of sparse tracking queries efficiently interacts with image features in our TransCenter Decoder to associate object positions through time. As a result, TransCenterexhibits remarkable performance improvements and outperforms by a large margin the current state-of-the-art methods in two standard MOT benchmarks with two tracking settings (public/private). TransCenter is also proven efficient and accurate by an extensive ablation study and, comparisons to more naive alternatives and concurrent works. The code is made publicly available at https://github.com/yihongxu/transcenter.

Consider a stream of$d$-dimensional rows (points in$\mathbb {R}^{d}$) arriving sequentially. An$\epsilon$-coreset is a positively weighted subset that approximates their sum of squared distances to any linear subspace of$\mathbb {R}^{d}$, up to a$1 \pm \epsilon$factor. Unlike other data summarizations, such a coreset: (1) can be used to minimize faster any optimization function that uses this sum, such as regularized or constrained regression, (2) preserves input sparsity; (3) easily interpretable; (4) avoids numerical errors; (5) applies to problems with constraints on the input, such as subspaces that are spanned by few input points. Our main result is the first algorithm that returns such an$\epsilon$-coreset using finite and constant memory during the streaming, i.e., independent of$n$, the number of rows seen so far. The coreset consists of$O(d \log ^{2}\;d / \epsilon ^{2})$weighted rows, which is nearly optimal according to existing lower bounds of$\Omega (d / \epsilon ^{2})$. We support our findings with experiments on the Wikipedia dataset benchmarked against state-of-the-art algorithms.

This article investigates the multirobot efficient search (MuRES) for a nonadversarial moving target problem from the multiagent reinforcement learning (MARL) perspective. MARL is deemed as a promising research field for cooperative multiagent applications. However, one of the main bottlenecks of applying MARL to the MuRES problem is the nonstationarity introduced by multiple learning agents. With learning agents simultaneously updating their policies, the environment cannot be modeled as astationaryMarkov decision process, which results in the inapplicability of fundamental reinforcement learning techniques such as deep$Q$-network and policy gradient (PG). In view of that, we adopt the centralized training and decentralized execution scheme and thereby propose a cross-entropy regularized policy gradient (CE-PG) method to train the learning agents/robots. We let the robotscommitto a predetermined policy during execution, collect the trajectories, and then perform centralized training for the corresponding policy improvement. In this way, the nonstationarity problem is overcome, in that the robots do not update their policies during execution. During the centralized training stage, we improve the canonical PG method to consider the interactions among robots by adding a cross-entropy regularization term, which essentially functions to “disperse” the robots in the environment. Extensive simulation results and comparisons with state of the art show CE-PG's superior performance, and we also validate the algorithm with a real multirobot system in an indoor moving target search scenario.

Many safety-critical applications of neural networks, such as robotic control, require safety guarantees. This article introduces a method for ensuring the safety of learned models for control using differentiable control barrier functions (dCBFs). dCBFs are end-to-end trainable and guarantee safety. They improve over classical control barrier functions (CBFs), which are usually overly conservative. Our dCBF solution relaxes the CBF definitions by: 1) using environmental dependencies; 2) embedding them into differentiable quadratic programs. These novel safety layers are called a BarrierNet. They can be used in conjunction with any neural network-based controller. They are trained by gradient descent. With BarrierNet, the safety constraints of a neural controller become adaptable to changing environments. We evaluate BarrierNet on the following several problems: 1) robot traffic merging; 2) robot navigation in 2-D and 3-D spaces; 3) end-to-end vision-based autonomous driving in a sim-to-real environment and in physical experiments; 4) demonstrate their effectiveness compared to state-of-the-art approaches.

We introduce a new statistical verification algorithm that formally quantifies the behavioral robustness of any time-continuous process formulated as a continuous-depth model. Our algorithm solves a set of global optimization (Go) problems over a given time horizon to construct a tight enclosure (Tube) of the set of all process executions starting from a ball of initial states. We call our algorithm GoTube. Through its construction, GoTube ensures that the bounding tube is conservative up to a desired probability and up to a desired tightness. GoTube is implemented in JAX and optimized to scale to complex continuous-depth neural network models. Compared to advanced reachability analysis tools for time-continuous neural networks, GoTube does not accumulate overapproximation errors between time steps and avoids the infamous wrapping effect inherent in symbolic techniques. We show that GoTube substantially outperforms state-of-the-art verification tools in terms of the size of the initial ball, speed, time-horizon, task completion, and scalability on a large set of experiments. GoTube is stable and sets the state-of-the-art in terms of its ability to scale to time horizons well beyond what has been previously possible.

Soft actuators with integrated sensing have shown utility in a variety of applications such as assistive wearables, robotics, and interactive input devices. Despite their promise, these actuators can be difficult to both design and fabricate. As a solution, we present a workflow for computationally designing and digitally fabricating soft pneumatic actuators via a machine knitting process. Machine knitting is attractive as a fabrication process because it is fast, digital (programmable), and provides access to a rich material library of functional yarns for specified mechanical behavior and integrated sensing. Our method uses elastic stitches to construct non-homogeneous knitting structures, which program the bending of actuators when inflated. Our method also integrates pressure and swept frequency capacitive sensing structures using conductive yarns. The entire knitted structure is fabricated automatically in a single machine run. We further provide a computational design interface for the user to interactively preview actuators’ quasi-static shape when authoring elastic stitches. Our sensing-integrated actuators are cost-effective, easy to design, robust to large actuation, and require minimal manual post-processing. We demonstrate five use-cases of our actuators in relevant application settings.

Simulation has the potential to transform the development of robust algorithms for mobile agents deployed in safety-critical scenarios. However, the poor photorealism and lack of diverse sensor modalities of existing simulation engines remain key hurdles towards realizing this potential. Here, we present VISTA††Full code release for the VISTA data-driven simulation engine is available here: vista.csail.mit.edu., an open source, data-driven simulator that integrates multiple types of sensors for autonomous vehicles. Using high fidelity, real-world datasets, VISTA represents and simulates RGB cameras, 3D LiDAR, and event-based cameras, enabling the rapid generation of novel viewpoints in simulation and thereby enriching the data available for policy learning with corner cases that are difficult to capture in the physical world. Using VISTA, we demonstrate the ability to train and test perception-to-control policies across each of the sensor types and showcase the power of this approach via deployment on a full scale autonomous vehicle. The policies learned in VISTA exhibit sim-to-real transfer without modification and greater robustness than those trained exclusively on real-world data.

Temporal patterns (how vehicles behave in our observed past) underline our reasoning of how people drive on the road, and can explain why we make certain predictions about interactions among road agents. In this paper we propose the ConceptNet trajectory predictor - a novel prediction framework that is able to incorporate agent interactions as explicit edges in a temporal knowledge graph. We demonstrate the sample efficiency and the overall accuracy of the proposed approach, and show that using the graphical structure to explicitly model interactions enables better detection of agent interactions and improved trajectory predictions on a large real-world driving dataset.

Soft robots can be incredibly robust and safe but typically fail to match the strength and precision of rigid robots. This dichotomy between soft and rigid is recently starting to break down, with emerging research interest in hybrid soft-rigid robots. In this work, we draw inspiration from Nature, which achieves the best of both worlds by coupling soft and rigid tissues-like muscle and bone-to produce biological systems capable of both robustness and strength. We present foundational, general-purpose pipelines to simulate and fabricate cable-driven soft-rigid robots with embedded skeletons. We show that robots built using these methods can fluidly mimic biological systems while achieving greater force output and external load resistance than purely soft robots. Finally, we show how our simulation and fabrication pipelines can be leveraged to create more complex robots and do model-based control.

World models learn behaviors in a latent imagination space to enhance the sample-efficiency of deep reinforcement learning (RL) algorithms. While learning world models for high-dimensional observations (e.g., pixel inputs) has become practicable on standard RL benchmarks and some games, their effectiveness in real-world robotics applications has not been explored. In this paper, we investigate how such agents generalize to real-world autonomous vehicle control tasks, where advanced model-free deep RL algorithms fail. In particular, we set up a series of time-lap tasks for an F1TENTH racing robot, equipped with a high-dimensional LiDAR sensor, on a set of test tracks with a gradual increase in their complexity. In this continuous-control setting, we show that model-based agents capable of learning in imagination substantially outperform model-free agents with respect to performance, sample efficiency, successful task completion, and generalization. Moreover, we show that the generalization ability of model-based agents strongly depends on the choice of their observation model. We provide extensive empirical evidence for the effectiveness of world models provided with long enough memory horizons in sim2real tasks.

Autonomous latching is essential for autonomous surface vessels (ASV) to reach full independence from human intervention. As part of the ASV Roboat project, a new solution for self-latching maneuvers has been developed and is presented here. We propose a system that has the key requirements of full integration with the navigation control system and zero-gap connection with the dock, the latter being essential for wireless charging of the ASV. Dedicated markers are used to identify docking targets, relying on computer vision algorithms to determine distance and bearing to the target. In its idle state, the locking solution uses mechanical power-off brakes, minimizing energy consumption while ensuring the boat stays in position indefinitely once docked. A prototype of the proposed mechanism has been built and installed in Roboat. Experimental tests showing the mechanism performance and capability to autonomously approach the docking station are discussed in this work.

We apply a novel framework for decomposing and reasoning about free space in an environment to a multi-agent persistent monitoring problem. Our decomposition method represents free space as a collection of ellipsoids associated with a weighted connectivity graph. The same ellipsoids used for reasoning about connectivity and distance during high level planning can be used as state constraints in a Model Predictive Control algorithm to enforce collision-free motion. This structure allows for streamlined implementation in distributed multi-agent tasks in 2D and 3D environments. We illustrate its effectiveness for a team of tracking agents tasked with monitoring a group of target agents. Our algorithm uses the ellipsoid decomposition as a primitive for the coordination, path planning, and control of the tracking agents. Simulations with four tracking agents monitoring fifteen dynamic targets in obstacle-rich environments demonstrate the performance of our algorithm.

This paper outlines the design, specifications, and algorithms for a new modular self-reconfigurable robotic system; at its foundation is a novel modular magnetically geared linear actuator paired with a kinematic coupling connector. Motivating this work is the core idea that high performance actuators as well as inexpensive, precise and repeatable connectors are the key ingredients required for useful real-world self-reconfiguring machines. This work builds upon existing research in the areas of modular self-reconfigurable robots, magnetic lead screws, modular machine tools and kinematic couplings. Magnetic lead screws (MLS) have many desirable characteristics applicable to modular robots, including a tolerance for slight misalignments, high efficiency, zero backlash, robustness, inherent series elasticity, high force capability, and the ability to gracefully separate and reattach. Due to their high mechanical efficiency, MLS actuators are able to be combined in parallel to provide for increased forces and stiffness. Our system implements a MLS through two separable elements: brushless motor powered actuators called carts which pair with modular passive tracks which constrain the carts' movement to a line. This paper also explores the design for a connector which is able to precisely align modules through the use of a 4-way symmetric kinematic coupling.

Data-driven simulators promise high data-efficiency for driving policy learning. When used for modelling interactions, this data-efficiency becomes a bottleneck: small underlying datasets often lack interesting and challenging edge cases for learning interactive driving. We address this challenge by proposing a data-driven simulation engine† that uses inpainted ado vehicles for learning robust driving policies. Thus, our approach can be used to learn policies that involve multi-agent interactions and allows for training via state-of-the-art policy learning methods. We evaluate the approach for learning standard interaction scenarios in driving. In extensive experiments, our work demonstrates that the resulting policies can be directly transferred to a full-scale autonomous vehicle without making use of any traditional sim-to-real transfer techniques such as domain randomization.

Autonomous underwater vehicles (AUVs) are spe-cialized robots that are commonly used for seafloor surveying and ocean water sampling. Computational design approaches have emerged to reduce the effort required to design both individual AUVs as well as fleets. As the number and scale of underwater missions increases beyond the capabilities of a single vehicle, fleet level design will become more important. Depending on the mission, the optimal fleet may consist of multiple distinct types of AUVs designed to a variety of specifications. Moreover, the AUVs may differ in both continuous parameters (such as battery capacity) and discrete parameters (such as number and model of thrusters). In this work, we present a computational pipeline for designing these heterogeneous AUV fleets. Using a novel shape design space based on a graph grammar and deformation cages, we can express a variety of AUV architectures with different topologies, component selections, and dimensions. We search this space using a combination of discrete graph search and gradient-based continuous optimization, enabled by a differentiable AUV simulator. Finally, we formulate heterogeneous fleet design as a modified knapsack problem, and solve it using an efficient backtracking-based algorithm. We evaluate our pipeline on a simulated mission with nonuniform design requirements-surveying a section of seafloor with varying depth-and show that the best heterogeneous fleet outperforms the best fleet composed of a single vehicle type.

In biomedical engineering, robotic implants provide new methods to restore and improve bodily function, and regenerate tissue. A significant challenge with the design of these devices is to safely actuate them for weeks or months, while they are residing in a patient's body. Magnetic, and other force-at-distance actuation methods, allow mechanisms to be controlled remotely and without contact or line of sight to the device. In this paper, we present a novel magnetic field driven wireless motor. The motor drives a robotic implant for the treatment of long gap esophageal atresia and short bowel syndrome. The motor is equipped with two oppositely oriented permanent magnets which experience forces in opposite directions when a magnetic field is applied tangential to the magnets' directions. The implant can produce a force of 2 N. It is demonstrated with an ex vivo porcine esophagus.

Unmanned aerial vehicles (UAVs) have broad applications including disaster response, transportation, photography, and mapping. A significant bottleneck in the development of UAVs is the limited availability of automatic tools for task-specific co-design of a UAV's shape and controller. The development of such tools is particularly challenging as UAVs can take many forms, including fixed-wing planes, radial copters, and hybrid topologies, with each class of topology showing different advantages. In this work, we present a computational design pipeline for UAVs based on a graph grammar that can search across a wide range of topologies. Graphs generated by the grammar encode different topologies and component selections, while continuous parameters encode the dimensions and properties of each component. We further augment the shape representation with deformation cages, which allow expressing a variety of wing shapes. Each UAV design is associated with an LQR controller with tunable continuous parameters. To search over this complex discrete and continuous design space, we develop a hybrid algorithm that combines discrete graph search strategies and gradient-based continuous optimization methods using a differentiable UAV simulator. We evaluate our pipeline on a set of simulated flight tasks requiring dynamic motions, showing that it discovers novel UAV designs that outperform canonical UAVs typically made by engineers.

This paper introduces ActionSense, a multimodal dataset and recording framework with an emphasis on wearable sensing in a kitchen environment. It provides rich, synchronized data streams along with ground truth data to facilitate learning pipelines that could extract insights about how humans interact with the physical world during activities of daily living, and help lead to more capable and collaborative robot assistants. The wearable sensing suite captures motion, force, and attention information; it includes eye tracking with a first-person camera, forearm muscle activity sensors, a body-tracking system using 17 inertial sensors, finger-tracking gloves, and custom tactile sensors on the hands that use a matrix of conductive threads. This is coupled with activity labels and with externally-captured data from multiple RGB cameras, a depth camera, and microphones. The specific tasks recorded in ActionSense are designed to highlight lower-level physical skills and higher-level scene reasoning or action planning. They include simple object manipulations (e.g., stacking plates), dexterous actions (e.g., peeling or cutting vegetables), and complex action sequences (e.g., setting a table or loading a dishwasher). The resulting dataset and underlying experiment framework are available at https://action-sense.csail.mit.edu. Preliminary networks and analyses explore modality subsets and cross-modal correlations. ActionSense aims to support applications including learning from demonstrations, dexterous robot control, cross-modal predictions, and fine-grained action segmentation. It could also help inform the next generation of smart textiles that may one day unobtrusively send rich data streams to in-home collaborative or autonomous robot assistants.

Two key challenges facing modern deep learning is mitigating deep networks vulnerability to adversarial attacks, and understanding deep learning's generalization capabilities. Towards the first issue, many defense strategies have been developed, with the most common being Adversarial Training (AT). Towards the second challenge, one of the dominant theories that has emerged is the Neural Tangent Kernel (NTK) -- a characterization of neural network behavior in the infinite-width limit. In this limit, the kernel is frozen and the underlying feature map is fixed. In finite-widths however, there is evidence that feature learning happens at the earlier stages of the training (kernel learning) before a second phase where the kernel remains fixed (lazy training). While prior work has aimed at studying adversarial vulnerability through the lens of the frozen infinite-width NTK, there is no work which studies adversarial robustness of NTK during training. In this work, we perform an empirical study of the evolution of the NTK under standard and adversarial training, aiming to disambiguate the effect of adversarial training on kernel learning and lazy training. We find under adversarial training, the NTK rapidly converges to a different kernel (and feature map) than standard training. This new kernel provides adversarial robustness, even when non-robust training is performed on top of it. Furthermore, we find that adversarial training on top of a fixed kernel can yield a classifier with $76.1\%$ robust accuracy under PGD attacks with $\varepsilon = 4/255$ on CIFAR-10.

Dataset distillation compresses large datasets into smaller synthetic coresets which retain performance with the aim of reducing the storage and computational burden of processing the entire dataset. Today's best performing algorithm, \textit{Kernel Inducing Points} (KIP), which makes use of the correspondence between infinite-width neural networks and kernel-ridge regression, is prohibitively slow due to the exact computation of the neural tangent kernel matrix, scaling $O(|S|^2)$, with $|S|$ being the coreset size. To improve this, we propose a novel algorithm that uses a random feature approximation (RFA) of the Neural Network Gaussian Process (NNGP) kernel which reduces the kernel matrix computation to $O(|S|)$. Our algorithm provides at least a 100-fold speedup over KIP and can run on a single GPU. Our new method, termed an RFA Distillation (RFAD), performs competitively with KIP and other dataset condensation algorithms in accuracy over a range of large-scale datasets, both in kernel regression and finite-width network training. We demonstrate the effectiveness of our approach on tasks involving model interpretability and privacy preservation.

The digitization of practically everything coupled with advances in machine learning, the automation of knowledge work, and advanced robotics promises a future with democratized use of machines and wide-spread use of AI, robots and customization. While the last 60 years have defined the field of industrial robots, and empowered hard bodied robots to execute complex assembly tasks in constrained industrial settings, the next 60 years could be ushering in our time with Pervasive robots that come in a diversity of forms and materials, helping people with physical and cognitive tasks. However, the pervasive use of machines remains a hard problem. How can we accelerate the creation of machines customized to specific tasks? Where are the gaps that we need to address in order to advance the bodies and brains of machines? How can we develop scalable and trustworthy reasoning engines?In this talk I will discuss recent developments in machine learning and robotics, focusing on about how computation can play a role in pervasive computation by (1) developing Neural Circuit Policies, an efficient approach to more interpretable machine learning engines, (2) making machines more capable of reasoning in the world, (3) making custom robots, and (4) making more intuitive interfaces between robots and people.

This paper investigates the stochastic on-time arrival (SOTA) problem in transportation networks. We propose a fourth moment approach (FMA), which calculates the tight lower bound of a given routing policy’s on-time-arrival probability, through estimating the first four moments of the policy’s travel time. Then, we employ the generalized policy iteration (GPI) scheme to gradually improve the policy towards the optimal one. Different from state-of-the-art algorithms for the SOTA problem, which require the full travel time distribution and usually incur high computational cost due to the convolution integration operation, FMA only requires the moments of travel-time statistics, which are easily estimated from the statistics perspective. Moreover, the algorithm’s computational complexity analysis indicates the relatively light computational load requirement of FMA. Experimental results in a range of transportation networks show FMA’s superior performance over state of the arts.

We present a novel, fast differentiable simulator for soft-body learning and control applications. Existing differentiable soft-body simulators can be classified into two categories based on their time integration methods: Simulators using explicit timestepping schemes require tiny timesteps to avoid numerical instabilities in gradient computation, and simulators using implicit time integration typically compute gradients by employing the adjoint method and solving the expensive linearized dynamics. Inspired by Projective Dynamics ( PD ), we present Differentiable Projective Dynamics ( DiffPD ), an efficient differentiable soft-body simulator based on PD with implicit time integration. The key idea in DiffPD is to speed up backpropagation by exploiting the prefactorized Cholesky decomposition in forward PD simulation. In terms of contact handling, DiffPD supports two types of contacts: a penalty-based model describing contact and friction forces and a complementarity-based model enforcing non-penetration conditions and static friction. We evaluate the performance of DiffPD and observe it is 4–19 times faster compared with the standard Newton’s method in various applications including system identification, inverse design problems, trajectory optimization, and closed-loop control. We also apply DiffPD in a reality-to-simulation ( real-to-sim ) example with contact and collisions and show its capability of reconstructing a digital twin of real-world scenes.

We introduce a new class of time-continuous recurrent neural network models. Instead of declaring a learning system's dynamics by implicit nonlinearities, we construct networks of linear first-order dynamical systems modulated via nonlinear interlinked gates. The resulting models represent dynamical systems with varying (i.e., liquid) time-constants coupled to their hidden state, with outputs being computed by numerical differential equation solvers. These neural networks exhibit stable and bounded behavior, yield superior expressivity within the family of neural ordinary differential equations, and give rise to improved performance on time-series prediction tasks. To demonstrate these properties, we first take a theoretical approach to find bounds over their dynamics, and compute their expressive power by the trajectory length measure in a latent trajectory space. We then conduct a series of time-series prediction experiments to manifest the approximation capability of Liquid Time-Constant Networks (LTCs) compared to classical and modern RNNs.

Learning composable policies for environments with complex rules and tasks is a challenging problem. We introduce a hierarchical reinforcement learning framework called the Logical Options Framework (LOF) that learns policies that are satisfying, optimal, and composable. LOF efficiently learns policies that satisfy tasks by representing the task as an automaton and integrating it into learning and planning. We provide and prove conditions under which LOF will learn satisfying, optimal policies. And lastly, we show how LOF’s learned policies can be composed to satisfy unseen tasks with only 10-50 retraining steps on our benchmarks. We evaluate LOF on four tasks in discrete and continuous domains, including a 3D pick-and-place environment.

Robustness to variations in lighting conditions is a key objective for any deep vision system. To this end, our paper extends the receptive field of convolutional neural networks with two residual components, ubiquitous in the visual processing system of vertebrates: On-center and off-center pathways, with an excitatory center and inhibitory surround; OOCS for short. The On-center pathway is excited by the presence of a light stimulus in its center, but not in its surround, whereas the Off-center pathway is excited by the absence of a light stimulus in its center, but not in its surround. We design OOCS pathways via a difference of Gaussians, with their variance computed analytically from the size of the receptive fields. OOCS pathways complement each other in their response to light stimuli, ensuring this way a strong edge-detection capability, and as a result an accurate and robust inference under challenging lighting conditions. We provide extensive empirical evidence showing that networks supplied with OOCS pathways gain accuracy and illumination-robustness from the novel edge representation, compared to other baselines.

We present methods for co-designing rigid robots over control and morphology (including discrete topology) over multiple objectives. Previous work has addressed problems in single-objective robot co-design or multi-objective control. However, the joint multi-objective co-design problem is extremely important for generating capable, versatile, algorithmically designed robots. In this work, we present Multi-Objective Graph Heuristic Search, which extends a single-objective graph heuristic search from previous work to enable a highly efficient multi-objective search in a combinatorial design topology space. Core to this approach, we introduce a new universal, multiobjective heuristic function based on graph neural networks that is able to effectively leverage learned information between different task trade-offs. We demonstrate our approach on six combinations of seven terrestrial locomotion and design tasks, including one three-objective example. We compare the captured Pareto fronts across different methods and demonstrate that our multi-objective graph heuristic search quantitatively and qualitatively outperforms other techniques.

Autonomous surface vessels (ASVs) always carry payloads such as passengers and cargoes. The change in the payload can sometimes be several times the weight of the vessel. The payload can cause significant changes in the dynamics of the vessel, thereby degrading the performance of the controller. This paper proposes an adaptive nonlinear model predictive control (A-NMPC) strategy for ASV trajectory tracking, which allows real-time changes in dynamics caused by severe payload variation. First, a nonlinear dynamic model that updates with the vessel’s payload is established. Then a pressure sensing method is proposed to estimate the payload of the vessel. Further, a parametric cost function that considers changing dynamics, as well as input and state constraints, is formulated in the NMPC algorithm. The tracking ability of A-NMPC is systematically studied on three different sizes of vessels in the simulation where the payload of these vessels changes eight times their inherent weight. Numerical results show that when the payload changes greatly the vessels with A-NMPC can accurately track the reference trajectory while the vessels with conventional NMPC cannot. Finally, the tracking experiments with a quarter-scale vessel in a swimming pool further verify the effectiveness of the proposed A-NMPC strategy.

Analyzing surgical workflow is crucial for surgical assistance robots to understand surgeries. With the understanding of the complete surgical workflow, the robots are able to assist the surgeons in intra-operative events, such as by giving a warning when the surgeon is entering specific keys or high-risk phases. Deep learning techniques have recently been widely applied to recognizing surgical workflows. Many of the existing temporal neural network models are limited in their capability to handle long-term dependencies in the data, instead, relying upon the strong performance of the underlying per-frame visual models. We propose a new temporal network structure that leverages task-specific network representation to collect long-term sufficient statistics that are propagated by a sufficient statistics model (SSM). We implement our approach within an LSTM backbone for the task of surgical phase recognition and explore several choices for propagated statistics. We demonstrate superior results over existing and novel state-of-the-art segmentation techniques on two laparoscopic cholecystectomy datasets: the publicly available Cholec80 dataset and MGH100, a novel dataset with more challenging and clinically meaningful segment labels.

This paper addresses the safe path planning problem for autonomous mobility with multi-modal perception uncertainties. Specifically, we assume that different sensor inputs lead to different Gaussian process regulated perception uncertainties (named as multi-modal perception uncertainties). We implement a Bayesian inference algorithm, which merges the multi-modal GP-regulated uncertainties into a unified one and translates the unified uncertainty into a dynamic risk map. With the safe path planner taking the risk map as input, we are able to plan a safe path for the autonomous vehicle to follow. Experimental results on an autonomous golf cart testbed validate the applicability and efficiency of the proposed algorithm.

Adversarial training is an effective method to train deep learning models that are resilient to norm-bounded perturbations, with the cost of nominal performance drop. While adversarial training appears to enhance the robustness and safety of a deep model deployed in open-world decision-critical applications, counterintuitively, it induces undesired behaviors in robot learning settings. In this paper, we show theoretically and experimentally that neural controllers obtained via adversarial training are subjected to three types of defects, namely transient, systematic, and conditional errors. We first generalize adversarial training to a safety-domain optimization scheme allowing for more generic specifications. We then prove that such a learning process tends to cause certain error profiles. We support our theoretical results by a thorough experimental safety analysis in a robot-learning task. Our results suggest that adversarial training is not yet ready for robot learning.

Deep learning has been used to demonstrate end-to-end neural network learning for autonomous vehicle control from raw sensory input. While LiDAR sensors provide reliably accurate information, existing end-to-end driving solutions are mainly based on cameras since processing 3D data requires a large memory footprint and computation cost. On the other hand, increasing the robustness of these systems is also critical; however, even estimating the model’s uncertainty is very challenging due to the cost of sampling-based methods. In this paper, we present an efficient and robust LiDAR-based end-to-end navigation framework. We first introduce Fast-LiDARNet that is based on sparse convolution kernel optimization and hardware-aware model design. We then propose Hybrid Evidential Fusion that directly estimates the uncertainty of the prediction from only a single forward pass and then fuses the control predictions intelligently. We evaluate our system on a full-scale vehicle and demonstrate lane-stable as well as navigation capabilities. In the presence of out-of-distribution events (e.g., sensor failures), our system significantly improves robustness and reduces the number of takeovers in the real world.

Autonomous urban driving among human-driven cars requires a holistic understanding of road rules, driver intents and driving styles. This is challenging as a short-term, single instance, driver intent of lane change may not correspond to their driving styles for a longer duration. This paper presents an interactive behavior planner which accounts for road context, short-term driver intent, and long-term driving style to infer beliefs over the latent states of surrounding vehicles. We use a specialized Partially Observable Markov Decision Process to provide risk-averse decisions. Specifically, we consider adversarial driving scenarios caused by irrational drivers to validate the robustness of our proposed interactive behavior planner in simulation as well as on a full-size self-driving car. Our experimental results show that our algorithm enables safer and more travel time-efficient autonomous driving compared to baselines even in adversarial scenarios.

Navigation through dynamic pedestrian environments in a socially compliant manner is still a challenging task for autonomous vehicles. Classical methods usually lead to unnatural vehicle behaviours for pedestrian navigation due to the difficulty in modeling social conventions mathematically. This paper presents an end-to-end path planning system that achieves autonomous navigation in dynamic environments through imitation learning. The proposed system is based on a fully convolutional neural network that maps the raw sensory data into a confidence map for path extraction. Additionally, a classification network is introduced to reduce the unnecessary re-plannings and ensures that the vehicle goes back to the global path when re-planning is not needed. The imitation learning based path planner is implemented on an autonomous wheelchair and tested in a new real-world dynamic pedestrian environment. Experimental results show that the proposed system is able to generate paths for different driving tasks, such as pedestrian following, static and dynamic obstacles avoidance, etc. In comparison to the state-of-the-art method, our system is superior in terms of generating human-like trajectories.

We propose a methodology for robust, real-time place recognition using an imaging lidar, which yields image-quality high-resolution 3D point clouds. Utilizing the intensity readings of an imaging lidar, we project the point cloud and obtain an intensity image. ORB feature descriptors are extracted from the image and encoded into a bag-of-words vector. The vector, used to identify the point cloud, is inserted into a database that is maintained by DBoW for fast place recognition queries. The returned candidate is further validated by matching visual feature descriptors. To reject matching outliers, we apply PnP, which minimizes the reprojection error of visual features’ positions in Euclidean space with their correspondences in 2D image space, using RANSAC. Combining the advantages from both camera and lidar-based place recognition approaches, our method is truly rotation-invariant, and can tackle reverse revisiting and upside down revisiting. The proposed method is evaluated on datasets gathered from a variety of platforms over different scales and environments. Our implementation and datasets are available at https://git.io/image-lidar.

We propose a framework for tightly-coupled lidar-visual-inertial odometry via smoothing and mapping, LVI-SAM, that achieves real-time state estimation and map-building with high accuracy and robustness. LVI-SAM is built atop a factor graph and is composed of two sub-systems: a visual-inertial system (VIS) and a lidar-inertial system (LIS). The two sub-systems are designed in a tightly-coupled manner, in which the VIS leverages LIS estimation to facilitate initialization. The accuracy of the VIS is improved by extracting depth information for visual features using lidar measurements. In turn, the LIS utilizes VIS estimation for initial guesses to support scan-matching. Loop closures are first identified by the VIS and further refined by the LIS. LVI-SAM can also function when one of the two sub-systems fails, which increases its robustness in both texture-less and feature-less environments. LVI-SAM is extensively evaluated on datasets gathered from several platforms over a variety of scales and environments. Our implementation is available at https://git.io/lvi-sam.

Autonomous driving in rainy conditions remains a big challenge. One of the issues is sensor degradation. LiDAR is commonly used in autonomous driving systems to perceive and understand surrounding environments. However, LiDAR performance can be degraded by rain, thereby influencing other system performance (e.g., perception or localization). Therefore, knowing how much degradation exists in current LiDAR measurements is necessary. Most existing methods can only measure LiDAR degradation in controlled environments (e.g., a chamber with simulated rain); how to quantify LiDAR degradation in dynamic environments while the autonomous vehicle is moving is still a difficult problem. In this work, we propose a novel approach to address this problem using an anomaly detection method. Our method has been evaluated on simulated and real-world data. Experimental results demonstrate the effectiveness of our method to capture LiDAR degradation and yield reasonable degradation estimations. Our experimental data and codes are accessible from http://rain.smart.mit.edu/smartrain/.

Autonomous control of an emergency vehicle will save lives through faster transport and shorter response. Towards this goal, it must overcome the challenge of inter- acting with existing human drivers on the road. We present a game-theoretic approach for semi-cooperative control of an autonomous emergency vehicle that can interact efficiently with humans on the road. We model the interactions between autonomous and human driven cars with Social Value Orientation, a metric from social psychology, that allows the controller to leverage their influence on the trajectories of neighboring human drivers. In addition, by using a modified version of iterative best response, we direct the algorithm to converge to Nash equilibria that are cooperative. We demonstrate the efficacy of our algorithm in simulations of drivers in traffic, with a variety of traffic densities and driver personalities. In simulations of prosocial human drivers, our algorithm provides an 8% improvement in distance-traveled compared to egoistic human drivers.

Autonomous vehicles on city roads and especially in pedestrian environments require agility to navigate narrow passages and turn in tight spaces, leading to the need for a real-time, robust and adaptable controller. In this paper, we present orientation and context aware controllers for autonomous vehicles that can closely track the reference path wit alh respect to the current state of the vehicle, environmental properties, and the desired target orientation at the desired target location. Our proposed controllers are derived from the widely used pure pursuit controller. We validate our proposed controllers with respect to the baseline pure pursuit controller in simulation and on a full-size autonomous vehicle in a pedestrian environment. Our experimental results suggest significant improvements in adaptability and tracking performance compared to the pure pursuit controller.

This paper presents a mobile UVC disinfection robot designed to mitigate the threat of airborne and surface pathogens. Our system comprises a mobile robot base, a custom UVC lamp assembly, and algorithms for autonomous navigation and path planning. We present a model of UVC disinfection and dosage of UVC light delivered by the mobile robot. We also discuss challenges and prototyping decisions for rapid deployment of the robot during the COVID-19 pandemic. Experimental results summarize a long-term deployment at The Greater Boston Food Bank, where the robot delivers (nightly) UVC dosages of at least 10 mJ/cm2to a 4000 ft2area in under 30 minutes. These dosages are capable of neutralizing 99% of coronaviruses, including SARS-CoV-2, on surfaces and in airborne particles. Further simulations present how this mobile UVC disinfection robot may be extended to classic problems in robotic path planning and adaptive multi-robot coverage control.

We address the problem of assigning a team of drones to autonomously capture a set desired shots of a dynamic target in the presence of obstacles. We present a two-stage planning pipeline that generates offline an assignment of drone to shots and locally optimizes online the viewpoint. Given desired shot parameters, the high-level planner uses a visibility heuristic to predict good times for capturing each shot and uses an Integer Linear Program to compute drone assignments. An online Model Predictive Control algorithm uses the assignments as reference to capture the shots. The algorithm is validated in hardware with a pair of drones and a remote controlled car.

This paper studies the group multi-object tracking (MOT) problem in dynamic pedestrian environments, with intended application to safe navigation for autonomous vehicles. We complete a full autonomous vehicle navigation pipeline from object detection, tracking, grouping, to risk map generation and safe path planning. Our main contribution is to instantiate a group multi-object tracking algorithm, which provides the crucial grouped activity information, i.e. group position, group velocity, group size, to the risk map generator, and therewith produce a stable and robust risk map for the downstream safe path planner. Experimental results with real world data show the socially acceptable, robust and stable performance of the proposed algorithm over its individual MOT counterpart.

Bladed tools such as jigsaws are common tools for wood workers on job-sites and in workshops, but do not currently have sufficient autonomous hardware or path planning algorithms to enable automation. Here we present a system of an autonomous robot and a path planning algorithm for automating jigsaw operations. The robot can drill holes, insert the jigsaw, and cut plywood. Our algorithm converts complex shapes into paths for the jigsaw, drill holes, and traversal movements for the robot. The algorithm decomposes input shapes into cuttable sections and determines possible locations for drilling entry holes for inserting the blade. We cast the drill hole problem as a set coverage problem with a trade-off between number of holes and cutting distance. We characterize the algorithm on a series of shapes and determined the algorithm found valid solutions. We executed an example on the robot to demonstrate the end-to-end system.

Summary form only given, as follows. The complete presentation was not made available for publication as part of the conference proceedings. Learning representations is critical for machine learning and very computation-intensive process. There are many opportunities to introduce efficiencies through parallel and distributed computing. The success of machine learning algorithms depends on data representation. Different representations can expose or hide different features of the data. As we think about the future of learning representations and its impact on machine learning it is important to consider the state of the art of machine learning today, the challenges and opportunities for addressing the computation issues around representation learning, and how to get to deeper understanding and more capabilities in machine learned models. In this talk I will describe 4 ideas related to computational issues in representation learning: reducing uncertainty, developing compact representations, debasing the training data, and developing privacy-preserving representations.

Continuous deep learning architectures enable learning of flexible probabilistic models for predictive modeling as neural ordinary differential equations (ODEs), and for generative modeling as continuous normalizing flows. In this work, we design a framework to decipher the internal dynamics of these continuous depth models by pruning their network architectures. Our empirical results suggest that pruning improves generalization for neural ODEs in generative modeling. We empirically show that the improvement is because pruning helps avoid mode-collapse and flatten the loss surface. Moreover, pruning finds efficient neural ODE representations with up to 98% less parameters compared to the original network, without loss of accuracy. We hope our results will invigorate further research into the performance-size trade-offs of modern continuous-depth models.

We present a novel global compression framework for deep neural networks that automatically analyzes each layer to identify the optimal per-layer compression ratio, while simultaneously achieving the desired overall compression. Our algorithm hinges on the idea of compressing each convolutional (or fully-connected) layer by slicing its channels into multiple groups and decomposing each group via low-rank decomposition. At the core of our algorithm is the derivation of layer-wise error bounds from the Eckart–Young–Mirsky theorem. We then leverage these bounds to frame the compression problem as an optimization problem where we wish to minimize the maximum compression error across layers and propose an efficient algorithm towards a solution. Our experiments indicate that our method outperforms existing low-rank compression approaches across a wide range of networks and data sets. We believe that our results open up new avenues for future research into the global performance-size trade-offs of modern neural networks.

Reinforcement learning (RL) for continuous control typically employs distributions whose support covers the entire action space. In this work, we investigate the colloquially known phenomenon that trained agents often prefer actions at the boundaries of that space. We draw theoretical connections to the emergence of bang-bang behavior in optimal control, and provide extensive empirical evaluation across a variety of recent RL algorithms. We replace the normal Gaussian by a Bernoulli distribution that solely considers the extremes along each action dimension - a bang-bang controller. Surprisingly, this achieves state-of-the-art performance on several continuous control benchmarks - in contrast to robotic hardware, where energy and maintenance cost affect controller choices. Since exploration, learning, and the final solution are entangled in RL, we provide additional imitation learning experiments to reduce the impact of exploration on our analysis. Finally, we show that our observations generalize to environments that aim to model real-world challenges and evaluate factors to mitigate the emergence of bang-bang solutions. Our findings emphasise challenges for benchmarking continuous control algorithms, particularly in light of potential real-world applications.

Imitation learning enables high-fidelity, vision-based learning of policies within rich, photorealistic environments. However, such techniques often rely on traditional discrete-time neural models and face difficulties in generalizing to domain shifts by failing to account for the causal relationships between the agent and the environment. In this paper, we propose a theoretical and experimental framework for learning causal representations using continuous-time neural networks, specifically over their discrete-time counterparts. We evaluate our method in the context of visual-control learning of drones over a series of complex tasks, ranging from short- and long-term navigation, to chasing static and dynamic objects through photorealistic environments. Our results demonstrate that causal continuous-time deep models can perform robust navigation tasks, where advanced recurrent models fail. These models learn complex causal control representations directly from raw visual inputs and scale to solve a variety of tasks using imitation learning.

In this article, we propose a trajectory planning algorithm that enables autonomous surface vessels to perform socially compliant navigation in a city's canal. The key idea behind the proposed algorithm is to adopt an optimal control formulation in which the deviation of movements of the autonomous vessel from nominal movements of human-operated vessels is penalized. Consequently, given a pair of origin and destination points, it finds vessel trajectories that resemble those of human-operated vessels. To formulate this, we adopt kernel density estimation (KDE) to build a nominal movement model of human-operated vessels from a prerecorded trajectory dataset, and use a Kullback-Leibler control cost to measure the deviation of the autonomous vessel's movements from the model. We establish an analogy between our trajectory planning approach and the maximum entropy inverse reinforcement learning (MaxEntIRL) approach to explain how our approach can learn the navigation behavior of human-operated vessels. On the other hand, we distinguish our approach from the MaxEntIRL approach in that it does not require well-defined bases, often referred to as features, to construct its cost function as required in many of inverse reinforcement learning approaches in the trajectory planning context. Through experiments using a dataset of vessel trajectories collected from the automatic identification system, we demonstrate that the trajectories generated by our approach resemble those of human-operated vessels and that using them for canal navigation is beneficial in reducing head-on encounters between vessels and improving navigation safety.

Information gathering while interacting with other agents under sensing and motion uncertainty is critical in domains such as driving, service robots, racing, or surveillance. The interests of agents may be at odds with others, resulting in a stochastic noncooperative dynamic game. Agents must predict others’ future actions without communication, incorporate their actions into these predictions, account for uncertainty and noise in information gathering, and consider what information their actions reveal. Our solution uses local iterative dynamic programming in Gaussian belief space to solve a game-theoretic continuous POMDP. Solving a quadratic game in the backward pass of a game-theoretic belief-space variant of iterative linear-quadratic Gaussian control (iLQG) achieves a runtime polynomial in the number of agents and linear in the planning horizon. Our algorithm yields linear feedback policies for our robot, and predicted feedback policies for other agents. We present three applications: Active surveillance, guiding eyes for a blind agent, and autonomous racing. Agents with game-theoretic belief-space planning win 44% more races than without game theory and 34% more than without belief-space planning.

We introduce a method to learn imitative policies from expert demonstrations that are interpretable and manipulable. We achieve interpretability by modeling the interactions between high-level actions as an automaton with connections to formal logic. We achieve manipulability by integrating this automaton into planning, so that changes to the automaton have predictable effects on the learned behavior. These qualities allow a human user to first understand what the model has learned, and then either correct the learned behavior or zero-shot generalize to new, similar tasks. We build upon previous work by no longer requiring additional supervised information which is hard to collect in practice. We achieve this by using a deep Bayesian nonparametric hierarchical model. We test our model on several domains and also show results for a real-world implementation on a mobile robotic arm platform.

As the capacity for machines to extend human capabilities continues to grow, the communication channels used must also expand. Allowing machines to interpret nonverbal commands such as gestures can help make interactions more similar to interactions with another person. Yet to be pervasive and effective in realistic scenarios, such interfaces should not require significant sensing infrastructure or per-user setup time. The presented work takes a step towards these goals by using wearable muscle and motion sensors to detect gestures without dedicated calibration or training procedures. An algorithm is presented for clustering unlabeled streaming data in real time, and it is applied to adaptively thresholding muscle and motion signals acquired via electromyography (EMG) and an inertial measurement unit (IMU). This enables plug-and-play online detection of arm stiffening, fist clenching, rotation gestures, and forearm activation. It also augments a neural network pipeline, trained only on strategically chosen training data from previous users, to detect left, right, up, and down gestures. Together, these pipelines offer a plug-and-play gesture vocabulary suitable for remotely controlling a robot. Experiments with 6 subjects evaluate classifier performance and interface efficacy. Classifiers correctly identified 97.6% of 1,200 cued gestures, and a drone correctly responded to 81.6% of 1,535 unstructured gestures as subjects remotely controlled it through target hoops during 119 minutes of total flight time.

We propose a neural information processing system obtained by re-purposing the function of a biological neural circuit model to govern simulated and real-world control tasks. Inspired by the structure of the nervous system of the soil-worm, C. elegans, we introduce ordinary neural circuits (ONCs), defined as the model of biological neural circuits reparameterized for the control of alternative tasks. We first demonstrate that ONCs realize networks with higher maximum flow compared to arbitrary wired networks. We then learn instances of ONCs to control a series of robotic tasks, including the autonomous parking of a real-world rover robot. For reconfiguration of the purpose of the neural circuit, we adopt a search-based optimization algorithm. Ordinary neural circuits perform on par and, in some cases, significantly surpass the performance of contemporary deep learning models. ONC networks are compact, 77% sparser than their counterpart neural controllers, and their neural dynamics are fully interpretable at the cell-level.

Many real-world control problems involve conflicting objectives where we desire a dense and high-quality set of control policies that are optimal for different objective preferences (called Pareto-optimal). While extensive research in multi-objective reinforcement learning (MORL) has been conducted to tackle such problems, multi-objective optimization for complex continuous robot control is still under-explored. In this work, we propose an efficient evolutionary learning algorithm to find the Pareto set approximation for continuous robot control problems, by extending a state-of-the-art RL algorithm and presenting a novel prediction model to guide the learning process. In addition to efficiently discovering the individual policies on the Pareto front, we construct a continuous set of Pareto-optimal solutions by Pareto analysis and interpolation. Furthermore, we design seven multi-objective RL environments with continuous action space, which is the first benchmark platform to evaluate MORL algorithms on various robot control problems. We test the previous methods on the proposed benchmark problems, and the experiments show that our approach is able to find a much denser and higher-quality set of Pareto policies than the existing algorithms.

The safety of an autonomous vehicle not only depends on its own perception of the world around it, but also on the perception and recognition from other vehicles. If an ego vehicle considers the uncertainty other vehicles have about itself, then by reducing the estimated uncertainty it can increase its safety. In this paper, we focus on how an ego vehicle plans its trajectories through the blind spots of other vehicles. We create visibility-aware planning, where the ego vehicle chooses its trajectories such that it reduces the perceived uncertainty other vehicles may have about the state of the ego vehicle. We present simulations of traffic and highway environments, where an ego vehicle must pass another vehicle, make a lane change, or traverse a partially-occluded intersection. Emergent behavior shows that when using visibility-aware planning, the ego vehicle spends less time in a blind spot, and may slow down before entering the blind spot so as to increase the likelihood other vehicles perceive the ego vehicle.

The success of hybrid suction + parallel-jaw grippers in the Amazon Robotics/Picking Challenge have demonstrated the effectiveness of multimodal grasping approaches. However, existing multimodal grippers combine grasping modes in isolation and do not incorporate the benefits of compliance found in soft robotic manipulators. In this paper, we present a gripper that integrates three modes of grasping: suction, parallel jaw, and soft fingers. Using complaint handed shearing auxetics actuators as the foundation, this gripper is able to multiplex manipulation by creating unique grasping primitives through permutations of these grasping techniques. This gripper is able to grasp 88% of tested objects, 14% of which could only be grasped using a combination of grasping modes. The gripper is also able to perform in-hand object re-orientation of flat objects without the need for pre-grasp manipulation.

As artificial intelligence becomes an increasingly prevalent method of enhancing robotic capabilities, it is important to consider effective ways to train these learning pipelines and to leverage human expertise. Working towards these goals, a master-apprentice model is presented and is evaluated during a grasping task for effectiveness and human perception. The apprenticeship model augments self-supervised learning with learning by demonstration, efficiently using the human's time and expertise while facilitating future scalability to supervision of multiple robots; the human provides demonstrations via virtual reality when the robot cannot complete the task autonomously. Experimental results indicate that the robot learns a grasping task with the apprenticeship model faster than with a solely self-supervised approach and with fewer human interventions than a solely demonstration-based approach; 100% grasping success is obtained after 150 grasps with 19 demonstrations. Preliminary user studies evaluating workload, usability, and effectiveness of the system yield promising results for system scalability and deployability. They also suggest a tendency for users to overestimate the robot's skill and to generalize its capabilities, especially as learning improves.

The goal of achieving `universal grasping' where many objects can be handled with minimal control input is the focus of much research due to potential high impact applications ranging from grocery packing to recycling. However, many of the grippers developed suffer from limited sensing capabilities which can prevent handing of both heavy bulky items and also lightweight delicate objects which require fine control when grasping. Sensorizing such grippers is often challenging due to the highly deformable surfaces. We propose a novel sensing approach which uses highly flexible latex bladders. By measuring changes in the air pressure of the bladders, normal force and longitudinal strain can be measured. These sensors have been integrated into a `Magic Ball' origami gripper to provide both tactile and proprioceptive sensing. The sensors show reasonable sensitivity and repeatability, are durable and low-cost, and can be easily integrated into the gripper without affecting performance. When the sensors are used for classification, they enabled identification of 10 objects with over 90% accuracy, and also allow failure to be detected through slippage detection. A control algorithm has been developed which uses the sensor feedback to extend the capabilities of the gripper to include both delicate and strong grasping. It is shown that this closed loop controller enables delicate grasping of potato chips; 80% of those tested were grasped without damage.

Traditional robotic control suits require profound task-specific knowledge for designing, building and testing control software. The rise of Deep Learning has enabled end-to-end solutions to be learned entirely from data, requiring minimal knowledge about the application area. We design a learning scheme to train end-to-end linear dynamical systems (LDS)s by gradient descent in imitation learning robotic domains. We introduce a new regularization loss component together with a learning algorithm that improves the stability of the learned autonomous system, by forcing the eigenvalues of the internal state updates of an LDS to be negative reals. We evaluate our approach on a series of real-life and simulated robotic experiments, in comparison to linear and nonlinear Recurrent Neural Network (RNN) architectures. Our results show that our stabilizing method significantly improves test performance of LDS, enabling such linear models to match the performance of contemporary nonlinear RNN architectures. A video of the obstacle avoidance performance of our method on a mobile robot, in unseen environments, compared to other methods can be viewed at https://youtu.be/mhEsCoNao5E.

This paper introduces the Weighted Buffered Voronoi tessellation, which allows us to define distributed, semicooperative multi-agent navigation policies with guarantees on collision avoidance. We generate the Voronoi cells with dynamic weights that bias the boundary towards the agent with the lower relative weight while always maintaining a buffered distance between two agents. By incorporating agent weights, we can encode selfish or prioritized behavior among agents, where a more selfish agent will have a larger relative cell over less selfish agents. We consider this semi-cooperative since agents do not cooperate in symmetric ways. Furthermore, when all agents start in a collision-free configuration and plan their control actions within their cells, we prove that no agents will collide. Simulations demonstrate the performance of our algorithm for agents navigating to goal locations in a position-swapping game. We observe that agents with more egoistic weights consistently travel shorter paths to their goal than more altruistic agents.

We propose a scalable cooperative control approach which coordinates a group of rigidly connected autonomous surface vessels to track desired trajectories in a planar water environment as a single floating modular structure. Our approach leverages the implicit information of the structure's motion for force and torque allocation without explicit communication among the robots. In our system, a leader robot steers the entire group by adjusting its force and torque according to the structure's deviation from the desired trajectory, while follower robots run distributed consensus-based controllers to match their inputs to amplify the leader's intent using only onboard sensors as feedback. To cope with the nonlinear system dynamics in the water, the leader robot employs a nonlinear model predictive controller (NMPC), where we experimentally estimated the dynamics model of the floating modular structure in order to achieve superior performance for leader-following control. Our method has a wide range of potential applications in transporting humans and goods in many of today's existing waterways. We conducted trajectory and orientation tracking experiments in hardware with three custom-built autonomous modular robotic boats, called Roboat, which are capable of holonomic motions and onboard state estimation. Simulation results with up to 65 robots also prove the scalability of our proposed approach.

This paper presents a novel autonomous surface vessel (ASV), called Roboat II for urban transportation. Roboat II is capable of accurate simultaneous localization and mapping (SLAM), receding horizon tracking control and estimation, and path planning. Roboat II is designed to maximize the internal space for transport, and can carry payloads several times of its own weight. Moreover, it is capable of holonomic motions to facilitate transporting, docking, and inter-connectivity between boats. The proposed SLAM system receives sensor data from a 3D LiDAR, an IMU, and a GPS, and utilizes a factor graph to tackle the multi-sensor fusion problem. To cope with the complex dynamics in the water, Roboat II employs an online nonlinear model predictive controller (NMPC), where we experimentally estimated the dynamical model of the vessel in order to achieve superior performance for tracking control. The states of Roboat II are simultaneously estimated using a nonlinear moving horizon estimation (NMHE) algorithm. Experiments demonstrate that Roboat II is able to successfully perform online mapping and localization, plan its path and robustly track the planned trajectory in the confined river, implying that this autonomous vessel holds the promise on potential applications in transporting humans and goods in many of the waterways nowadays.

Spatial mapping of surface roughness is a critical enabling technology for automating adaptive sanding operations. We leverage GelSight sensors to convert the problem of surface roughness measurement into a vision classification problem. By combining GelSight sensors with Optitrack positioning systems we attempt to develop an accurate spatial mapping of surface roughness that can compare to human touch, the current state of the art for large scale manufacturing. To perform the classification, we propose the use of Bayesian neural networks in conjunction with uncertainty-aware prediction. We compare the sensor and network with a human baseline for both absolute and relative texture classification. To establish a baseline, we collected performance data from humans on their ability to classify materials into 60, 120, and 180 grit sanded pine boards. Our results showed that the probabilistic network performs at the level of human touch for absolute and relative classifications. Using the Bayesian approach enables establishing a confidence bound on our prediction. We were able to integrate the sensor with Optitrack to provide a spatial map of sanding grit applied to pine boards. From this result, we can conclude that GelSight with Bayesian neural networks can learn accurate representations for sanding, and could be a significant enabling technology for closed loop robotic sanding operations.

This paper investigates the safe path planning problem for an autonomous vehicle operating in unstructured, cluttered environments. While some objects may be accurately with canonical perception algorithms, other objects and clutter may be harder to track. We present an approach that combines two methods of risk assessment: for objects with reliable tracking, we use a Gaussian Process (GP) regulated risk map to describe the risk map information; for unknown objects that we fail to accurately track, we compute a Dynamic Risk Density (DRD) from the overall occupancy and velocity field from LiDAR scan snapshots. Several methods are proposed for combining the GP risk map and DRD, and the resultant hybrid risk map is used for the proposed safe path planning algorithm. Experimental results on an autonomous buggy show that the hybrid risk map is able to yield a safe path planner to navigate the autonomous testbed within the cluttered environments.

Many indoor spaces have constantly changing layouts and may not be mapped by an autonomous vehicle, yet maps such as floor plans or evacuation maps of these places are common. We propose a method for an autonomous robot to localize itself on such maps with inconsistent scale using Stochastic Gradient Descent (SGD) with scan matching using a 2D LiDAR. We also introduce a new scale state in 2D localization to manage the possible inconsistent scale of the input map. Experiments are conducted in an indoor corridor using three different input maps - a point cloud, a floor plan, and a hand-drawn map. The SGD localization algorithm is bench-marked to Adaptive Monte Carlo Localization (AMCL). In a point cloud mapped environment, our algorithm achieves 0.264m and 5.26° average position and heading error respectively. On the hand-drawn map, our SGD localization algorithm remains robust while AMCL fails. The role of the scale state in our SGD localization algorithm is demonstrated in poorly scaled maps.

We propose a framework for tightly-coupled lidar inertial odometry via smoothing and mapping, LIO-SAM, that achieves highly accurate, real-time mobile robot trajectory estimation and map-building. LIO-SAM formulates lidar-inertial odometry atop a factor graph, allowing a multitude of relative and absolute measurements, including loop closures, to be incorporated from different sources as factors into the system. The estimated motion from inertial measurement unit (IMU) pre-integration de-skews point clouds and produces an initial guess for lidar odometry optimization. The obtained lidar odometry solution is used to estimate the bias of the IMU. To ensure high performance in real-time, we marginalize old lidar scans for pose optimization, rather than matching lidar scans to a global map. Scan-matching at a local scale instead of a global scale significantly improves the real-time performance of the system, as does the selective introduction of keyframes, and an efficient sliding window approach that registers a new keyframe to a fixed-size set of prior "sub-keyframes." The proposed method is extensively evaluated on datasets gathered from three platforms over various scales and environments.

Deterministic neural networks (NNs) are increasingly being deployed in safety critical domains, where calibrated, robust, and efficient measures of uncertainty are crucial. In this paper, we propose a novel method for training non-Bayesian NNs to estimate a continuous target as well as its associated evidence in order to learn both aleatoric and epistemic uncertainty. We accomplish this by placing evidential priors over the original Gaussian likelihood function and training the NN to infer the hyperparameters of the evidential distribution. We additionally impose priors during training such that the model is regularized when its predicted evidence is not aligned with the correct output. Our method does not rely on sampling during inference or on out-of-distribution (OOD) examples for training, thus enabling efficient and scalable uncertainty learning. We demonstrate learning well-calibrated measures of uncertainty on various benchmarks, scaling to complex computer vision tasks, as well as robustness to adversarial and OOD test samples.

Abstract The k-nearest neighbors (kNN) classifier predicts a class of a query, q, by taking the majority class of its k neighbors in an existing (already classified) database, S. In secure kNN, q and S are owned by two different parties and q is classified without sharing data. In this work we present a classifier based on kNN, that is more efficient to implement with homomorphic encryption (HE). The efficiency of our classifier comes from a relaxation we make to consider κ nearest neighbors for κ ≈k with probability that increases as the statistical distance between Gaussian and the distribution of the distances from q to S decreases. We call our classifier k-ish Nearest Neighbors (k-ish NN). For the implementation we introduce double-blinded coin-toss where the bias and output of the toss are encrypted. We use it to approximate the average and variance of the distances from q to S in a scalable circuit whose depth is independent of |S|. We believe these to be of independent interest. We implemented our classifier in an open source library based on HElib and tested it on a breast tumor database. Our classifier has accuracy and running time comparable to current state of the art (non-HE) MPC solution that have better running time but worse communication complexity. It also has communication complexity similar to naive HE implementation that have worse running time.

We present RoboGrammar , a fully automated approach for generating optimized robot structures to traverse given terrains. In this framework, we represent each robot design as a graph, and use a graph grammar to express possible arrangements of physical robot assemblies. Each robot design can then be expressed as a sequence of grammar rules. Using only a small set of rules our grammar can describe hundreds of thousands of possible robot designs. The construction of the grammar limits the design space to designs that can be fabricated. For a given input terrain, the design space is searched to find the top performing robots and their corresponding controllers. We introduce Graph Heuristic Search - a novel method for efficient search of combinatorial design spaces. In Graph Heuristic Search, we explore the design space while simultaneously learning a function that maps incomplete designs (e.g., nodes in the combinatorial search tree) to the best performance values that can be achieved by expanding these incomplete designs. Graph Heuristic Search prioritizes exploration of the most promising branches of the design space. To test our method we optimize robots for a number of challenging and varied terrains. We demonstrate that RoboGrammar can successfully generate nontrivial robots that are optimized for a single terrain or a combination of terrains.

Recent research has highlighted the vulnerabilities of modern machine learning based systems to bias, especially towards segments of society that are under-represented in training data. In this work, we develop a novel, tunable algorithm for mitigating the hidden, and potentially unknown, biases within training data. Our algorithm fuses the original learning task with a variational autoencoder to learn the latent structure within the dataset and then adaptively uses the learned latent distributions to re-weight the importance of certain data points while training. While our method is generalizable across various data modalities and learning tasks, in this work we use our algorithm to address the issue of racial and gender bias in facial detection systems. We evaluate our algorithm on the Pilot Parliaments Benchmark (PPB), a dataset specifically designed to evaluate biases in computer vision systems, and demonstrate increased overall performance as well as decreased categorical bias with our debiasing approach.

A coreset for a set of points is a small subset of weighted points that approximately preserves important properties of the original set. Specifically, if $P$ is a set of points, $Q$ is a set of queries, and $f:P\times Q\to\mathbb{R}$ is a cost function, then a set $S\subseteq P$ with weights $w:P\to[0,\infty)$ is an $ε$-coreset for some parameter $ε>0$ if $\sum_{s\in S}w(s)f(s,q)$ is a $(1+ε)$ multiplicative approximation to $\sum_{p\in P}f(p,q)$ for all $q\in Q$. Coresets are used to solve fundamental problems in machine learning under various big data models of computation. Many of the suggested coresets in the recent decade used, or could have used a general framework for constructing coresets whose size depends quadratically on what is known as total sensitivity $t$. In this paper we improve this bound from $O(t^2)$ to $O(t\log t)$. Thus our results imply more space efficient solutions to a number of problems, including projective clustering, $k$-line clustering, and subspace approximation. Moreover, we generalize the notion of sensitivity sampling for sup-sampling that supports non-multiplicative approximations, negative cost functions and more. The main technical result is a generic reduction to the sample complexity of learning a class of functions with bounded VC dimension. We show that obtaining an $(ν,α)$-sample for this class of functions with appropriate parameters $ν$ and $α$ suffices to achieve space efficient $ε$-coresets. Our result implies more efficient coreset constructions for a number of interesting problems in machine learning; we show applications to $k$-median/$k$-means, $k$-line clustering, $j$-subspace approximation, and the integer $(j,k)$-projective clustering problem.

Deep learning has revolutionized the ability to learn “end-to-end” autonomous vehicle control directly from raw sensory data. While there have been recent extensions to handle forms of navigation instruction, these works are unable to capture the full distribution of possible actions that could be taken and to reason about localization of the robot within the environment. In this paper, we extend end-to-end driving networks with the ability to perform point-to-point navigation as well as probabilistic localization using only noisy GPS data. We define a novel variational network capable of learning from raw camera data of the environment as well as higher level roadmaps to predict (1) a full probability distribution over the possible control commands; and (2) a deterministic control command capable of navigating on the route specified within the map. Additionally, we formulate how our model can be used to localize the robot according to correspondences between the map and the observed visual road topology, inspired by the rough localization that human drivers can perform. We test our algorithms on real-world driving data that the vehicle has never driven through before, and integrate our point-topoint navigation algorithms onboard a full-scale autonomous vehicle for real-time performance. Our localization algorithm is also evaluated over a new set of roads and intersections to demonstrates rough pose localization even in situations without any GPS prior.

Compliant robotic grippers are more robust to uncertainties in grasping and manipulation tasks, especially when paired with tactile and proprioceptive feedback. Although considerable progress has been made towards achieving proprioceptive soft robotic grippers, current efforts require complex driving hardware or fabrication techniques. In this paper, we present a simple scalable soft robotic gripper integrated with high-deformation strain and pressure sensors. The gripper is composed of structurally-compliant handed shearing auxetic structures actuated by electric motors. Coupling deformable sensors with the compliant grippers enables gripper proprioception and object classification. With this sensorized system, we are able to identify objects' size to within 33% of actual radius and sort objects as hard/soft with 78% accuracy.

Seamless communication of desired motions and goals is essential for enabling effective physical human-robot collaboration. In such cases, muscle activity measured via surface electromyography (EMG) can provide insight into a person's intentions while minimally distracting from the task. The presented system uses two muscle signals to create a control framework for team lifting tasks in which a human and robot lift an object together. A continuous setpoint algorithm uses biceps activity to estimate changes in the user's hand height, and also allows the user to explicitly adjust the robot by stiffening or relaxing their arm. In addition to this pipeline, a neural network trained only on previous users classifies biceps and triceps activity to detect up or down gestures on a rolling basis; this enables finer control over the robot and expands the feasible workspace. The resulting system is evaluated by 10 untrained subjects performing a variety of team lifting and assembly tasks with rigid and flexible objects.

Physical simulators have been widely used in robot planning and control. Among them, differentiable simulators are particularly favored, as they can be incorporated into gradient-based optimization algorithms that are efficient in solving inverse problems such as optimal control and motion planning. Therefore, rigid body simulators and recently their differentiable variants are studied extensively. Simulating deformable objects is, however, more challenging compared to rigid body dynamics. The underlying physical laws of deformable objects are more complex, and the resulting systems have orders of magnitude more degrees of freedom and there-fore they are significantly more computationally expensive to simulate. Computing gradients with respect to physical design or controller parameters is typically even more computationally challenging. In this paper, we propose a real-time, differentiable hybrid Lagrangian-Eulerian physical simulator for deformable objects, ChainQueen, based on the Moving Least Squares Material Point Method (MLS-MPM). MLS-MPM can simulate deformable objects with collisions and can be seamlessly incorporated into soft robotic systems. We demonstrate that our simulator achieves high precision in both forward simulation and backward gradient computation. We have successfully employed it in a diverse set of inference, control and co-design tasks for soft robotics.

Soft robotics has yielded numerous examples of soft grippers that utilize compliance to achieve impressive grasping performances with great simplicity, adaptability, and robustness. Designing soft grippers with substantial grasping strength while remaining compliant and gentle is one of the most important challenges in this field. In this paper, we present a light-weight, vacuum-driven soft robotic gripper made of an origami “magic-ball” and a flexible thin membrane. We also describe the design and fabrication method to rapidly manufacture the gripper with different combinations of low-cost materials for diverse applications. Grasping experiments demonstrate that our gripper can lift a large variety of objects, including delicate foods, heavy bottles, and other miscellaneous items. The grasp force on 3D-printed objects is also characterized through mechanical load tests. The results reveal that our soft gripper can produce significant grasp force on various shapes using negative pneumatic pressure (vacuum). This new gripper holds the potential for many practical applications that require safe, strong, and simple grasping.

Autonomous robotic boats are devised to transport people and goods similar to self-driving cars. One of the attractive features specially applied in water environment is to dynamically link and join multiple boats into one unit in order to form floating infrastructure such as bridges, markets or concert stages, as well as autonomously self-detach to perform individual tasks.In this paper we present a novel latching system that enables robotic boats to create dynamic united floating infrastructure while overcoming water disturbances. The proposed latching mechanism is based on the spherical joint (ball and socket) that allows rotation and free movements in two planes at the same time. In this configuration, the latching system is capable to securely and efficiently assemble/disassemble floating structures. The vision-based robot controller guides the self-driving robotic boats to latch with high accuracy in the millimeter range. Moreover, in case the robotic boat fails to latch due to harsh weather, the autonomous latching system is capable to recompute and reposition to latch successfully. We present experimental results from latching and docking in indoor environments. Also, we present results in outdoor environments from latching a couple of robotic boats in open water with calm and turbulent currents.

We propose a feedback control system for a reconfigurable multi-vessel platform. The platform consists of N propeller-driven vessels each of which is capable of latching to another vessel to form a rigid body of connected vessels. The main technical challenges are that i) depending on configurations of the platform the dynamic model would be different, and ii) the number of control variables in control system design increases as does the total number of vessels in the platform. To address these challenges, we develop a coordinated robust control scheme. Through experiments we assess trajectory tracking and disturbance attenuation performance of the control scheme in various configurations of the platform. Experiment results yield that average position and orientation tracking error are approximately 0.09m and 3°, and the maximum tracking error-to-disturbance ratio is 1.12.

In this paper, we examine the problem of navigating cluttered environments without explicit object detection and tracking. We introduce the dynamic risk density to map the congestion density and spatial flow of the environment to a cost function for the agent to determine risk when navigating that environment. We build upon our prior work, wherein the agent maps the density and motion of objects to an occupancy risk, then navigate the environment over a specified risk level set. Here, the agent does not need to identify objects to compute the occupancy risk, and instead computes this cost function using the occupancy density and velocity fields around them. Simulations show how this dynamic risk density encodes movement information for the ego agent and closely models the object-based congestion cost. We implement our dynamic risk density on an autonomous wheelchair and show how it can be used for navigating unstructured, crowded and cluttered environments.

Mobility-on-demand (MoD) systems are revolutionizing urban transit with the introduction of ride-sharing. Such systems have the potential to reduce vehicle congestion and improve accessibility of a city's transportation infrastructure. Recently developed algorithms can compute routes for vehicles in real-time for a city-scale volume of requests while allowing vehicles to carry multiple passengers at the same time. However, these algorithms focus on optimizing the performance for a given fleet of vehicles and do not tell us how many vehicles are needed to service all the requests. In this paper, we present an offline method to optimize the vehicle distributions and fleet sizes on historical demand data for MoD systems that allow passengers to share vehicles. We present an algorithm to determine how many vehicles are needed, where they should be initialized, and how they should be routed to service all the travel demand for a given period of time. Evaluation using 23,529,740 historical taxi requests from one month in Manhattan shows that on average 2864 four passenger vehicles are needed to service all of the taxi demand in a day with an average added travel delay of 2.8 mins.

In this paper, we introduce a novel method to interpret recurrent neural networks (RNNs), particularly long short-term memory networks (LSTMs) at the cellular level. We propose a systematic pipeline for interpreting individual hidden state dynamics within the network using response characterization methods. The ranked contribution of individual cells to the network's output is computed by analyzing a set of interpretable metrics of their decoupled step and sinusoidal responses. As a result, our method is able to uniquely identify neurons with insightful dynamics, quantify relationships between dynamical properties and test accuracy through ablation analysis, and interpret the impact of network capacity on a network's dynamical distribution. Finally, we demonstrate the generalizability and scalability of our method by evaluating a series of different benchmark sequential datasets.

Current methods for autonomous management use strict first-come, first-serve (FCFS) ordering to manage incoming autonomous vehicles at an intersection. In this work, we present a coordination policy that swaps agent ordering to increase the system-wide performance while ensuring that the swaps are socially compliant. By considering an agent's Social Value Orientation (SVO), a social psychology metric for their willingness to help another vehicle, the central coordinator can reduce system delays while ensuring each individual vehicle increases their own utility. The FCFS-SVO algorithm is both computationally tractable and accounts for a variety of real-world agent types, such as human drivers and a variety of social orientations. Simulation results show that average vehicle delays decrease with swapping by enabling cooperation between agents. In addition, we show that the proportion of human drivers, as well as, the distribution of prosocial and egoistic vehicles in the system can have a prominent effect on the performance of the system.

Government data identifies driver behaviour errors as a factor in 94% of car crashes, and autonomous vehicles (AVs), which avoids risky driver behaviours completely, are expected to reduce the number of road crashes significantly. Thus, one of the central focuses of developing AVs is to ensure safety during navigation. However, in reality, AV safety has been far below its expectation, and so far, no government has allowed for complete autonomous driving without human supervision. This paper proposes a dynamic safe path planning algorithm for AVs with Gaussian process regulated risk map. By reasonably assuming that the output of the object detection and tracking module follows a multi-variate Gaussian distribution, we put forward a safe path planning paradigm with Gaussian process regulated risk map, ensuring safety with high confidence. Both simulation results and in-vehicle tests demonstrate the effectiveness of the proposed algorithm.

This paper investigates an online system identification problem of estimating unknown parameters in nonlinear system dynamics in the absence of persistently excitation. To estimate parameters, we develop an algorithm that updates parameter estimates using sensor data and a basis that is built on a finite number of recorded sensor data. Based on our proposed approach we show that the algorithm achieves exponential convergence in both state and parameter estimation errors without the persistent excitation condition. We demonstrate the effectiveness of the proposed approach using both simulations and experiments on a reconfiguration autonomous multi-vessel platform: Simulation results illustrate that the parameter estimated by the developed algorithm converge to their ground truths. Experiment results validate the performance of the developed algorithm in estimating platform's system parameters across different multi-vessel configurations.

This paper presents a Learning-based Nonlinear Model Predictive Control (LB-NMPC) algorithm for reconfigurable autonomous vessels to facilitate high-accurate path tracking. Each vessel is designed to latch to a pre-defined point of another vessel that allows the vessels to form a rigid body. The number of possible configurations of such vessels exponentially grows as the total number of vessels increases, which imposes a technical challenge in modeling and identification. In this work, we propose a framework consisting of a real-time parameter estimator and a feedback control strategy, which is capable of ensuring high-accurate path tracking for any feasible configuration of vessels. Novelty of our method is in that the parameter is estimated on-line and adjusts control parameters (e.g., cost function and dynamic model) simultaneously to improve path-tracking performance. Through experiments on different configurations of connected-vessels, we demonstrate stability of our proposed approach and its effectiveness in high-accuracy in path tracking.

Volume change has become a critical actuation method in robotics. However, the need for fluid flow or thermal processes to generate volume changes limits the durability, speed, and efficiency of these actuators. In this paper, we develop a new electromechanical actuator that volumetrically expands. By combining auxetic materials with a servo, we produce a simple isotropically expanding actuator that can be modularly composed. We discuss the symmetry considerations in selecting an appropriate auxetic framework for our actuator, eventually choosing a double-layered polyhedral auxetic design. Characterization shows that a single actuator can expand in radius to 119% of the original size and generate 90N of force, while maintaining a small package and a speedy expansion / contraction cycle. Finally, we demonstrate the modularity of our actuators by linking three actuators to create a vertical tube-crawling robot. The small package and fast cycle time of our system highlight how viable these electromechanical volumetric actuators can be as an important actuator modality.

Modular reconfigurable robots are touted for their flexibility, as their bodies can assume a wide range of shapes. A particular challenge is to make them move efficiently in 3D without compromising the scalability of the system. This paper proposes decentralized and fully reactive controllers for pose control of 3D modular reconfigurable robots. The robots operate in liquid environments, and move by routing fluid through themselves. Each module uses only two bits of sensory information per face. Additionally, the modules can use up to five bits of information that are exchanged via shared power lines. We prove that robots of convex shape are guaranteed to reach a goal object with a preferred orientation. Using computer simulations of Modular Hydraulic Propulsion robots, all controllers are assessed for different environments, system sizes and noise, and their performances compared against a centralized controller. Given the simplicity of the solutions, modules could be realized at scales below a millimeter-cube, where robots of high spatial resolution could perform accurate movements in 3D liquid environments.

We present a novel autonomous driving system which uses the road contextual information and intentions of other road users for urban driving. Unlike highways, urban environments require the drivers to follow traffic signs and signals while using their best judgment for anomalous situations. In such scenarios, a self-driving car needs to understand and take into account the uncertainties in the environment to plan and decide its action accordingly. Our planner models the intentions of the surrounding vehicles leveraging a neural network, and integrates the road contextual information to reduce its environment uncertainties and also speed up the decision making process. We validate our planner in simulation and in a real urban environment. Our experimental results show that integrating intention inference and road contextual information for prediction, planning and decision making help improve safety and efficiency of our autonomous driving system.

Current perception systems mostly require direct line of sight to anticipate and ultimately prevent potential collisions at intersections with other road users. We present a fully integrated autonomous system capable of detecting shadows or weak illumination changes on the ground caused by a dynamic obstacle in NLoS scenarios. This additional virtual sensor “ShadowCam” extends the signal range utilized so far by computer-vision ADASs. We show that (1) our algorithm maintains the mean classification accuracy of around 70% even when it doesn't rely on infrastructure - such as AprilTags - as an image registration method. We validate (2) in real-world experiments that our autonomous car driving in night time conditions detects a hidden approaching car earlier with our virtual sensor than with the front facing 2-D LiDAR.

This paper presents a decentralized control frame-work for lattice-based Modular Self-Reconfigurable Robots (MSRR) which utilizes a novel magnetic fiducial system to facilitate neighbor identification and to enable algorithms which promise scalable functionality for systems with many modules. In this system individual modules autonomously follow simple behaviors while periodically accepting input from a centralized controller. This system is demonstrated with three initial behaviors: (1) Path following: modules follow a three dimensional path based on magnetic fiducial tags embedded in their neighbors, (2) Line formation: modules transform from a 3D structure into a line following a partially decentralized control algorithm, and (3) Light gradient aggregation: the formation of a group of modules guided by a global stimulus (i.e. visible light). This paper provides details of the neighbor identification system, introduces the three behaviors and presents the results of physical experiments performed with a system of twelve 3D M-Block robotic modules.

The development of effective reduced order models for soft robots is paving the way toward the development of a new generation of model based techniques, which leverage classic rigid robot control. However, several soft robot features differentiate the soft-bodied case from the rigid-bodied one. First, soft robots are built to work in the environment, so the presence of obstacles in their path should always be explicitly accounted by their control systems. Second, due to the complex kinematics, the actuation of soft robots is mapped to the state space nonlinearly resulting in spaces with different sizes. Moreover, soft robots often include internal constraints and thus actuation is typically limited in the range of action and it is often unidirectional. This paper proposes a control pipeline to tackle the challenge of controlling soft robots with internal constraints in environments with obstacles. We show how the constraints on actuation can be propagated and integrated with geometrical constraints, taking into account physical limits imposed by the presence of obstacles. We present a hierarchical control architecture capable of handling these constraints, with which we are able to regulate the position in space of the tip of a soft robot with the discussed characteristics.

In this paper, we focus on a rain removal task from a single image of the urban street scene for autonomous driving in rain. We develop a Convolutional Neural Network which takes a rainy image as input, and directly recovers a clean image in the presence of rain streaks, atmospheric veiling effect (haze, fog, mist) caused by distant rain streak accumulation. We propose a synthetic dataset containing images of urban street scenes with different rain intensities, orientations and haziness levels for training and evaluation. We evaluate our method quantitatively and qualitatively on the synthetic data. Experiments show that our model outperforms state-of-the-art methods. We also test our method qualitatively on the real-world data. Our model is fast and it takes 0.05s for an image of 1024 × 512. Our model can be seamlessly integrated with existing image-based high-level perception algorithms for autonomous driving in rain. Experiment results show that our deraining method improves semantic segmentation and object detection largely for autonomous driving in rain.

Unmanned surface vehicles (USVs) are typically designed for open area marine applications. In this paper, we present a new autonomy system (Roboat) for urban waterways which requires robust localization, perception, planning, and control. A novel localization system, based on the extended Kalman filter (EKF), is proposed for USVs, which utilizes LiDAR, camera, and IMU to provide a decimeter-level precision in dynamic GPS-attenuated urban waterways. Area and shape filters are proposed to crop water reflections and street obstacles from a pointcloud. Euclidean clustering and multi-object contour tracking are then introduced to detect and track the static and moving objects reliably in urban waters. An efficient path planner is tailored to calculate optimal trajectories to avoid these static and dynamic obstacles. Lastly, a nonlinear model predictive control (NMPC) scheme with full state integration is formulated for the four-control-input robot to accurately track the trajectory from the planner in rough water. Extensive experiments show that the robot is able to autonomously navigate in both the indoor waterway and the cluttered outdoor waterway in the presence of static and dynamic obstacles, implying that Roboat could have a great impact on the future of transportation in many coastal and riverside cities.

This paper examines how vehicles can quickly quantify the level of congestion in their environment for planning. We use risk level sets to define a metric of congestion for the vehicles. Using this metric, we can quickly identify distributions of environment and driver features, such as velocities and number of neighbors, based on risk within human driving data sets. We use the NGSIM and highD data sets to study how risk influences behaviors in city and highway driving. From these data sets, we learn common risk thresholds for classifying low, medium, and high-risk situations. Using these thresholds, we develop simulations of an autonomous vehicle driving along a highway, and demonstrate how the chosen risk threshold influences the autonomous vehicle behavior.

Soft robots have continuum solid bodies that can deform in an infinite number of ways. Controlling soft robots is very challenging as there are no closed form solutions. We present a learning-in-the-loop co-optimization algorithm in which a latent state representation is learned as the robot figures out how to solve the task. Our solution marries hybrid particle-grid-based simulation with deep, variational convolutional autoencoder architectures that can capture salient features of robot dynamics with high efficacy. We demonstrate our dynamics-aware feature learning algorithm on both 2D and 3D soft robots, and show that it is more robust and faster converging than the dynamics-oblivious baseline. We validate the behavior of our algorithm with visualizations of the learned representation.

With the fast penetration of commercial Virtual Reality (VR) and Augmented Reality (AR) systems into our daily life, the security issues of those devices have attracted significant interests from both academia and industry. Modern VR/AR systems typically use head-mounted devices (i.e., headsets) to interact with users, and often store private user data, e.g., social network accounts, online transactions or even payment information. This poses significant security threats, since in practice the headset can be potentially obtained and accessed by unauthenticated parties, e.g., identity thieves, and thus cause catastrophic breach. In this paper, we propose a novel GaitLock system, which can reliably authenticate users using their gait signatures. Our system doesn't require extra hardware, e.g., fingerprint sensors or retina scanners, but only uses the on-board inertial measurement units (IMUs) equipped in almost all mainstream VR/AR headsets to authenticate the legitimate users from intruders, by simply asking them to walk a few steps. To achieve that, we propose a new gait recognition model Dynamic-SRC, which combines the strength of Dynamic Time Warping (DTW) and Sparse Representation Classifier (SRC), to extract unique gait patterns from the inertial signals during walking. We implement GaitLock on Google Glass (a typical AR headset), and extensive experiments show that GaitLock outperforms the state-of-the-art systems significantly in recognition accuracy (> 98 percent success in 5 steps), and is able to run in-situ on the resource-constrained VR/AR headsets without incurring high energy cost.

Deep learning has been successfully applied to “end-to-end” learning of the autonomous driving task, where a deep neural network learns to predict steering control commands from camera data input. However, the learned representations do not support higher-level decision making required for autonomous navigation, nor the uncertainty estimates required for parallel autonomy, where vehicle control is shared between human and robot. This paper tackles the problem of learning a representation to predict a continuous control probability distribution, and thus steering control options and bounds for those options, which can be used for autonomous navigation. Each mode of the distribution encodes a possible macro-action that the system could execute at that instant, and the covariances of the modes place bounds on safe steering control values. Our approach has the added advantage of being trained on unlabeled data collected from inexpensive cameras. The deep neural network based algorithm generates a probability distribution over the space of steering angles, from which we leverage Variational Bayesian methods to extract a mixture model and compute the different possible actions in the environment. A bound, which the autonomous vehicle must respect in our parallel autonomy setting, is then computed for each of these actions. We evaluate our approach on a challenging dataset containing a wide variety of driving conditions, and show that our algorithm is capable of parameterizing Gaussian Mixture Models for possible actions, and extract steering bounds with a mean error of only 2 degrees. Additionally, we demonstrate our system working on a full scale autonomous vehicle and evaluate its ability to successful handle various different parallel autonomy situations.

Autonomous driving requires 3D perception of vehicles and other objects in the in environment. Much of the current methods support 2D vehicle detection. This paper proposes a flexible pipeline to adopt any 2D detection network and fuse it with a 3D point cloud to generate 3D information with minimum changes of the 2D detection networks. To identify the 3D box, an effective model fitting algorithm is developed based on generalised car models and score maps. A two-stage convolutional neural network (CNN) is proposed to refine the detected 3D box. This pipeline is tested on the KITTI dataset using two different 2D detection networks. The 3D detection results based on these two networks are similar, demonstrating the flexibility of the proposed pipeline. The results rank second among the 3D detection algorithms, indicating its competencies in 3D detection.

In this paper we consider a fleet of self-driving cars operating in a road network governed by rules of the road, such as the Vienna Convention on Road Traffic, providing rides to customers to serve their demands with desired deadlines. We focus on the associated motion planning problem that trades-off the demands' delays and level of violation of the rules of the road to achieve social optimum among the vehicles. Due to operating in the same environment, the interaction between the cars must be taken into account, and can induce further delays. We propose an integrated route and motion planning approach that achieves scalability with respect to the number of cars by resolving potential collision situations locally within so-called bubble spaces enclosing the conflict. The algorithms leverage the road geometries, and perform joint planning only for lead vehicles in the conflict and use queue scheduling for the remaining cars. Furthermore, a framework for storing previously resolved conflict situations is proposed, which can be use for quick querying of joint motion plans. We show the mobility-on-demand setup and effectiveness of the proposed approach in simulated case studies involving up to 10 self-driving vehicles.

Despite the ubiquity of carpentered items, the customization of carpentered items remains labor intensive. The generation of laymen editable templates for carpentry is difficult. Current design tools rely heavily on CNC fabrication, limiting applicability. We develop a template based system for carpentry and a robotic fabrication system using mobile robots and standard carpentry tools. Our end-to-end design and fabrication tool democratizes design and fabrication of carpentered items. Our method combines expert knowledge for template design, allows laymen users to customize and verify specific designs, and uses robotics system to fabricate parts. We validate our system using multiple designs to make customizable, verifiable templates and fabrication plans and show an end-to-end example that was designed, manufactured, and assembled using our tools.

State-of-the-art autonomous driving systems rely heavily on detailed and highly accurate prior maps. However, outside of small urban areas, it is very challenging to build, store, and transmit detailed maps since the spatial scales are so large. Furthermore, maintaining detailed maps of large rural areas can be impracticable due to the rapid rate at which these environments can change. This is a significant limitation for the widespread applicability of autonomous driving technology, which has the potential for an incredibly positive societal impact. In this paper, we address the problem of autonomous navigation in rural environments through a novel mapless driving framework that combines sparse topological maps for global navigation with a sensor-based perception system for local navigation. First, a local navigation goal within the sensor view of the vehicle is chosen as a waypoint leading towards the global goal. Next, the local perception system generates a feasible trajectory in the vehicle frame to reach the waypoint while abiding by the rules of the road for the segment being traversed. These trajectories are updated to remain in the local frame using the vehicle's odometry and the associated uncertainty based on the least-squares residual and a recursive filtering approach, which allows the vehicle to navigate road networks reliably, and at high speed, without detailed prior maps. We demonstrate the performance of the system on a full-scale autonomous vehicle navigating in a challenging rural environment and benchmark the system on a large amount of collected data.

In this paper, we address the problem of navigating in a cluttered environment by introducing a congestion cost that maps the density and motion of objects to an occupancy risk. We propose that an agent can choose a “risk level set” from this cost function and construct a planning space from this set. In choosing different levels of risk, the agent adjusts its interactions with the other agents. From the assumption that agents are self-preserving, we show that any agent planning within their risk level set will avoid collisions with other agents. We then present an application of planning with risk level sets in the framework of an autonomous vehicle driving along a highway. Using the risk level sets, the agent can determine safe zones when planning a sequence of lane changes. Through simulations in Matlab, we demonstrate how the choice of risk threshold manifests as aggressive or conservative behavior.

When performing multi-robot tasks, sensory feedback is crucial in reducing uncertainty for correct execution. Yet the utilization of sensors should be planned as an integral part of the task planning, taken into account several factors such as the tolerance of different inferred properties of the scene and interaction with different agents. In this paper we handle this complex problem in a principled, yet efficient way. We use surrogate predictors based on open-loop simulation to estimate and bound the probability of success for specific tasks. We reason about such task-specific uncertainty approximants and their effectiveness. We show how they can be incorporated into a multi-robot planner, and demonstrate results with a team of robots performing assembly tasks.

In this paper, we consider the problem of data association in feature cloud matching. While Joint Compatibility (JC) test is a widely adopted technique for searching the global optimal data association, it becomes less restrictive as more features are well matched. The early well-matched features contribute little to total matching cost while the gating threshold increases in the chi-square test, which allows the acceptance of bad feature pairings in the last step. In this paper, we propose the Conditional Compatibility (CC) test, which is not only more restrictive than JC test, but also probabilistically sound. The proposed test of a new feature pairing is based on the conditional probability distribution of feature locations given the early pairings. CC test can be added into any JC test based search algorithm, such as Joint Compatibility Branch and Bound (JCBB), Incremental Posterior Joint Compatibility (IPJC) and FastJCBB, without increasing much computational complexity. The more restrictive criterion of accepting a feature pairing, not only helps to reject bad associations, but also bounds the search space, which substantially improves the search efficiency. The real matching experiments justify that our algorithm produces better feature cloud matching results in a more efficient manner.

We present a real-time vehicle detection and tracking system to accomplish the complex task of driving behavior analysis in urban environments. We propose a robust fusion system that combines a monocular camera and a 2D Lidar. This system takes advantage of three key components: robust vehicle detection using deep learning techniques, high precision range estimation from Lidar, and road context from the prior map knowledge. The camera and Lidar sensor fusion, data association and track management are all performed in the global map coordinate system by taking into account the sensors' characteristics. Lastly, behavior reasoning is performed by examining the tracked vehicle states in the lane coordinate system in which the road context is encoded. We validated our approach by tracking a leading vehicle while it performed usual urban driving behaviors such as lane keeping, stop-and-go at intersections, lane changing, overtaking and turning. The leading vehicle was tracked consistently throughout the 2.3 km route and its behavior was classified reliably.

In this paper, we present the design, modeling, and real-time nonlinear model predictive control (NMPC) of an autonomous robotic boat. The robot is easy to manufacture, highly maneuverable, and capable of accurate trajectory tracking in both indoor and outdoor environments. In particular, a cross type four-thruster configuration is proposed for the robotic boat to produce efficient holonomic motions. The robot prototype is rapidly 3D-printed and then sealed by adhering several layers of fiberglass. To achieve accurate tracking control, we formulate an NMPC strategy for the four-control-input boat with control input constraints, where the nonlinear dynamic model includes a Coriolis and centripetal matrix, the hydrodynamic added mass, and damping. By integrating “GPS” modules and an inertial measurement unit (IMU) into the robot, we demonstrate accurate trajectory tracking of the robotic boat along preplanned paths in both a swimming pool and a natural river. Furthermore, the code generation strategy employed in our paper yields a two order of magnitude improvement in the run time of the NMPC algorithm compared to similar systems. The robot is designed to form the basis for surface swarm robotics testbeds, on which collective algorithms for surface transportation and self-assembly of dynamic floating infrastructures can be assessed.

When driving in urban environments, an autonomous vehicle must account for the interaction with other traffic participants. It must reason about their future behavior, how its actions affect their future behavior, and potentially consider multiple motion hypothesis. In this paper we introduce a method for joint behavior estimation and trajectory planning that models interaction and multi-policy decision-making. The method leverages Partially Observable Markov Decision Processes to estimate the behavior of other traffic participants given the planned trajectory for the ego-vehicle, and Receding-Horizon Control for generating safe trajectories for the ego-vehicle. To achieve safe navigation we introduce chance constraints over multiple motion policies in the receding-horizon planner. These constraints account for uncertainty over the behavior of other traffic participants. The method is capable of running in real-time and we show its performance and good scalability in simulated multi-vehicle intersection scenarios.

Gastric ulcer is a chronic and complex (and often complete) erosion of the stomach wall that happens as a complication of a previous chronic, inflammatory process. It represents a catastrophic situation in which the patient is critical and its conditions need to be treated fast. This study presents a remotely navigatable and deployable ingestible patch and plug for gastric ulcer treatment. The patch/plug structure is made of agarose hydrogel that can change rigidity through hydration and dehydration. When dehydrated, it is rigid and can maintain a folded configuration so it can be ingested as a “pill”. This can be guided to the targeted location by a magnetic field, and be deployed instantly by hydration, namely by supplying water from the mouth. Due to the deployable origami design, it exhibits an expansion of 10 times its initial surface area, making the device suitable for the use of dressing a surface as a patch, and filling a hole as a plug.

This paper introduces a new method for end-to-end training of deep neural networks (DNNs) and evaluates it in the context of autonomous driving. DNN training has been shown to result in high accuracy for perception to action learning given sufficient training data. However, the trained models may fail without warning in situations with insufficient or biased training data. In this paper, we propose and evaluate a novel architecture for self-supervised learning of latent variables to detect the insufficiently trained situations. Our method also addresses training data imbalance, by learning a set of underlying latent variables that characterize the training data and evaluate potential biases. We show how these latent distributions can be leveraged to adapt and accelerate the training pipeline by training on only a fraction of the total dataset. We evaluate our approach on a challenging dataset for driving. The data is collected from a full-scale autonomous vehicle. Our method provides qualitative explanation for the latent variables learned in the model. Finally, we show how our model can be additionally trained as an end-to-end controller, directly outputting a steering control command for an autonomous vehicle.

Recent developments in Mobility-on-Demand (MoD) systems have demonstrated the potential of road vehicles as an efficient mode of urban transportation Newly developed algorithms can compute vehicle routes in real-time for batches of requests and allow for multiple requests to share vehicles. These algorithms have primarily focused on optimally producing vehicle schedules to pick up and drop off requests. The redistribution of idle vehicles to areas of high demand, known as rebalancing, on the contrary has received little attention in the context of ride-sharing. In this paper, we present a method to rebalance idle vehicles in a ride-sharing enabled MoD fleet. This method consists of an algorithm to optimally partition the fleet operating area into rebalancing regions, an algorithm to determine a real-time demand estimate for every region using incoming requests, and an algorithm to optimize the assignment of idle vehicles to these rebalancing regions using an integer linear program. Evaluation with historical taxi data from Manhattan shows that we can service 99.8% of taxi requests in Manhattan using 3000 vehicles with an average waiting time of 57.4 seconds and an average in-car delay of 13.7 seconds. Moreover, we can achieve a higher service rate using 2000 vehicles than prior work achieved with 3000. Furthermore, with a fleet of 3000 vehicles, we reduce the average travel delay by 86%, the average waiting time by 37%, and the amount of ignored requests by 95% compared to earlier work at the expense of an increased distance travelled by the fleet.

This paper introduces a map-based localization method aiming to increase robustness in rainy conditions. This method utilizes two types of features: ground reflectivity features and vertical features extracted from 3D LIDAR scans and builds vehicle pose belief with two filters: a histogram filter and a particle filter. The posterior distributions from the two filters are integrated to estimate vehicle poses. This method exploits advantages of both features and filters, compensating respective weakness to deal with complex urban environments. Testing was performed in the fair and rainy weather. Road test results prove robustness and reliability of the proposed method.

High-end vehicles are already equipped with safety systems, such as assistive braking and automatic lane following, enhancing vehicle safety. Yet, these current solutions can only help in low-complexity driving situations. In this paper, we introduce a parallel autonomy, or shared control, framework that computes safe trajectories for an automated vehicle, based on human inputs. We minimize the deviation from the human inputs while ensuring safety via a set of collision avoidance constraints. Our method achieves safe motion even in complex driving scenarios, such as those commonly encountered in an urban setting. We introduce a receding horizon planner formulated as nonlinear model predictive control (NMPC), which includes the analytic descriptions of road boundaries and the configuration and future uncertainties of other road participants. The NMPC operates over both steering and acceleration simultaneously. We introduce a nonslip model suitable for handling complex environments with dynamic obstacles, and a nonlinear combined slip vehicle model including normal load transfer capable of handling static environments. We validate the proposed approach in two complex driving scenarios. First, in an urban environment that includes a left-turn across traffic and passing on a busy street. And second, under snow conditions on a race track with sharp turns and under complex dynamic constraints. We evaluate the performance of the method with various human driving styles. We consequently observe that the method successfully avoids collisions and generates motions with minimal intervention for parallel autonomy. We note that the method can also be applied to generate safe motion for fully autonomous vehicles.

While computer-aided design is a major part of many modern manufacturing pipelines, the design files typically generated describe raw geometry. Lost in this representation is the procedure by which these designs were generated. In this paper, we present a method for reverse-engineering the process by which 3D models may have been generated, in the language of constructive solid geometry (CSG). Observing that CSG is a formal grammar, we formulate this inverse CSG problem as a program synthesis problem. Our solution is an algorithm that couples geometric processing with state-of-the-art program synthesis techniques. In this scheme, geometric processing is used to convert the mixed discrete and continuous domain of CSG trees to a pure discrete domain where modern program synthesizers excel. We demonstrate the efficiency and scalability of our algorithm on several different examples, including those with over 100 primitive parts. We show that our algorithm is able to find simple programs which are close to the ground truth, and demonstrate our method's applicability in mesh re-editing. Finally, we compare our method to prior state-of-the-art. We demonstrate that our algorithm dominates previous methods in terms of resulting CSG compactness and runtime, and can handle far more complex input meshes than any previous method.

This paper presents an end-to-end approach to automate the design and fabrication process for self-folding origami structures. Self-folding origami structures are robotic sheets composed of rigid tiles and joint actuators. When they are exposed to heat, each joint folds into a preprogrammed angle. Those folding motions transform themselves into a structure, which can be used as body of 3-D origami robots, including walkers, analog circuits, rotational actuators, and microcell grippers. Given a 3-D model, the design algorithm automatically generates a layout printing design of the sheet form of the structure. The geometric information, such as the fold angles and the folding sequences, is embedded in the sheet design. When the sheet is printed and baked in an oven, the sheet self-folds into the given 3-D model. We discuss, first, the design algorithm generating multiple-step self-folding sheet designs, second, verification of the algorithm running in O(n2) time, where n is the number of the vertices, third, implementation of the algorithm, and finally, experimental results, several self-folded 3-D structures with up to 55 faces and two sequential folding steps.

We provide a deterministic data summarization algorithm that approximates the mean $\bar{p}=\frac{1}{n}\sum_{p\in P} p$ of a set $P$ of $n$ vectors in $\mathbb{R}^d$, by a weighted mean $\tilde{p}$ of a subset of $O(1/\epsilon)$ vectors, i.e., independent of both $n$ and $d$. We prove that the squared Euclidean distance between $\bar{p}$ and $\tilde{p}$ is at most $\epsilon$ multiplied by the variance of $P$. We use this algorithm to maintain an approximated sum of vectors from an unbounded stream, using memory that is independent of $d$, and logarithmic in the $n$ vectors seen so far. Our main application is to extract and represent in a compact way friend groups and activity summaries of users from underlying data exchanges. For example, in the case of mobile networks, we can use GPS traces to identify meetings; in the case of social networks, we can use information exchange to identify friend groups. Our algorithm provably identifies the Heavy Hitter entries in a proximity (adjacency) matrix. The Heavy Hitters can be used to extract and represent in a compact way friend groups and activity summaries of users from underlying data exchanges. We evaluate the algorithm on several large data sets.

The multi-robot path planning problem has been extensively studied for the cases of flying and driving vehicles. However, path planning for the case of vehicles that can both fly and drive has not yet been considered. Driving robots, while stable and energy efficient, are limited to mostly flat terrain. Quadcopters, on the other hand, are agile and highly mobile but have low energy efficiency and limited battery life. Combining a quadcopter with a driving mechanism presents a path planning challenge by enabling the selection of paths based off of both time and energy consumption. In this paper, we introduce a framework for multi-robot path planning for a swarm of flying-and-driving vehicles. By putting a lightweight driving platform on a quadcopter, we create a robust vehicle with an energy efficient driving mode and an agile flight mode. We extend two algorithms, priority planning with Safe Interval Path Planning and a multi-commodity network flow ILP, to accommodate multimodal locomotion, and we show that these algorithms can indeed plan collision-free paths for flying-and-driving vehicles on 3D graphs. Finally, we demonstrate that our system is able to plan paths and control the motions of 8 of our vehicles in a miniature town.

We consider the problem of monitoring stochastic, time-varying events occurring at discrete locations. Our problem formulation extends prior work in persistent surveillance by considering the objective of maximizing event detections in unknown, dynamic environments where the rates of events are time-inhomogeneous and may be subject to abrupt changes. We propose a novel monitoring algorithm that effectively strikes a balance between exploration and exploitation as well as a balance between remembering and discarding information to handle temporal variations in unknown environments. We present an analysis proving the long-run average optimality of the policies generated by our algorithm under the assumption that the total temporal variations are sub-linear. We present simulation results demonstrating the effectiveness of our algorithm in several monitoring scenarios inspired by real-world applications, and its robustness to both continuous-random and abrupt changes in the statistics of the observed processes.

We present a distributed control strategy for the aggregation of multiple modular robots into one connected structure optimized for use with 3D modular pivoting cube robots such as the 3D M-Blocks [1]. We use the intensity from a light source as input to a decentralized control algorithm that drives the robots together. We describe the algorithm, give provable guarantees on convergence, and discuss experiments carried out in simulation and with a hardware platform of ten 3D M-Blocks modules. In this paper we contribute provably correct algorithms for the aggregation of generic modular robots; we show how these algorithms can be applied on real hardware by evaluating them on the 3D M-Blocks platform.

Autonomous control of magnetically-actuated miniature robots enables greater versatility and complexity in function but has so far been a challenge to implement. In this paper, we present closed-loop position feedback control of a miniature origami robot utilizing its integrated magnet and an array of Hall effect sensors, enabling the robot's actuation, detection, and locomotion to be initiated from outside its body. An array of 33 Hall effect sensors arranged in repeated triangles cover a range of 60 mm by 75 mm, enabling position detection of the robot with average error of 0.995±0.520 mm. The robot's speed response to applied magnetic field was characterized, and a controller was designed to actuate the robot dependably. We demonstrate autonomous movement of the robot along preplanned paths and the viability of magnetic detection and actuation.

Linear bladed cutting tools, such as jigsaws and reciprocating saws are vital manufacturing tools for humans. They enable people to cut structures that are much larger than themselves. Robots currently lack a generic path planner for linear bladed cutting tools. We developed a model for bladed tools based on Reeds-Shepp cars, and used the model to make a generic path planning algorithm for closed curves. We built an autonomous mobile robot which can implement the algorithm to cut arbitrarily large shapes in a 2D plane. We tested the robots performance and demonstrated the algorithm on several test cases.

Duckietown is an open, inexpensive and flexible platform for autonomy education and research. The platform comprises small autonomous vehicles (“Duckiebots”) built from off-the-shelf components, and cities (“Duckietowns”) complete with roads, signage, traffic lights, obstacles, and citizens (duckies) in need of transportation. The Duckietown platform offers a wide range of functionalities at a low cost. Duckiebots sense the world with only one monocular camera and perform all processing onboard with a Raspberry Pi 2, yet are able to: follow lanes while avoiding obstacles, pedestrians (duckies) and other Duckiebots, localize within a global map, navigate a city, and coordinate with other Duckiebots to avoid collisions. Duckietown is a useful tool since educators and researchers can save money and time by not having to develop all of the necessary supporting infrastructure and capabilities. All materials are available as open source, and the hope is that others in the community will adopt the platform for education and research.

Communication with a robot using brain activity from a human collaborator could provide a direct and fast feedback loop that is easy and natural for the human, thereby enabling a wide variety of intuitive interaction tasks. This paper explores the application of EEG-measured error-related potentials (ErrPs) to closed-loop robotic control. ErrP signals are particularly useful for robotics tasks because they are naturally occurring within the brain in response to an unexpected error. We decode ErrP signals from a human operator in real time to control a Rethink Robotics Baxter robot during a binary object selection task. We also show that utilizing a secondary interactive error-related potential signal generated during this closed-loop robot task can greatly improve classification performance, suggesting new ways in which robots can acquire human feedback. The design and implementation of the complete system is described, and results are presented for realtime closed-loop and open-loop experiments as well as offline analysis of both primary and secondary ErrP signals. These experiments are performed using general population subjects that have not been trained or screened. This work thereby demonstrates the potential for EEG-based feedback methods to facilitate seamless robotic control, and moves closer towards the goal of real-time intuitive interaction.

Current state-of-the-art vehicle safety systems, such as assistive braking or automatic lane following, are still only able to help in relatively simple driving situations. We introduce a Parallel Autonomy shared-control framework that produces safe trajectories based on human inputs even in much more complex driving scenarios, such as those commonly encountered in an urban setting. We minimize the deviation from the human inputs while ensuring safety via a set of collision avoidance constraints. We develop a receding horizon planner formulated as a Non-linear Model Predictive Control (NMPC) including analytic descriptions of road boundaries, and the configurations and future uncertainties of other traffic participants, and directly supplying them to the optimizer without linearization. The NMPC operates over both steering and acceleration simultaneously. Furthermore, the proposed receding horizon planner also applies to fully autonomous vehicles. We validate the proposed approach through simulations in a wide variety of complex driving scenarios such as left-turns across traffic, passing on busy streets, and under dynamic constraints in sharp turns on a race track.

We present parametric trajectory optimization, a method for simultaneously computing physical parameters, actuation requirements, and robot motions for more efficient robot designs. In this scheme, robot dimensions, masses, and other physical parameters are solved for concurrently with traditional motion planning variables, including dynamically consistent robot states, actuation inputs, and contact forces. Our method requires minimal user domain knowledge, requiring only a coarse guess of the target robot configuration sequence and a parameterized robot topology as input. We demonstrate our results on four simulated robots, one of which we physically fabricated in order to demonstrate physical consistency. We demonstrate that by optimizing robot body parameters alongside robot trajectories, motion planning problems which would otherwise be infeasible can be made feasible, and actuation requirements can be significantly reduced.

This paper describes additive self-folding, an origami-inspired rapid fabrication approach for creating actuatable compliant structures. Recent work in 3-D printing and other rapid fabrication processes have mostly focused on rigid objects or objects that can achieve small deformations. In contrast, soft robots often require elastic materials and large amounts of movement. Additive self-folding is a process that involves cutting slices of a 3-D object in a long strip and then pleat folding them into a likeness of the original model. The zigzag pattern for folding enables large bending movements that can be actuated and controlled. Gaps between slices in the folded model can be designed to provide larger deformations or higher shape accuracy. We advance existing planar fabrication and self-folding techniques to automate the fabrication process, enabling highly compliant structures with complex 3-D geometries to be designed and fabricated within a few hours. We describe this process in this paper and provide algorithms for converting 3-D meshes into additive self-folding designs. The designs can be rapidly instrumented for global control using magnetic fields or tendon-driven for local bending. We also describe how the resulting structures can be modeled and their responses to tendon-driven control predicted. We test our design and fabrication methods on three models (a bunny, a tuna fish, and a starfish) and demonstrate the method's potential for actuation by actuating the tuna fish and starfish models using tendons and magnetic control.

This work focuses on integrated routing and motion planning for an autonomous vehicle in a road network. We consider a problem in which customer demands need to be met within desired deadlines, and the rules of the road need to be satisfied. The vehicle might not, however, be able to satisfy these two goals at the same time. We propose a systematic way to compromise between delaying the satisfaction of the given demand and violating the road rules. We utilize scLTL formulas to specify desired behavior and develop a receding horizon approach including a periodically interacting routing algorithm and a RRT*-based motion planner. The proposed solution yields a provably minimum-violation trajectory. An illustrative case study is included.

Context-aware segmentation of laparoscopic and robot assisted surgical video has been shown to improve performance and perioperative workflow efficiency, and can be used for education and time-critical consultation. Modern pressures on productivity preclude manual video analysis, and hospital policies and legacy infrastructure are often prohibitive of recording and storing large amounts of data. In this paper we present a system that automatically generates a video segmentation of laparoscopic and robot-assisted procedures according to their underlying surgical phases using minimal computational resources, and low amounts of training data. Our system uses an SVM and HMM in combination with an augmented feature space that captures the variability of these video streams without requiring analysis of the nonrigid and variable environment. By using the data reduction capabilities of online k-segment coreset algorithms we can efficiently produce results of approximately equal quality, in realtime. We evaluate our system in cross-validation experiments and propose a blueprint for piloting such a system in a real operating room environment with minimal risk factors.

This work introduces a wearable system to provide situational awareness for blind and visually impaired people. The system includes a camera, an embedded computer and a haptic device to provide feedback when an obstacle is detected. The system uses techniques from computer vision and motion planning to (1) identify walkable space; (2) plan step-by-step a safe motion trajectory in the space, and (3) recognize and locate certain types of objects, for example the location of an empty chair. These descriptions are communicated to the person wearing the device through vibrations. We present results from user studies with low- and high-level tasks, including walking through a maze without collisions, locating a chair, and walking through a crowded environment while avoiding people.

microMVP is an affordable, portable, and open source micro-scale mobile robot platform designed for robotics research and education. As a complete and unique multi-vehicle platform enabled by 3D printing and the maker culture, microMVP can be easily reproduced and requires little maintenance: a set of six micro vehicles, each measuring 8 × 5 × 6 cubic centimeters and weighing under 100 grams, and the accompanying tracking platform can be fully assembled in under two hours, all from readily available components. In this paper, we describe microMVP's hardware and software architecture, and the design thoughts that go into the making of the platform. The capabilities of microMVP APIs are then demonstrated with several single- and multi-robot path and motion planning algorithms. microMVP supports all common operation systems.

Mobile sensor networks are a great source of data. By collecting data with mobile sensor nodes from individuals in a user community, e.g. using their smartphones, we can learn global information such as traffic congestion patterns in the city, location of key community facilities, and locations of gathering places. Can we publish and run queries on mobile sensor network databases without disclosing information about individual nodes?

Ride-sharing, or carpooling, systems with autonomous vehicles will provide efficient and reliable urban mobility on demand. In this work we present a method for dynamic vehicle routing that leverages historical data to improve the performance of a network of self-driving taxis. In particular, we describe a constrained optimization method capable of assigning requests to autonomous vehicles in an informed way, to minimize the expected cost of serving both current and future travel requests. We allow several passengers with independent trips to share a vehicle and allow vehicles to pick additional passengers as they progress through their route. Based on historical data, we compute a probability distribution over future demand. Then, samples from the learned probability distribution are incorporated into a decoupled vehicle routing and passenger assignment method to take into account the predicted future demand. This method consists of three steps, namely pruning of feasible trips, assignment of trips to vehicles and rebalancing of idle vehicles. We show the benefits and trade-offs of this predictive approach in an experimental evaluation with over three million rides extracted from a dataset of taxi trips in New York City. Our method produces routes and assignments that, in expectation, reduce the travel and waiting times for passengers, with respect to a purely reactive approach. Besides the mobility on demand application, the method we present is general and could also be applied to other multi-task multi-vehicle assignment and routing problems.

Technologies in autonomous vehicles have seen dramatic advances in recent years; however, it still lacks of robust perception systems for car detection. With the recent development in deep learning research, in this paper, we propose a LIDAR and vision fusion system for car detection through the deep learning framework. It consists of three major parts. The first part generates seed proposals for potential car locations in the image by taking LIDAR point cloud into account. The second part refines the location of the proposal boxes by exploring multi-layer information in the proposal network and the last part carries out the final detection task through a detection network which shares part of the layers with the proposal network. The evaluation shows that the proposed framework is able to generate high quality proposal boxes more efficiently (77.6% average recall) and detect the car at the state of the art accuracy (89.4% average precision). With further optimization of the framework structure, it has great potentials to be implemented onto the autonomous vehicle.

Modern autonomous systems such as driverless vehicles need to safely operate in a wide range of conditions. A potential solution is to employ a hybrid systems approach, where safety is guaranteed in each individual mode within the system. This offsets complexity and responsibility from the individual controllers onto the complexity of determining discrete mode transitions. In this work we propose an efficient framework based on recursive neural networks and coreset data summarization to learn the transitions between an arbitrary number of controller modes that can have arbitrary complexity. Our approach allows us to efficiently gather annotation data from the large-scale datasets that are required to train such hybrid nonlinear systems to be safe under all operating conditions, favoring underexplored parts of the data. We demonstrate the construction of the embedding, and efficient detection of switching points for autonomous and non-autonomous car data. We further show how our approach enables efficient sampling of training data, to further improve either our embedding or the controllers.

Recent advances in autonomous driving have raised the problem of safety to the forefront and incentivized research into establishing safety guarantees. In this paper, we propose a safety verification framework as a safety standard for driving controllers with full or shared autonomy based on compositional and contract-based principles. Our framework enables us to synthesize safety guarantees over entire road networks by first building a library of locally verified models, and then composing local models together to verify the entire network. Composition is achieved using assume-guarantee contracts that are synthesized concurrently during verification. Thus, we can reuse local models within and across networks, add additional models to cover local road geometries without re-verifying the entire library, and perform all computations in a parallel and distributed way, which enables computational tractability. Furthermore, we employ controller contracts such that any controller satisfying them can be certified safe. We demonstrate the practical effectiveness of our framework by certifying controllers over parts of the Manhattan road network.

We present the development of a full-scale “parallel autonomy” research platform including software and hardware. In the parallel autonomy paradigm, the control of the vehicle is shared; the human is still in control of the vehicle, but the autonomy system is always running in the background to prevent accidents. Our holistic approach includes: (1) a drive-by-wire conversion method only based on reverse engineering mounting of relatively inexpensive sensors onto the vehicle implementation of a localization and mapping system, (4) obstacle detection and (5) a shared controller as well as (6) integration with an advanced autonomy simulation system (Drake) for rapid development and testing. The system can operate in three modes: (a) manual driving, (b) full autonomy, where the system is in complete control of the vehicle and (c) parallel autonomy, where the shared controller is implemented. We present results from extensive testing of a full-scale vehicle on closed tracks that demonstrate these capabilities.

We present a new teleoperated micromanipulation system in which all units of the system, wearable user interface devices and a slave micromanipulator, are manufactured by engraving, cutting, and folding two-dimensional materials. The designed manipulation system employs a simple hydraulic mechanism consisting of pairs of syringes that have different diameters, which allows for motion reduction and physical interaction between the master and the slave. As a result, users can precisely manipulate micro-objects without tremor, which was previously difficult with bare hands. This paper presents design considerations and features fabrication methods, performance metrics of this creative manipulation system, and a range of high-level micromanipulation abilities such as pick-and-place, microseparation, and three-dimensional microassembly. Highlighting rapid design and fabrication of a low-cost precision micromanipulation system, this paper proposes new applications of folded machines to wearable robots and microrobotics.

The digitization of practically everything coupled with the mobile Internet, the automation of knowledge work, and advanced robotics promises a future with democratized use of machines and wide-spread use of robots and customization. However, pervasive use of robots remains a hard problem. Where are the gaps that we need to address in order to advance toward a future where robots are common in the world and they help reliably with physical tasks? What is the role of geometric reasoning along this trajectory? In this talk I will discuss challenges toward pervasive use of robots and recent developments in geometric algorithms for customizing robots. I will focus on a suite of gemetric algorithms for automatically designing, fabricating, and tasking robots using a print-and-fold approach. I will also describe how geometric reasoning can play a role in creating robots more capable of reasoning in the world. By enabling on-demand creation of programmable robots, we can begin to imagine a world with one robot for every physical task.

Sensor planning and active sensing, long studied in robotics, adapt sensor parameters to maximize a utility function while constraining resource expenditures. Here we consider information gain as the utility function. While these concepts are often used to reason about 3D sensors, these are usually treated as a predefined, black-box, component. In this paper we show how the same principles can be used as part of the 3D sensor. We describe the relevant generative model for structured-light 3D scanning and show how adaptive pattern selection can maximize information gain in an open-loop-feedback manner. We then demonstrate how different choices of relevant variable sets (corresponding to the subproblems of locatization and mapping) lead to different criteria for pattern selection and can be computed in an online fashion. We show results for both subproblems with several pattern dictionary choices and demonstrate their usefulness for pose estimation and depth acquisition.

This paper presents a distributed method for navigating a team of robots in formation in 2D and 3D environments with static and dynamic obstacles. The robots are assumed to have a reduced communication and visibility radius and share information with their neighbors. Via distributed consensus the robots compute (a) the convex hull of the robot positions and (b) the largest convex region within free space. The robots then compute, via sequential convex programming, the locally optimal parameters for the formation within this convex neighborhood of the robots. Reconfiguration is allowed, when required, by considering a set of target formations. The robots navigate towards the target collision-free formation with individual local planners that account for their dynamics. The approach is efficient and scalable with the number of robots and performs well in simulations with up to sixteen quadrotors.

In this paper we present a visual verification approach for robotic assembly manipulation which enables robots to verify their assembly state. Given shape models of objects and their expected placement configurations, our approach estimates the probability of the success of the assembled state using a depth sensor. The proposed approach takes into account uncertainties in object pose. Probability distributions of depth and surface normal depending on the uncertainties are estimated to classify the assembly state in a Bayesian formulation. The effectiveness of our approach is validated in comparative experiments with other approaches.

This paper introduces a novel technique for fabricating functional robots using 3D printers. Simultaneously depositing photopolymers and a non-curing liquid allows complex, pre-filled fluidic channels to be fabricated. This new printing capability enables complex hydraulically actuated robots and robotic components to be automatically built, with no assembly required. The technique is showcased by printing linear bellows actuators, gear pumps, soft grippers and a hexapod robot, using a commercially-available 3D printer. We detail the steps required to modify the printer and describe the design constraints imposed by this new fabrication approach.

Developing miniature robots that can carry out versatile clinical procedures inside the body under the remote instructions of medical professionals has been a long time challenge. In this paper, we present origami-based robots that can be ingested into the stomach, locomote to a desired location, patch a wound, remove a foreign body, deliver drugs, and biodegrade. We designed and fabricated composite material sheets for a biocompatible and biodegradable robot that can be encapsulated in ice for delivery through the esophagus, embed a drug layer that is passively released to a wounded area, and be remotely controlled to carry out underwater maneuvers specific to the tasks using magnetic fields. The performances of the robots are demonstrated in a simulated physical environment consisting of an esophagus and stomach with properties similar to the biological organs.

The agility and ease of control make a quadrotor aircraft an attractive platform for studying swarm behavior, modeling, and control. The energetics of sustained flight for small aircraft, however, limit typical applications to only a few minutes. Adding payloads - and the mechanisms used to manipulate them - reduces this flight time even further. In this paper we present the flying monkey, a novel robot platform having three main capabilities: walking, grasping, and flight. This new robotic platform merges one of the world's smallest quadrotor aircraft with a lightweight, single-degree-of-freedom walking mechanism and an SMA-actuated gripper to enable all three functions in a 30 g package. The main goal and key contribution of this paper is to design and prototype the flying monkey that has increased mission life and capabilities through the combination of the functionalities of legged and aerial robots.

Undulating structures are one of the most diverse and successful forms of locomotion in nature, both on ground and in water. This paper presents a comparative study for actuation by undulation in water. We focus on actuating a 1DOF systems with several mechanisms. A hydraulic pump attached to a soft body allows for water movement between two inner cavities, ultimately leading to a flexing actuation in a side-to-side manner. The effectiveness of six different, self-contained designs based on centrifugal pump, flexible impeller pump, external gear pump and rotating valves are compared. These hydraulic actuation systems combined with soft test bodies were then measured at a lower and higher oscillation frequency. The deflection characteristics of the soft body, the acoustic noise of the pump and the overall efficiency of the system are recorded. A brushless, centrifugal pump combined with a novel rotating valve performed at both test frequencies as the most efficient pump, producing sufficiently large cyclic body deflections along with the least acoustic noise among all pumps tested. An external gear pump design produced the largest body deflection, but consumes an order of magnitude more power and produced high noise levels. Further refinement remains on determining the suitable oscillation frequencies and inner cavity designs for optimal efficiency and movement.

Impact protection and vibration isolation are an important component of the mobile robot designer's toolkit; however, current damping materials are available only in bulk or molded form, requiring manual fabrication steps and restricting material property control. In this paper we demonstrate a new method for 3D printing viscoelastic materials with specified material properties. This method allows arbitrary net-shape material geometries to be rapidly fabricated and enables continuously varying material properties throughout the finished part. This new ability allows robot designers to tailor the properties of viscoelastic damping materials in order to reduce impact forces and isolate vibrations. We present a case study for using this material to create jumping robots with programmed levels of bouncing.

In this work, we address the problem of robust data association for feature cloud matching. For matching two feature clouds observed at two different poses, we discover that the covariance matrix of the measurement prediction error can be written as the sum of a low rank matrix and a block diagonal matrix, if we assume that the features are observed independently at each pose. This special structure of the covariance matrix allows us to compute its inverse analytically and efficiently. Together with a good bookkeeping strategy, the complexity of the Joint Compatibility (JC) test is reduced to O(1). Contrary to the approximated JC test, ours is both exact and fast. Based on the efficient JC test algorithm and a branch and bound search procedure, we devise an algorithm, called Fast Joint Compatibility Branch and Bound (FastJCBB), to quickly obtain robust data association. The FastJCBB algorithm is essentially modified from the conventional Joint Compatibility Branch and Bound (JCBB) algorithm and both of these algorithms are able to produce exactly the same data association results. However, with the substantial improvement in the efficiency of JC tests, our FastJCBB algorithm is much faster than the conventional JCBB, especially when matching two large feature clouds. It is reported that our FastJCBB algorithm is more than 740 times faster than the conventional JCBB in carrying out one million JC tests when matching two clouds with about 100 features each. Since both FastJCBB and JCBB share the same branch and bound procedure in exploring the interpretation tree, the search complexity remains exponential. Our main contribution is the significant improvement in the efficiency of exploring each node of the interpretation tree.

In this paper we present a practical solution with performance guarantees to the problem of dimensionality reduction for very large scale sparse matrices. We show applications of our approach to computing the Principle Component Analysis (PCA) of any $n\times d$ matrix, using one pass over the stream of its rows. Our solution uses coresets: a scaled subset of the $n$ rows that approximates their sum of squared distances to \emph{every} $k$-dimensional \emph{affine} subspace. An open theoretical problem has been to compute such a coreset that is independent of both $n$ and $d$. An open practical problem has been to compute a non-trivial approximation to the PCA of very large but sparse databases such as the Wikipedia document-term matrix in a reasonable time. We answer both of these questions affirmatively. Our main technical result is a new framework for deterministic coreset constructions based on a reduction to the problem of counting items in a stream.

We propose the correlated orienteering problem (COP) as a novel nonlinear extension to the classic orienteering problem (OP). With the introduction of COP, it becomes possible to model the planning of informative tours for the persistent monitoring of a spatiotemporal field with time-invariant spatial correlations using autonomous mobile robots, in which the robots are range- or time-constrained. Our focus in this paper is QCOP, a quadratic COP instantiation that looks at correlations between neighboring nodes in a node network. The main feature of QCOP is a quadratic utility function capturing the said spatial correlation. We solve QCOP using mixed integer quadratic programming, with the resulting anytime algorithm capable of planning multiple disjoint tours that maximize the quadratic utility. In particular, our algorithm can quickly plan a near-optimal tour over a network with up to 150 nodes. Beside performing extensive simulation studies to verify the algorithm's correctness and characterize its performance, we also successfully applied QCOP to two realistic persistent monitoring tasks: 1) estimation over a synthetic spatiotemporal field and 2) estimating the temperature distribution in the state of Massachusetts in the United States.

This paper presents a formalism that exploits deformability during manipulation of soft objects by robot teams. A hybrid centralized/distributed approach restricts centralized planning to high-level global guidance of the object for consensus. Low-level control is thus delegated to the individual manipulator robots, which retain manipulation and collision avoidance guarantees by passing forces to one another through the object. A distributed receding horizon planner provides local control, formulated as a convex optimization problem in velocity space and incorporating constraints for both collision avoidance and shape maintenance. We demonstrate teams of mobile manipulators autonomously carrying various deformable objects.

This paper addresses the problem of finding robot configurations to grasp assembly parts during a sequence of collaborative assembly operations. We formulate the search for such configurations as a constraint satisfaction problem (CSP). Collision constraints in an operation and transfer constraints between operations determine the sets of feasible robot configurations. We show that solving the connected constraint graph with off-the-shelf CSP algorithms can quickly become infeasible even for a few sequential assembly operations. We present an algorithm which, through the assumption of feasible regrasps, divides the CSP into independent smaller problems that can be solved exponentially faster. The algorithm then uses local search techniques to improve this solution by removing a gradually increasing number of regrasps from the plan. The algorithm enables the user to stop the planner anytime and use the current best plan if the cost of removing regrasps from the plan exceeds the cost of executing those regrasps. We present simulation experiments to compare our algorithm's performance to a naive algorithm which directly solves the connected constraint graph. We also present a real robot system which uses the output of our planner to grasp and bring parts together in assembly configurations.

The goal of this work is to develop a soft robotic manipulation system that is capable of autonomous, dynamic, and safe interactions with humans and its environment. First, we develop a dynamic model for a multi-body fluidic elastomer manipulator that is composed entirely from soft rubber and subject to the self-loading effects of gravity. Then, we present a strategy for independently identifying all unknown components of the system: the soft manipulator, its distributed fluidic elastomer actuators, as well as drive cylinders that supply fluid energy. Next, using this model and trajectory optimization techniques we find locally optimal open-loop policies that allow the system to perform dynamic maneuvers we call grabs. In 37 experimental trials with a physical prototype, we successfully perform a grab 92% of the time. By studying such an extreme example of a soft robot, we can begin to solve hard problems inhibiting the mainstream use of soft machines.

A miniature robotic device that can fold-up on the spot, accomplish tasks, and disappear by degradation into the environment promises a range of medical applications but has so far been a challenge in engineering. This work presents a sheet that can self-fold into a functional 3D robot, actuate immediately for untethered walking and swimming, and subsequently dissolve in liquid. The developed sheet weighs 0.31 g, spans 1.7 cm square in size, features a cubic neodymium magnet, and can be thermally activated to self-fold. Since the robot has asymmetric body balance along the sagittal axis, the robot can walk at a speed of 3.8 body-length/s being remotely controlled by an alternating external magnetic field. We further show that the robot is capable of conducting basic tasks and behaviors, including swimming, delivering/carrying blocks, climbing a slope, and digging. The developed models include an acetone-degradable version, which allows the entire robot's body to vanish in a liquid. We thus experimentally demonstrate the complete life cycle of our robot: self-folding, actuation, and degrading.

This paper presents the mechanical design of a modular robot called the 3D M-Block, a 50mm cubic module capable of both independent and lattice-based locomotion. The first M-Blocks described in [1] could pivot about one axis of rotation only. In contrast, the 3D M-blocks can exert on demand both forward and backward torques about three orthogonal axes, for a total of six directions. The 3D M-Blocks transform these torques into pivoting motions which allow the new 3D M-Blocks to move more freely than their predecessors. Individual modules can employ pivoting motions to independently roll across a wide variety of surfaces as well as to join and move relative to other M-Blocks as part of a larger collective structure. The 3D M-Block maintains the same form factor and magnetic bonding system as the one-dimensional M-Blocks [1], but a new fabrication process supports more efficient and precise production. The 3D M-blocks provide a robust and capable modular self-reconfigurable robotic platform able to support swarm robot applications through individual module capabilities and self-reconfiguring robot applications using connected lattices of modules.

Computational thinking is an important part of a modern education, and robotics provides a powerful tool for teaching programming logic in an interactive and engaging way. The robot garden presented in this paper is a distributed multi-robot system capable of running autonomously or under user control from a simple graphical interface. Over 100 origami flowers are actuated with LEDs and printed pouch motors, and are deployed in a modular array around additional swimming and crawling folded robots. The garden integrates state-of-the-art rapid design and fabrication technologies with distributed systems software techniques to create a scalable swarm in which robots can be controlled individually or as a group. The garden can be used to teach basic algorithmic concepts through its distributed algorithm demonstration capabilities and can teach programming concepts through its education-oriented user interface.

In this paper, we present algorithms for self-reconfiguration of modular robots that move by pivoting. The modules are cubes that can pivot about their edges along the x̂, ŷ, or ẑ axes to move on a 3-dimensional substrate. This is a different model from prior work, which usually considers modules that slide along their faces. We analyze the pivoting cube model and give sufficient conditions for reconfiguration to be feasible. In particular, we show that if an initial configuration does not contain any of three subconfigurations, which we call rules, then it can reconfigure into a line. We provide provably correct algorithms for reconfiguration for both 2-D and 3-D systems, and we verify our algorithms via simulation on randomly generated 2-D and 3-D configurations.

In continuously operating robotic systems, efficient representation of the previously seen camera feed is crucial. Using a highly efficient compression coreset method, we formulate a new method for hierarchical retrieval of frames from large video streams collected online by a moving robot. We demonstrate how to utilize the resulting structure for efficient loop-closure by a novel sampling approach that is adaptive to the structure of the video. The same structure also allows us to create a highly-effective search tool for large-scale videos, which we demonstrate in this paper. We show the efficiency of proposed approaches for retrieval and loop closure on standard datasets, and on a large-scale video from a mobile camera.

Vehicle-based vision algorithms, such as the collision alert systems [4], are able to interpret a scene in real-time and provide drivers with immediate feedback. However, such technologies are based on cameras on the car, limited to the vicinity of the car, severely limiting their potential. They cannot find empty parking slots, bypass traffic jams, or warn about dangers outside the car's immediate surrounding. An intelligent driving system augmented with additional sensors and network inputs may significantly reduce the number of accidents, improve traffic congestion, and care for the safety and quality of people's lives.

This paper presents a method for navigating a team of robots in formation in 2D and 3D environments with static and dynamic obstacles. The method is local and computes the optimal parameters for the formation within a neighborhood of the robots, allowing for reconfigurations, when required, by considering a set of target formations. The method consists of first computing the largest collision-free convex polytope in a neighborhood of the robots, followed by a constrained optimization via sequential convex programming where the optimal parameters for the formation are obtained. The robots navigate towards the target collision-free formation with individual local planners that account for their dynamics. The approach is efficient and scalable with the number of robots and performed well in simulations with a large team of quadrators and in experiments with two mobile manipulators carrying a rigid object.

This work presents a soft hand capable of robustly grasping and identifying objects based on internal state measurements. A highly compliant hand allows for intrinsic robustness to grasping uncertainty, but the specific configuration of the hand and object is not known, leaving undetermined if a grasp was successful in picking up the right object. A soft finger was adapted and combined to form a three finger gripper that can easily be attached to existing robots, for example, to the wrist of the Baxter robot. Resistive bend sensors were added within each finger to provide a configuration estimate sufficient for distinguishing between a set of objects. With one data point from each finger, the object grasped by the gripper can be identified. A clustering algorithm to find the correspondence for each grasped object is presented for both enveloping grasps and pinch grasps. This hand is a first step towards robust proprioceptive soft grasping.

Soft-bodied robots are designed to work in the physical world with a high compliance, while most of them lack in highly dynamic motion. In this paper, we present a soft-bodied jumping robot, which leverages its body's elasticity to achieve a highly dynamic passive bouncing motion after an active jumping motion. This robot has a cubic shape. It is covered by silicone foam, and each of its six faces has an opening to allow for jumping actuation. By winding up and releasing an elastic strip, the robot can jump in two directions at any orientation. We present the design, and fabrication process, and experimental results. By comparing this robot with a rigid version of the robot, we show that this soft-bodied robot can use a single jump to travel longer forward than its rigid counterpart.

We detail the design of autonomous golf cars which were used in public trials in Singapore's Chinese and Japanese Gardens, for the purpose of raising public awareness and gaining user acceptance of autonomous vehicles. The golf cars were designed to be robust, reliable, and safe, while operating under prolonged durations. Considerations that went in to the overall system design included the fact that any member of the public had to not only be able to easily use the system, but to also not have the option to use the system in an unintended manner. This paper details the hardware and software components of the golf cars with these considerations, and also how the booking system and mission planner facilitated users to book for a golf car from any of ten stations within the gardens. We show that the vehicles performed robustly throughout the prolonged operations with a small localization variance, and that users were very receptive from the user survey results.

In a mixed environment of autonomous driverless vehicles and human driven vehicles operating on the same road, identifying intentions of human drivers and interacting with them in a compliant and responsible manner becomes a challenging problem for the driverless vehicles. In this paper, the problem of vehicle interaction at an intersection merging scenario is formulated as an Intention-Aware motion planning problem using the tools from Mixed Observability Markov Decision Process (MOMDP). We utilize the tools from recent intention aware planning framework to demonstrate a merging behavior in the presence of human drivers by trying to infer and act according to the intentions of the human drivers. A driver behavior model for T-junction intersections is developed in order to calculate the probabilistic state transition functions of the MOMDP model. With proposed solution, it is demonstrated that using intention aware planning improves performance in comparison to present time to merge approach by lowering accident probability and intersection navigation duration. The proposed method is tested on a real autonomous vehicle (AV) in the presence of human driven vehicles to validate our approach.

We study the problem in which a mobile sensing robot is tasked to travel among and gather intelligence at a set of spatially distributed points-of-interest (POIs). The quality of the information collected at a POI is characterized by some sensory (reward) function of time. With limited fuel, the robot must balance between spending time traveling to more POIs and performing time-consuming sensing activities at POIs to maximize the overall reward. In a dual formulation, the robot is required to acquire a minimum amount of reward with the least amount of time. We propose an anytime planning algorithm for solving these two NP-hard problems to arbitrary precision for arbitrary reward functions. The algorithm is effective on large instances with tens to hundreds of POIs, as demonstrated with an extensive set of computational experiments. Besides mobile sensor scheduling, our algorithm also applies to automation scenarios such as intelligent and optimal itinerary planning.

We propose soft planar actuators enhanced by free-form fabrication that are suitable for making everyday objects move. The actuator consists of one or more inflatable pouches with an adhesive back. We have developed a machine for the fabrication of free-from pouches; squares, circles and ribbons are all possible. The deformation of the pouches can provide linear, rotational, and more complicated motion corresponding to the pouch's geometry. We also provide a both manual and programmable control system. In a user study, we organized a hands-on workshop of actuated origami for children. The results show that the combination of the actuator and classic materials can enhance rapid prototyping of animated objects.

This study demonstrates a new approach to autonomous folding for the body of a 3D robot from a 2D sheet, using heat. We approach this challenge by folding a 0.27-mm sheetlike material into a structure. We utilize the thermal deformation of a contractive sheet sandwiched by rigid structural layers. During this baking process, the heat applied on the entire sheet induces contraction of the contracting layer and thus forms an instructed bend in the sheet. To attain the targeted folding angles, the V-fold spans method is used. The targeted angle θout can be kinematically encoded into crease geometry. The realization of this angle in the folded structure can be approximately controlled by a contraction angle θin. The process is non-reversible, is reliable, and is relatively fast. Our method can be applied simultaneously to all the folds in multi-crease origami structures. We demonstrate the use of this method to create a lightweight mobile robot.

In this paper, we present a multivehicle cooperative driving system architecture using cooperative perception along with experimental validation. For this goal, we first propose a multimodal cooperative perception system that provides see-through, lifted-seat, satellite and all-around views to drivers. Using the extended range information from the system, we then realize cooperative driving by a see-through forward collision warning, overtaking/lane-changing assistance, and automated hidden obstacle avoidance. We demonstrate the capabilities and features of our system through real-world experiments using four vehicles on the road.

Long-term sensor network deployments demand careful power management. While managing power requires understanding the amount of energy harvestable from the local environment, current solar prediction methods rely only on recent local history, which makes them susceptible to high variability. In this article, we present a model and algorithms for distributed solar current prediction based on multiple linear regression to predict future solar current based on local, in situ climatic and solar measurements. These algorithms leverage spatial information from neighbors and adapt to the changing local conditions not captured by global climatic information. We implement these algorithms on our Fleck platform and run a 7-week-long experiment validating our work. In analyzing our results from this experiment, we determined that computing our model requires an increased energy expenditure of 4.5mJ over simpler models (on the order of 10 -7 % of the harvested energy) to gain a prediction improvement of 39.7%.

This article describes iDiary, a system that takes as input GPS data streams generated by users’ phones and turns them into textual descriptions of the trajectories. The system features a user interface similar to Google Search that allows users to type text queries on their activities (e.g., “Where did I buy books?”) and receive textual answers based on their GPS signals. iDiary uses novel algorithms for semantic compression and trajectory clustering of massive GPS signals in parallel to compute the critical locations of a user. We encode these problems as follows. The k-segment mean is a k -piecewise linear function that minimizes the regression distance to the signal. The ( k,m )- segment mean has an additional constraint that the projection of the k segments on R d consists of only m ≤ k segments. A coreset for this problem is a smart compression of the input signal that allows computation of a (1+ε)-approximation to its k -segment or ( k,m )-segment mean in O ( n log n ) time for arbitrary constants ε, k , and m . We use coresets to obtain a parallel algorithm that scans the signal in one pass, using space and update time per point that is polynomial in log n . Using an external database, we then map these locations to textual descriptions and activities so that we can apply text mining techniques on the resulting data (e.g., LSA or transportation mode recognition). We provide experimental results for both the system and algorithms and compare them to existing commercial and academic state of the art. This is the first GPS system that enables text-searchable activities from GPS data.

This paper introduces a new mobile sensor scheduling problem involving a single robot tasked to monitor several events of interest that are occurring at different locations (stations). Of particular interest is the monitoring of transient events of a stochastic nature, with applications ranging from natural phenomena (e.g., monitoring abnormal seismic activity around a volcano using a ground robot) to urban activities (e.g., monitoring early formations of traffic congestion using an aerial robot). Motivated by examples like these, this paper focuses on problems in which the precise occurrence times of the events are unknown apriori, but statistics for their interarrival times are available. In monitoring such events, the robot seeks to: (1) maximize the number of events observed and (2) minimize the delay between two consecutive observations of events occurring at the same location. This paper considers the case when a robot is tasked with optimizing the event observations in a balanced manner, following a cyclic patrolling route. To tackle this problem, first, assuming that the cyclic ordering of stations is known, we prove the existence and uniqueness of the optimal solution and show that the solution has desirable convergence rate and robustness. Our constructive proof also yields an efficient algorithm for computing the unique optimal solution with O(n) time complexity, in which n is the number of stations, with O(log n) time complexity for incrementally adding or removing stations. Except for the algorithm, our analysis remains valid when the cyclic order is unknown. We then provide a polynomial-time approximation scheme that computes for any ε > 0 a (1 + ε)-optimal solution for this more general, NP-hard problem.

In this paper, we investigate the behavior of agents in mean field games where each agent evolves according to a dynamic equation containing the input average and seeks to minimize its long time average (LTA) cost encompassing a population state average (PSA), which is also known as the mean field term. Due to the informational burden resulting from the PSA coupling to the states of all agents, our idea is to find a deterministic function φ to approximate it. It is shown that φ is an approximation of the PSA as the population size N goes to infinity. The resulting decentralized mean field control laws lead the system to achieve mean-consensus asymptotically as time goes to infinity. Furthermore, the optimal controls generate an almost sure asymptotic Nash equilibrium, which implies that the LTA cost of each agent can reach its minimal value as the number of agents increases to infinity. Finally, we consider the socially optimal case where the basic objective is to minimize the social cost as the sum of the individual LTA cost containing the PSA. In this case, it is shown that the decentralized mean field social control strategies are the same as the mean field Nash controls for infinite population systems.

This paper presents an end-to-end approach for creating 3D shapes by self-folding planar sheets activated by uniform heating. These shapes can be used as the mechanical bodies of robots. The input to this process is a 3D geometry (e.g. an OBJ file). The output is a physical object with the specified geometry. We describe an algorithm pipeline that (1) identifies the overall geometry of the input, (2) computes a crease pattern that causes the sheet to self-fold into the desired 3D geometry when activated by uniform heating, (3) automatically generates the design of a 2D sheet with the desired pattern and (4) automatically generates the design files required to fabricate the 2D structure. We demonstrate these algorithms by applying them to complex 3D shapes. We demonstrate the fabrication of a self-folding object with over 50 faces from automatically generated design files.

We present a novel end-to-end solution for distributed multirobot coordination that translates multitouch gestures into low-level control inputs for teams of robots. Highlighting the need for a holistic solution to the problem of scalable human control of multirobot teams, we present a novel control algorithm with provable guarantees on the robots' motion that lends itself well to input from modern tablet and smartphone interfaces. Concretely, we develop an iOS application in which the user is presented with a team of robots and a bounding box (prism). The user carefully translates and scales the prism in a virtual environment; these prism coordinates are wirelessly transferred to our server and then received as input to distributed onboard robot controllers. We develop a novel distributed multirobot control policy which provides guarantees on convergence to a goal with distance bounded linearly in the number of robots, and avoids interrobot collisions. This approach allows the human user to solve the cognitive tasks such as path planning, while leaving precise motion to the robots. Our system was tested in simulation and experiments, demonstrating its utility and effectiveness.

In this paper we describe the design, fabrication, control, and experimental validation of a soft and highly compliant 2D manipulator. The arm consists of several body segments actuated using bi-directional fluidic elastomer actuators and is fabricated using a novel composite molding process. We use a cascaded PI and PID computation and novel fluidic drive cylinders to provide closed-loop control of curvature for each soft and highly compliant body segment. Furthermore, we develop algorithms to compute the arm's forward and inverse kinematics in a manner consistent with piece-wise constant curvature continuum manipulators. These computation and control systems enable this highly compliant robot to autonomously follow trajectories. Experimental results with a robot consisting of six segments show that controlled movement of a soft and highly compliant manipulator is feasible.

This work presents a script-based development environment aimed at allowing users to easily design and create mechanical bodies for folded plastic robots. The origami-inspired fabrication process is inexpensive and widely accessible, and the tools developed in this work allow for open source design sharing and modular reuse. Designs are generated by recursively combining mechanical components — from primitive building blocks, through mechanisms and assemblies, to full robots — in a flexible yet well-defined manner. This process was used to design robotic elements of increasing complexity up to a multi-degree-of-freedom compliant manipulator arm, demonstrating the power of this system. The developed system is extensible, opening avenues for further research ultimately leading to the development of a complete robot compiler.

This paper presents a methodology and validation of print-and-self-fold electric devices. For printing functional structures for robotic use, we realize electric circuitry based on metallic polyester film (MPF). By exploiting the unique material properties of MPF, we developed fundamental electric devices, namely a resistor, capacitor, and inductor. The developed polyvinyl chloride laminated MPF sheet shows reliable self-folding processes under a heat application, and it configures 3D electric devices. Due to the pre-resolved kinematic design, these devices feature elasticity, making them suitable as sensors and actuators in soft circuits. Here we testify to a self-assembled variable resistor and capacitive strain sensor. An actuation mechanism consisting of a folded contractible coil is also considered and shown. Finally, an RLC circuit obtained from the integration of all the developed devices is demonstrated, in which the coil based actuator is controlled by reading a variable capacitive strain sensor.

We propose a new family of fluidic soft actuators called Pouch Motors. The pouch motors are developed to create printable actuators for enhancing mass-fabrication of robots from sheet materials using easily accessible tools. The pouch motor consists of one or more gas-tight bladders (called pouches) fabricated by heat bonding. We developed two types of actuators from inflatable pouches: the linear pouch motor and the rotational pouch motor. Our theoretical analysis predicts the static force-length and moment-angle relationships of these actuators under pressure control. We compare the theoretical bounds with actual results achieved using several fabricated devices. We developed a fabrication process of pouch motors using a heat stamping technique that allows mass-manufacturing. We also demonstrate three robot bodies with embedded pouch motors: a parallel gripper, a robotic arm with antagonistic actuation, and legged walking robot with a self-contained miniature pneumatic system.

Given an image stream, our on-line algorithm will select the semantically-important images that summarize the visual experience of a mobile robot. Our approach consists of data pre-clustering using coresets followed by a graph based incremental clustering procedure using a topic based image representation. A coreset for an image stream is a set of representative images that semantically compresses the data corpus, in the sense that every frame has a similar representative image in the coreset. We prove that our algorithm efficiently computes the smallest possible coreset under natural well-defined similarity metric and up to provably small approximation factor. The output visual summary is computed via a hierarchical tree of coresets for different parts of the image stream. This allows multi-resolution summarization (or a video summary of specified duration) in the batch setting and a memory-efficient incremental summary for the streaming case.

This paper proposes a semantic mapping method based on pedestrian activity in the urban road environment. Pedestrian activity patterns are learned from pedestrian tracks collected by a mobile platform. With the learned knowledge of pedestrian activity, semantic mapping is performed using Bayesian classification techniques. The proposed method is tested in real experiments, and shows promising results in recognizing four activity-related semantic properties of the urban road environment: pedestrian path, entrance/exit, pedestrian crossing and sidewalk.

This paper is concerned with a novel mobile sensor scheduling problem, involving a single robot tasked with monitoring several events of interest that occur at different locations. Of particular interest is the monitoring of events that can not be easily forecast. Prominent examples range from natural phenomena (e.g., monitoring abnormal seismic activity around a volcano using a ground robot) to urban activities (e.g., monitoring early formations of traffic congestion in the Boston area using an aerial robot). Motivated by these examples, this paper focuses on problems where the precise occurrence time of the events is not known a priori, but some statistics for their inter-arrival times are available from past observations. The robot's task is to monitor the events to optimize the following two objectives: (i) maximize the number of events observed and (ii) minimize the delay between two consecutive observations of events occurring at the same location. Provided with only one robot, it is crucial to optimize these objectives in a balanced way, so that they are optimized at each station simultaneously. Our main theoretical result is that this complex mobile sensor scheduling problem can be reduced to a quasi-convex program, which can be solved in polynomial time. In other words, a globally optimal solution can be computed in time that is polynomial in the number of locations. We also provide computational experiments that validate our theoretical results.

Soft continuum manipulators have the advantage of being more compliant and having more degrees of freedom than rigid redundant manipulators. This attribute should allow soft manipulators to autonomously execute highly dexterous tasks. However, current approaches to motion planning, inverse kinematics, and even design limit the capacity of soft manipulators to take full advantage of their inherent compliance. We provide a computational approach to whole arm planning for a soft planar manipulator that advances the arm's end effector pose in task space while simultaneously considering the arm's entire envelope in proximity to a confined environment. The algorithm solves a series of constrained optimization problems to determine locally optimal inverse kinematics. Due to inherent limitations in modeling the kinematics of a highly compliant soft robot and the local optimality of the planner's solutions, we also rely on the increased softness of our newly designed manipulator to accomplish the whole arm task, namely the arm's ability to harmlessly collide with the environment. We detail the design and fabrication of the new modular manipulator as well as the planner's central algorithm. We experimentally validate our approach by showing that the robotic system is capable of autonomously advancing the soft arm through a pipe-like environment in order to reach distinct goal states.

Designing and fabricating new robotic systems is typically limited to experts, requiring engineering background, expensive tools, and considerable time. In contrast, to facilitate everyday users developing custom robots for personal use, this work presents a new system to easily create printable foldable robots from high-level structural specifications. A user merely needs to select electromechanical components from a library of basic building blocks and pre-designed mechanisms, then connect them to define custom robot assemblies. The system then generates complete mechanical drawings suitable for fabrication, instructions for the assembly of electronics, and software to control and drive the final robot. Several robots designed in this manner demonstrate the ability and versatility of this process.

We propose a novel non-linear extension to the Orienteering Problem (OP), called the Correlated Orienteering Problem (COP). We use COP to plan informative tours (cyclic paths) for persistent monitoring of an environment with spatial correlations, where the tours are constrained to a fixed length or time budget. The main feature of COP is a quadratic utility function that captures spatial correlations among points of interest that are close to each other. COP may be solved using mixed integer quadratic programming (MIQP) that can plan multiple disjoint tours that maximize the quadratic utility function. We perform extensive characterization of our method to verify its correctness, as well as its applicability to the estimation of a realistic, time-varying, and spatially correlated scalar field.

Recent years have seen the advent of new RF-localization systems that demonstrate tens of centimeters of accuracy. However, such systems require either deployment of new infrastructure, or extensive fingerprinting of the environment through training or crowdsourcing, impeding their wide-scale adoption.

This paper proposes a method for multi-agent path planning on a road network in the presence of congestion. We suggest a distributed method to find paths for multiple agents by introducing a probabilistic path choice achieving global goals such as the user equilibrium or the social optimum. This approach, which shows that the global goals can be achieved by local processing using only local information, can be parallelized and sped-up using massive parallel processing. The probabilistic assignment reliably copes with the case of random choices of unidentified agents or random route changes of agents who ignore our path guidance. We provide the analytical result on convergence and running time. We demonstrate and evaluate our algorithm by an implementation using asynchronous computation on multi-core computers.

This paper presents a decentralized algorithm for the coordinated assembly of 3-D objects that consist of multiple types of parts, using a networked team of robots. We describe the algorithm and analyze its convergence and adaptation properties. We partition construction in two tasks: tool delivery and assembly. Each task is performed by a networked team of specialized robots. We analyze the performance of the algorithms using the balls into bins problem and show their adaptation to failure of robots, dynamic constraints, multiple types of elements, and reconfiguration. We instantiate the algorithm to building truss-like objects using rods and connectors. We implement the algorithm in simulation and show results to construct 2-D and 3-D parts. Finally, we describe hardware implementation of the algorithms, where mobile manipulators assemble smart parts with IR beacons.

We consider the problem of automatic control strategy synthesis, for discrete models of robotic systems, to fulfill a task that requires reaching a goal state while obeying a given set of safety rules. In this paper, we focus on the case when the said task is not feasible without temporarily violating some of the rules. We propose an algorithm that {synthesizes} a motion which violates only lowest priority rules for the shortest amount of time. Although the proposed algorithm can be applied in a variety of control problems, throughout the paper, we motivate this problem with an autonomous car navigating in an urban environment while abiding by the rules of the road, such as "always stay in the right lane" and "do not enter the sidewalk." We evaluate the algorithm on a case study with several illustrative scenarios.

This paper presents a precise localization algorithm for vehicles in 3D urban environment with only one 2D LIDAR and odometry information. A novel idea of synthetic 2D LIDAR is proposed to solve the localization problem on a virtual 2D plane. A Monte Carlo Localization scheme is adopted for vehicle position estimation, based on synthetic LIDAR measurements and odometry information. The accuracy and robustness of the proposed algorithm are demonstrated by performing real time localization in a 1.5 km driving test around the NUS campus area.

We present a real-time video delivery solution based on free-space optical communication for underwater applications. This solution comprises of AquaOptical II, a high-bandwidth wireless optical communication device, and a two-layer digital encoding scheme designed for error-resistant communication of high resolution images. Our system can transmit digital video reliably through a unidirectional underwater channel, with minimal infrastructural overhead. We present empirical evaluation of this system's performance for various system configurations, and demonstrate that it can deliver high quality video at up to 15 Hz, with near-negligible communication latencies of 100 ms. We further characterize the corresponding end-to-end latencies, i.e. from time of image acquisition until time of display, and reveal optimized results of under 200 ms, which facilitates a wide range of applications such as underwater robot tele-operation and interactive remote seabed monitoring.

We present an approach to position k servers (e.g. mobile robots) to provide a service to n independently moving clients; for example, in mobile ad-hoc networking applications where inter-agent distances need to be minimized, connectivity constraints exist between servers, and no a priori knowledge of the clients' motion can be assumed. Our primary contribution is an algorithm to compute and maintain a small representative set, called a kinematic coreset, of the n moving clients.We prove that, in any given moment, the maximum distance between the clients and any set of k servers is approximated by the coreset up to a factor of (1 ± ε), where ε > 0 is an arbitrarily small constant. We prove that both the size of our coreset and its update time is polynomial in k log(n)/ε. Although our optimization problem is NP-hard (i.e., takes time exponential in the number of servers to solve), solving it on the small coreset instead of the original clients results in a tractable controller. The approach is validated in a small scale hardware experiment using robot servers and human clients, and in a large scale numerical simulation using thousands of clients.

Printing and folding are fast and inexpensive methods for prototyping complex machines. Self-assembly of the folding step would expand the possibilities of this method to include applications where external manipulation is costly, such as micro-assembly, mass production, and space applications. This paper presents a method for self-folding of printed robots from two-dimensional materials based on shape memory polymers actuated by joule heating using embedded circuits. This method was shown to be capable of sequential folding, angle-controlled folds, slot-and-tab assembly, and mountain and valley folds. An inchworm robot was designed to demonstrate the merits of this technique. Upon the application of sufficient current, the robot was able to fold into its functional form with fold angle deviations within six degrees. This printed robot demonstrated locomotion at a speed of two millimeters per second.

We present an automated assembly system that directs the actions of a team of heterogeneous robots in the completion of an assembly task. From an initial user-supplied geometric specification, the system applies reasoning about the geometry of individual parts in order to deduce how they fit together. The task is then automatically transformed to a symbolic description of the assembly-a sort of blueprint. A symbolic planner generates an assembly sequence that can be executed by a team of collaborating robots. Each robot fulfills one of two roles: parts delivery or parts assembly. The latter are equipped with specialized tools to aid in the assembly process. Additionally, the robots engage in coordinated co-manipulation of large, heavy assemblies. We provide details of an example furniture kit assembled by the system.

We provide a set of sufficient conditions to establish the completeness of an ensemble of motion planners-that is, a set of loosely-coupled motion planners that produce a unified result. The planners are assumed to divide the total planning problem across some parameter space(s), such as task space, state space, action space, or time. Robotic applications have employed ensembles of planners for decades, although the concept has not been formally unified or analyzed until now. We focus on applications in multi-robot navigation and collision avoidance. We show that individual resolutionor probabilistically-complete planners that meet certain communication criteria constitute a (respectively, resolution- or probabilistically-) complete ensemble of planners. This ensemble of planners, in turn, guarantees that the robots are free of deadlock, livelock, and starvation.

We consider the problem of task assignment for a multi-robot system where each robot must attend to one or more queues of tasks. We assume that individual robots have no knowledge of tasks in the environment that are not in their queue. Robots in communication with each other may share information about active tasks and exchange queues to achieve lower cost for the system. We show that allowing this kind of task switching causes tasks to be completed more efficiently. In addition, we present conditions under which queues can be guaranteed to make progress, and we support these claims with simulation and experimental results. This work has potential applications in manufacturing, environmental exploration, and pickup-delivery tasks.

We consider automatic synthesis of control policies for non-independent, heterogeneous multi-agent systems with the objective of maximizing the probability of satisfying a given specification. The specification is expressed as a formula in linear temporal logic. The agents are modeled by Markov decision processes with a common set of actions. These actions, however, may or may not affect the behaviors of all the agents. To alleviate the well-known state explosion problem, an incremental approach is proposed where only a small subset of agents is incorporated in the synthesis procedure initially and more agents are successively added until the limitations on computational resources are reached. The proposed algorithm runs in an anytime fashion, where the probability of satisfying the specification increases as the algorithm progresses.

In this paper, we report a fully automated detailed mapping of a challenging urban environment using single LIDAR. To improve scan matching, extended correlative scan matcher is proposed. Also, a Monte Carlo loop closure detection is implemented to perform place recognition efficiently. Automatic recovery of the pose graph map in the presence of false place recognition is realized through a heuristic based loop closure rejection. This mapping framework is evaluated through experiments on the real world dataset obtained from NUS campus environment.

In this paper we examine the correlation between the information content and the spatial realization of range measurements taken by a mapping robot. To do so, we consider the task of constructing an occupancy grid map with a binary Bayesian filter. Using a narrow beam-based sensor model (versus an additive white Gaussian noise model), we prove that any controller tasked to maximize a mutual information reward function is eventually attracted to unexplored space. This intuitive behavior is derived solely from the geometric dependencies of the occupancy grid mapping algorithm and the monotonie properties of mutual information. Since it is a function of both the robot's position and the uncertainty of the surrounding cells, mutual information encodes geometric relationships that are fundamental to robot control, thus yielding geometrically relevant reward surfaces on which the robot can navigate. Lastly, we present the results of two experiments employing an omnidirectional ground robot equipped with a laser rangefinder.

We present Pose and Paste (P&P) - an intuitive interface designed to facilitate interaction between a single user and a number of robots equipped with cameras. With this interface, a user wearing a head-mounted display is able to cycle through the real-time video streams originating from the robots' cameras. The user is also able to select a robot and remotely position it by simply walking or turning his/her head, i.e., control the robot's motion in a master/slave-type fashion. We report the results of an initial hardware experiment where a user located in the USA is tasked to position two quadrotor robots within a motion capture laboratory located in Germany. These results suggest that P&P is a feasible approach to remotely inspect disaster affected sites. Lastly, we conduct a user study to compare P&P with a baseline interface composed of a traditional computer monitor and a video game controller. The quantitative results and qualitative discussions resulting from this user study highlight how such multi-robot interfaces can be further improved.

In this study, we demonstrate a new approach to autonomous folding for the body of a 3D robot from a 2D sheet using heat. We approach this challenge by folding a 0.27 mm sheet-like material into a structure. We utilize the thermal deformation of a contractive sheet sandwiched by rigid structural layers. During this “baking” process, the heat applied on the entire sheet induces contraction of the contracting layer and, thus, forms an instructed bend in the sheet. To attain the targeted folding angles, the V-fold Spans method is used. The targeted angle θoutcan be kinematically encoded into crease geometry. The realization of this angle in the folded structure can be approximately controlled by a contraction angle θin. The process is non-reversible, is reliable, and it is relatively fast. Our method can be applied simultaneously to all the folds in multi-creased origami structures. We demonstrate the use of this method to create a light-weight mobile robot.

In this paper, we describe a novel self-assembling, self-reconfiguring cubic robot that uses pivoting motions to change its intended geometry. Each individual module can pivot to move linearly on a substrate of stationary modules. The modules can use the same operation to perform convex and concave transitions to change planes. Each module can also move independently to traverse planar unstructured environments. The modules achieve these movements by quickly transferring angular momentum accumulated in a self-contained flywheel to the body of the robot. The system provides a simplified realization of the modular actions required by the sliding cube model using pivoting. We describe the principles, the unit-module hardware, and extensive experiments with a system of eight modules.

Our prior work presented a system for decentralized robotic assembly[1]: given a team of robots, a cache of components, and a desired structure specified as a blueprint, the algorithm computes a sequence of part deliveries and assembly steps to achieve the desired structure, while considering physical dependencies and reachability constraints for the goal structure. In this paper we introduce a new algorithm that extends our prior result to incorporate the duration of each assembly operation. We also extend the algorithm to be adaptive to the availability of parts. When a part is not available, the assembly sequence is recomputed. The algorithms are provably convergent and their execution does not depend on the size of the robot team. We implement the algorithms using a team of four youBot robots that can (1) locate and identify parts; (2) use communication to achieve coordinated hand-off of parts; and (3) create complex log-cabin style structures.

This paper presents the design, fabrication and operation of a hexapod fabricated using a combination of printing and folding flat sheets of polyester. The polyester sheets are cut and engraved with crease patterns, which are then manually folded to create 3D functional modules, inspired by the Japanese art of Origami. These modules, when connected, form a hexapod with two degrees of freedom per leg. All custom mechanical parts are manufactured in a planar fashion using a laser cutter. We created this print-and-fold hexapod as a miniature version of a commercially available platform, to which we compare several metrics, such as weight, walking speed, and cost of transportation. Our print-and-fold hexapod has a mass of 195 g, can walk at speeds of up to 38.1 cm/sec (two body lengths per second), and can be manufactured and assembled from scratch by a single person in approximately seven hours. Experimental results of gait control and trajectory tracking are provided.

Nature regularly uses self-folding as an efficient approach to automated fabrication. In engineered systems, however, the use of self-folding has been primarily restricted to the assembly of small structures using exotic materials and/or complex infrastructures. In this paper we present three approaches to the self-folding of structures using low-cost, rapid-prototyped shape memory laminates. These structures require minimal deployment infrastructure, and are activated by light, heat, or electricity. We compare the fabrication of a fundamental structure (a cube) using each approach, and test ways to control fold angles in each case. Finally, for each self-folding approach we present a unique structure that the approach is particularly suited to fold, and discuss the advantages and disadvantages of each approach.

This paper presents a method of road detection and mapping using accumulated 3D data from 2D scans. The idea of 3D rolling window is introduced, and its probabilistic characteristics are studied. A cascaded road detection process is developed with region-growing and classification methods. A probabilistic framework is utilized for road mapping purposes with the detection results. The performance of detection and mapping algorithm is evaluated through experiments.

Modern robots have to interact with their environment, search for objects, and move them around. Yet, for a robot to pick up an object, it needs to identify the object's orientation and locate it to within centimeter-scale accuracy. Existing systems that provide such information are either very expensive (e.g., the VICON motion capture system valued at hundreds of thousands of dollars) and/or suffer from occlusion and narrow field of view (e.g., computer vision approaches).

This paper describes a system that takes as input GPS data streams generated by users' phones and creates a searchable database of locations and activities. The system is called iDiary and turns large GPS signals collected from smartphones into textual descriptions of the trajectories. The system features a user interface similar to Google Search that allows users to type text queries on their activities (e.g., "Where did I buy books?") and receive textual answers based on their GPS signals.

In this paper, we address the problem of continually parsing a stream of 3D point cloud data acquired from a laser sensor mounted on a road vehicle. We leverage an online star clustering algorithm coupled with an incremental belief update in an evolving undirected graphical model. The fusion of these techniques allows the robot to parse streamed data and to continually improve its understanding of the world. The core competency produced is an ability to infer object classes from similarities based on appearance and shape features, and to concurrently combine that with a spatial smoothing algorithm incorporating geometric consistency. This formulation of feature-space star clustering modulating the potentials of a spatial graphical model is entirely novel. In our method, the two sources of information: feature similarity and geometrical consistency are fed continu- ally into the system, improving the belief over the class distributions as new data arrives. The algorithm obviates the need for hand-labeled training data and makes no apriori assumptions on the number or characteristics of object categories. Rather, they are learnt incrementally over time from streamed input data. In experiments per- formed on real 3D laser data from an outdoor scene, we show that our approach is capable of obtaining an ever- improving unsupervised scene categorization.

We present algorithms for simplifying and clustering patterns from sensors such as GPS, LiDAR, and other devices that can produce high-dimensional signals. The algorithms are suitable for handling very large (e.g. terabytes) streaming data and can be run in parallel on networks or clouds. Applications include compression, denoising, activity recognition, road matching, and map generation.

This paper describes a robot in the form of a self-folding sheet that is capable of origami-style autonomous folding. We describe the hardware device we designed and fabricated. The device is a sheet with a box-pleated pattern and an integrated electronic substrate and actuators. The sheet is programmed and controlled to achieve different shapes using an idea called sticker programming. We describe the sticker controller and its instantiation. We also describe the algorithms for programming and controlling a given sheet to self-fold into a desired shape. Finally we present experiments with a 4×4 hardware device and an 8×8 hardware device.

We present our digital fabrication technique for manufacturing active objects in 2D from a collection of smart particles. Given a passive model of the object to be formed, we envision submerging this original in a vat of smart particles, executing the new shape duplication algorithm described in this paper, and then brushing aside any extra modules to reveal both the original object and an exact copy, side-by-side. Extensions to the duplication algorithm can be used to create a magnified version of the original or multiple copies of the model object. Our novel duplication algorithm uses a distributed approach to identify the geometric specification of the object being duplicated and then forms the duplicate from spare modules in the vicinity of the original. This paper details the duplication algorithm and the features that make it robust to (1) an imperfect packing of the modules around the original object; (2) missing communication links between neighboring modules; and (3) missing modules in the vicinity of the duplicate object(s). We show that the algorithm requires O(1) storage space per module and that the algorithm exchanges O(n) messages per module. Finally, we present experimental results from 60 hardware trials and 150 simulations. These experiments demonstrate the algorithm working correctly and reliably despite broken communication links and missing modules.

This paper proposes a method for multi-agent path planning on a road network in the presence of congestion. We suggest a distributed method to find paths for multiple agents by introducing a probabilistic path choice achieving global goals such as the social optimum. This approach, which shows that the global goals can be achieved by local processing using only local information, can be parallelized and sped-up using massive parallel processing. The probabilistic assignment reliably copes with the case of random choices of unidentified agents or random route changes of agents who ignore our path guidance. We provide the analytical result on convergence and running time. We demonstrate and evaluate our algorithm by an implementation using asynchronous computation on multi-core computers.

We present algorithms for a motion planning for multiple agents whose goals are to visit multiple locations with probabilistic guarantees for achieving the goal. Though much research has been done in stochastic shortest path algorithms, the existing algorithms focus on the single-origin single-destination problem for one agent. This paper formulates a general framework for the stochastic shortest path problem with visit node constraints designed to achieve a specific goal with multiple agents, multiple resources, and multiple destinations. The constraints are defined by a set of sequences of nodes to be visited. Given predetermined constraints, our motion planning problem consists of finding the best agents, resources, and destinations, and the path through a sequence of nodes representing them. The technique in this paper solves the problem at the same level of complexity as solving the single-origin single-destination problem by parallelization. We demonstrate the algorithm by a Web-based traffic navigation guide system and evaluate the algorithm's performance.

This paper presents the design, modeling, and experimental verification of a novel, programmable connection mechanism for robots separated by a surface. The connector uses electropermanent magnets (EPMs) [1] to establish a continuum of clamping force between the robots, enabling the motion of one robot to slave the other during a variety of maneuvers. The authors design a novel, solid-state EPM arrangement capable of generating up to an estimated 890N of clamping force under environmental loading conditions. A relationship between geometric and environmental variables and connection assembly performance is first modeled and subsequently experimentally characterized. By implementing these connectors in a custom manufactured pair of assembly robots, the authors demonstrate the connection assembly and magnetizing hardware can be compactly fit within an autonomous robot application. We offer this mechanism as a repeatable, easily-automated alternative to robotic systems that depend on mechanic means to regulate clamping force [2].

Someday mobile robots will operate continually. Day after day, they will be in receipt of a never ending stream of images. In anticipation of this, this paper is about having a mobile robot generate apt and compact summaries of its life experience. We consider a robot moving around its environment both revisiting and exploring, accruing images as it goes. We describe how we can choose a subset of images to summarise the robot's cumulative visual experience. Moreover we show how to do this such that the time cost of generating an summary is largely independent of the total number of images processed. No one day is harder to summarise than any other.

One of the most prominent features on an urban road is the curb, which defines the boundary of a road surface. An intersection is a junction of two or more roads, appearing where no curb exists. The combination of curb and intersection features and their idiosyncrasies carry significant information about the urban road network that can be exploited to improve a vehicle's localization. This paper introduces a Monte Carlo Localization (MCL) method using the curb-intersection features on urban roads. We propose a novel idea of “Virtual LIDAR” to get the measurement models for these features. Under the MCL framework, above road observation is fused with odometry information, which is able to yield precise localization. We implement the system using a single tilted 2D LIDAR on our autonomous test bed and show robust performance in the presence of occlusion from other vehicles and pedestrians.

This paper presents decentralized algorithms for coverage with mobile robots on a graph. Coverage is an important capability of multi-robot systems engaged in a number of different applications, including placement for environmental modeling, deployment for maximal quality surveillance, and even coordinated construction. We use distributed vertex substitution for locational optimization, and the controllers minimize the corresponding cost functions. We prove that the proposed controller with two-hop communication guarantees convergence to the locally optimal configuration. We evaluate the algorithms in simulations and compare them to the coverage algorithms in a continuous domain.

The wide availability of networked sensors such as GPS and cameras is enabling the creation of sensor networks that generate huge amounts of data. For example, vehicular sensor networks where in-car GPS sensor probes are used to model and monitor traffic can generate on the order of gigabytes of data in real time. How can we compress streaming high-frequency data from distributed sensors? In this paper we construct coresets for streaming motion. The coreset of a data set is a small set which approximately represents the original data. Running queries or fitting models on the coreset will yield similar results when applied to the original data set.

We present an autonomous vehicle providing mobility-on-demand service in a crowded urban environment. The focus in developing the vehicle has been to attain autonomous driving with minimal sensing and low cost, off-the-shelf sensors to ensure the system's economic viability. The autonomous vehicle has successfully completed over 50 km handling numerous mobility requests during the course of multiple demonstrations. The video provides an overview of our approach, with special comments on our localization and perception modules showcasing one such request being serviced.

We consider the task of providing communication coverage to a group of sensing robots (sensors) moving independently to collect data. We provide communication via controlled placement of router vehicles that relay messages from any sensor to any other sensor in the system under the assumptions of 1) no cooperation from the sensors, and 2) only sensor-router or router-router communication over a maximum distance of R is reliable. We provide a formal framework and design provable exact and approximate (faster) algorithms for finding optimal router vehicle locations that are updated according to sensor movement. Using vehicle limitations, such as bounded control effort and maximum velocities of the sensors, our algorithm approximates areas that each router can reach while preserving connectivity and returns an expiration time window over which these positions are guaranteed to maintain communication of the entire system. The expiration time is compared against computation time required to update positions as a decision variable for choosing either the exact or approximate solution for maintaining connectivity with the sensors on-line.

This paper presents a novel semantic categorization method for 3D point cloud data using supervised, multiclass Gaussian Process (GP) classification. In contrast to other approaches, and particularly Support Vector Machines, which probably are the most used method for this task to date, GPs have the major advantage of providing informative uncertainty estimates about the resulting class labels. As we show in experiments, these uncertainty estimates can either be used to improve the classification by neglecting uncertain class labels or - more importantly - they can serve as an indication of the under-representation of certain classes in the training data. This means that GP classifiers are much better suited in a lifelong learning framework, where not all classes are represented initially, but instead new training data arrives during the operation of the robot.

We present an online algorithm for a robot to shape its path to a locally optimal configuration for collecting information in an unknown dynamic environment. As the robot travels along its path, it identifies both where the environment is changing, and how fast it is changing. The algorithm then morphs the robot's path online to concentrate on the dynamic areas in the environment in proportion to their rate of change. A Lyapunov-like stability proof is used to show that, under our proposed path shaping algorithm, the path converges to a locally optimal configuration according to a Voronoi-based coverage criterion. The path shaping algorithm is then combined with a previously introduced speed controller to produce guaranteed persistent monitoring trajectories for a robot in an unknown dynamic environment. Simulation and experimental results with a quadrotor robot support the proposed approach.

We investigate a data-driven approach to robotic path planning and analyze its performance in the context of interception tasks. Trajectories of moving objects often contain repeated patterns of motion, and learning those patterns can yield interception paths that succeed more often. We therefore propose an original trajectory clustering algorithm for extracting motion patterns from trajectory data and demonstrate its effectiveness over the more common clustering approach of using k-means. We use the results to build a Hidden Markov Model of a target's motion and predict movement. Our simulations show that these predictions lead to more effective interception. The results of this work have potential applications in coordination of multi-robot systems, tracking and surveillance tasks, and dynamic obstacle avoidance.

We consider the synthesis of control policies from temporal logic specifications for robots that interact with multiple dynamic environment agents. Each environment agent is modeled by a Markov chain whereas the robot is modeled by a finite transition system (in the deterministic case) or Markov decision process (in the stochastic case). Existing results in probabilistic verification are adapted to solve the synthesis problem. To partially address the state explosion issue, we propose an incremental approach where only a small subset of environment agents is incorporated in the synthesis procedure initially and more agents are successively added until we hit the constraints on computational resources. Our algorithm runs in an anytime fashion where the probability that the robot satisfies its specification increases as the algorithm progresses.

We present a distributed collision avoidance algorithm for multiple mobile robots that is model-predictive, sampling-based, and intuitive for operation around humans. Unlike purely reactive approaches, the proposed algorithm incorporates arbitrary trajectories as generated by a motion planner running on each navigating robot as well as predicted human trajectories. Our approach, inspired by human navigation in crowded pedestrian environments, draws from the sociology literature on pedestrian interaction. We propose a simple two-phase algorithm in which agents initially cooperate to avoid each other and then initiate civil inattention, thus lessening reactivity and committing to a trajectory. This process entails a pedestrian bargain in which all agents act competently to avoid each other and, once resolution is achieved, to avoid interfering with others' planned trajectories. This approach, being human-inspired, fluidly permits navigational interaction between humans and robots. We report experimental results for the algorithm running on real robots with and without human presence and in simulation.

Traffic congestion, volumes, origins, destinations, routes, and other road-network performance metrics are typically collected through survey data or via static sensors such as traffic cameras and loop detectors. This information is often out-of-date, difficult to collect and aggregate, difficult to analyze and quantify, or all of the above. In this paper we conduct a case study that demonstrates that it is possible to accurately infer traffic volume through data collected from a roving sensor network of taxi probes that log their locations and speeds at regular intervals. Our model and inference procedures can be used to analyze traffic patterns and conditions from historical data, as well as to infer current patterns and conditions from data collected in real-time. As such, our techniques provide a powerful new sensor network approach for traffic visualization, analysis, and urban planning.

This paper introduces CarSpeak, a communication system for autonomous driving. CarSpeak enables a car to query and access sensory information captured by other cars in a manner similar to how it accesses information from its local sensors. CarSpeak adopts a content-centric approach where information objects -- i.e., regions along the road -- are first class citizens. It names and accesses road regions using a multi-resolution system, which allows it to scale the amount of transmitted data with the available bandwidth. CarSpeak also changes the MAC protocol so that, instead of having nodes contend for the medium, contention is between road regions, and the medium share assigned to any region depends on the number of cars interested in that region.

Accurate vehicular localization is important for various cooperative vehicle safety (CVS) applications such as collision avoidance, turning assistant, etc. In this paper, we propose a cooperative vehicular distance measurement technique based on the sharing of GPS pseudorange measurements and a weighted least squares method. The classic double difference pseudorange solution, which was originally designed for high-end survey level GPS systems, is adapted to low-end navigation level GPS receivers for its wide availability in ground vehicles. The Carrier to Noise Ratio (CNR) of raw pseudorange measurements are taken into account for noise mitigation. We present a Dedicated Short Range Communications (DSRC) based mechanism to implement the exchange of pseudorange information among neighboring vehicles. As demonstrated in field tests, our proposed technique increases the accuracy of the distance measurement significantly compared with the distance obtained from the GPS fixes.

In this paper, we present controllers that enable mobile robots to persistently monitor or sweep a changing environment. The environment is modeled as a field that is defined over a finite set of locations. The field grows linearly at locations that are not within the range of a robot and decreases linearly at locations that are within range of a robot. We assume that the robots travel on given closed paths. The speed of each robot along its path is controlled to prevent the field from growing unbounded at any location. We consider the space of speed controllers that are parametrized by a finite set of basis functions. For a single robot, we develop a linear program that computes a speed controller in this space to keep the field bounded, if such a controller exists. Another linear program is derived to compute the speed controller that minimizes the maximum field value over the environment. We extend our linear program formulation to develop a multirobot controller that keeps the field bounded. We characterize, both theoretically and in simulation, the robustness of the controllers to modeling errors and to stochasticity in the environment.

We present a suite of algorithms that enable a team of mobile robots to repair connectivity in a wireless mesh network. Each robot carries a wireless router and can act as a mobile access point. The algorithms are distributed, with each robot computing it's trajectory using its position, the positions of its neighbors within communication range, and the position of a gateway node. The algorithms are validated via an analytical model as well as field experiments with 7 Create robots.

This paper describes several novel algorithms for shape formation by subtraction in programmable matter systems. These algorithms allow the simultaneous formation of multiple different shapes from a single block of host material. The resulting shapes are allowed to intertwine in arbitrarily complex ways. We also present a proof that the algorithms operate correctly to form the desired shapes. Finally, we show experimental results from close to 100 trials using both the Robot Pebbles hardware and a unique software simulator. Multiple trials of several different experiments demonstrate the algorithms operating correctly.

This work presents a technique which allows the application of 2-D fabrication methods to build 3-D robotic systems. The ability to print robots introduces a fast and low-cost fabrication method to modern, real-world robotic applications. To this end, we employ laser-engraved origami patterns to build a new class of robotic systems for mobility and manipulation. Origami is suitable for printable robotics as it uses only a flat sheet as the base structure for building complicated functional shapes, which can be utilized as robot bodies. An arbitrarily complex folding pattern can be used to yield an array of functionalities, in the form of actuated hinges or active spring elements. For actuation, we use compact NiTi coil actuators placed on the body to move parts of the structure on-demand. We demonstrate, as a proof-of-concept case study, the end-to-end fabrication and assembly of a simple mobile robot that can undergo worm-like peristaltic locomotion.

This paper examines the dynamic interplay between decentralized controllers and mesh networking protocols for controlling groups of robots. A proportional controller is used to maintain robots in a formation based on estimates of the robots' states observed through the network. The state information is propagated through the network using a flooding algorithm, which introduces topology-dependent time delays. The coupled interaction of information flow over the network with the dynamics of the robots is modeled as a linear dynamical system. With this model it is shown that systems made up of robots with stable first order dynamics are stable for all network update times, positive feedback gains, and connected communication graphs. With higher order robot dynamics it is found that stability is a complex and counter intuitive function of feedback gain and network update time. A performance metric is proposed for analyzing the convergence rate of the multi-robot system. Experiments with flying quadrotor robots verify the predictions of the model and the performance metric.

This paper presents controllers that enable a mobile robot to persistently monitor or sweep a changing environment. The changing environment is modeled as an accumulation function which grows in areas that are not within range of the robot, and decreases in areas that are within range of the robot. The robot must continually move through the environment to prevent the accumulation of any area from growing unbounded. We consider the case in which a predefined path is given for the robot, and we focus on controlling the robot's speed along the path. We characterize necessary and sufficient conditions on the speed controller of the robot for keeping the accumulation function bounded. We then search among the space of speed controllers that are parametrized by a finite set of basis functions. We develop a linear program to compute the optimal speed controller; that which minimizes the accumulation over the environment. Simulation results illustrate the performance of the controllers.

This paper proposes a path planning algorithm and a velocity control algorithm for underwater gliders to persistently monitor a patch of ocean. The algorithms address a pressing need among ocean scientists to collect high-value data for studying ocean events of scientific and environmental interest, such as the occurrence of harmful algal blooms. The path planner optimizes a cost function that blends two competing factors: it maximizes the information value of the path, while minimizing the deviation from the path due to ocean currents. The speed control algorithm then optimizes the speed along the planned path so that higher resolution samples are collected in areas of higher information value. The resulting paths are closed circuits that can be repeatedly traversed to collect long term ocean data in dynamic environments. The algorithms were tested during sea trials on an underwater glider operating off the coast of southern California over the course of several weeks. The results show significant improvements in data resolution and path reliability compared to a sampling path that is typically used in the region.

This paper presents a scalable information theoretic approach to infer the state of an environment by distributively controlling robots equipped with sensors. The robots iteratively estimate the environment state using a recursive Bayesian filter, while continuously moving to improve the quality of the estimate by following the gradient of mutual information. Both the filter and the controller use a novel algorithm for approximating the robots' joint measurement probabilities, which combines consensus (for decentralization) and sampling (for scalability). The approximations are shown to approach the true joint measurement probabilities as the size of the consensus rounds grows or as the network becomes complete. The resulting gradient controller runs in constant time with respect to the number of robots, and linear time with respect to the number of sensor measurements and environment discretization cells, while traditional mutual information methods are exponential in all of these quantities. Furthermore, the controller is proven to be convergent between consensus rounds and, under certain conditions, is locally optimal. The complete distributed inference and coordination algorithm is demonstrated in experiments with five quad-rotor flying robots and simulations with 100 robots.

This paper presents the design, fabrication, and evaluation of a novel type of valve that uses an electropermanent magnet [1]. This valve is then used to build actuators for a soft robot. The developed EPM valves require only a brief (5 ms) pulse of current to turn flow on or off for an indefinite period of time. EPMvalves are characterized and demonstrated to be well suited for the control of elastomer fluidic actuators. The valves drive the pressurization and depressurization of fluidic channels within soft actuators. Furthermore, the forward locomotion of a soft, multi-actuator rolling robot is driven by EPM valves. The small size and energy-efficiency of EPM valves may make them valuable in soft mobile robot applications.

Persistent robot tasks such as monitoring and cleaning are concerned with controlling mobile robots to act in a changing environment in a way that guarantees that the uncertainty in the system (due to change and to the actions of the robot) remains bounded for all time. Prior work in persistent robot tasks considered only robot systems with collision-free paths that move following speed controllers. In this paper we describe a solution to multi-robot persistent monitoring, where robots have intersecting trajectories. We develop collision and deadlock avoidance algorithms that are based on stopping policies, and quantify the impact of the stopping times on the overall stability of the speed controllers.

This paper presents a constraint-aware decentralized approach to construction with teams of robots. We present an extension to existing work on a distributed controller for robotic construction of simple structures. Our previous work described a set of adaptive algorithms for constructing truss structures given a target geometry using continuous and graph-based equal-mass partitioning [1], [2]. Using this work as a foundation, we present an algorithm which performs construction tasks and conforms to physical constraints while considering those constraints to parallelize tasks. This is accomplished by defining a mass function which reflects the priority of part placement and prevents physically impossible states. This mass function generates a set of pointmasses in ℝn, and we present a novel algorithm for finding a locally optimal, equal-mass, convex tessellation of such a set.

In this paper we present a method for automatically planning optimal paths for a group of robots that satisfy a common high level mission specification. Each robot's motion in the environment is modeled as a weighted transition system. The mission is given as a Linear Temporal Logic formula. In addition, an optimizing proposition must repeatedly be satisfied. The goal is to minimize the maximum time between satisfying instances of the optimizing proposition. Our method is guaranteed to compute an optimal set of robot paths. We utilize a timed automaton representation in order to capture the relative position of the robots in the environment. We then obtain a bisimulation of this timed automaton as a finite transition system that captures the joint behavior of the robots and apply our earlier algorithm for the single robot case to optimize the group motion. We present a simulation of a persistent monitoring task in a road network environment.

This paper presents a decentralized control strategy for positioning and orienting multiple robotic cameras to collectively monitor an environment. The cameras may have various degrees of mobility from six degrees of freedom, to one degree of freedom. The control strategy is proven to locally minimize a novel metric representing information loss over the environment. It can accommodate groups of cameras with heterogeneous degrees of mobility (e.g., some that only translate and some that only rotate), and is adaptive to robotic cameras being added or deleted from the group, and to changing environmental conditions. The robotic cameras share information for their controllers over a wireless network using a specially designed multihop networking algorithm. The control strategy is demonstrated in repeated experiments with three flying quadrotor robots indoors, and with five flying quadrotor robots outdoors. Simulation results for more complex scenarios are also presented.

This paper presents a solution to decentralized Voronoi coverage in non-convex polygonal environments. We show that complications arise when existing approaches to Voronoi coverage are applied for deploying a group of robots in non-convex environments. We present an algorithm that is guaranteed to converge to a local optimum. Our algorithm combines classical Voronoi coverage with the Lloyd algorithm and the local path planning algorithm TangentBug to compute the motion of the robots around obstacles and corners. We present the algorithm and prove convergence and optimality. We also discuss experimental results from an implementation with five robots.

We present a control algorithm for autonomous underwater robots with modular thruster configuration. The algorithm can handle arbitrary thruster configurations. It maintains the robot's desired attitude while solving for translational motion. The attitude can be arbitrarily chosen from the special orthogonal group SO3allowing the robot all possible orientations. The desired translational velocities can be chosen from R3allowing the robot to follow arbitrary trajectories underwater. If the robot is not fully holonomic then the controller chooses the closest possible solution using least squares and outputs the error vector. We verify the controller with experiments using our autonomous underwater robot AMOUR. We achieve roll errors of 1.0 degree (2.1 degrees standard deviation) and pitch errors of 1.5 degrees (1.8 degrees standard deviation). We also demonstrate experimentally that the controller can handle both nonholonomic and fully holonomic thruster configurations of the robot. In the later case we show how depth can be maintained while performing 360 degree rolls. Further, we demonstrate an input device that allows a user to control the robot's attitude while moving along a desired trajectory.

We develop a distributed controller to position a team of aerial vehicles in a configuration that optimizes communication-link quality, to support a team of ground vehicles performing a collaborative task. We propose a gradient-based control approach where agents' positions locally minimize a physically motivated cost function. The contributions of this paper are threefold. We formulate of a cost function that incorporates a continuous, physical model of signal quality, SIR. We develop a non-smooth gradient-based controller that positions aerial vehicles to acheive optimized signal quality amongst all vehicles in the system. This controller is provably convergent while allowing for non-differentiability due to agents moving in or out of communication with one another. Lastly, we guarantee that given certain initial conditions or certain values of the control parameters, aerial vehicles will never disconnect the connectivity graph. We demonstrate our controller on hardware experiments using AscTec Hummingbird quadrotors and provide aggregate results over 10 trials. We also provide hardware-in-the-loop and MATALB simulation results, which demonstrate positioning of the aerial vehicles to minimize the cost function H and improve signal-quality amongst all communication links in the ground/air robot team.

This paper describes the design, fabrication, and experimental results of a programmable matter system capable of 2D shape formation through subtraction. The system is composed of autonomous 1cm modules which use custom-designed electropermanent magnets to bond, communicate, and share power with their neighbors. Given an initial block composed of many of these modules latched together in a regular crystalline structure, our system is able to form shapes by detaching the unnecessary modules. Many experiments show that the modules in our system are able to distribute data at 9600bps to their neighbors with a 98.5% success rate after four retries, and the connectors are able to support over 85 times the weight of a single module.

This paper presents a soft robotic platform that exhibits peristaltic locomotion. The design principle is based on the unique antagonistic arrangement of radial/circular and longitudinal muscle groups of Oligochaeta. Sequential antagonistic motion is achieved in a flexible braided mesh-tube structure with NiTi coil actuators. A numerical model for the mesh structure describes how peristaltic motion induces robust locomotion and details the deformation by the contraction of NiTi actuators. Several peristaltic locomotion modes are modeled, tested, and compared on the basis of locomotion speed. The entire mechanical structure is made of flexible mesh materials and can withstand significant external impacts during locomotion. This approach can enable a completely soft robotic platform by employing a flexible control unit and energy sources.

Our prior work [1] presented a decentralized algorithm for coordinating the construction of a truss structure out of multiple components. In this paper, we discuss adaptation in decentralized construction. We partition construction in two tasks, tool delivery and assembly. Each task is performed by a networked team of specialized robots. We analyze the performance of the algorithms using the balls into bins problem, and show their adaptation to failure of robots, dynamic constraints, multiple types of elements and reconfiguration. The algorithms can be used for general types of source elements.

We present a distributed algorithm for creating a modular shape by magnification. The input to the algorithm is presented with a small scale version of the desired shape and a magnification factor m . The output of the system is the object that corresponds to the m-fold magnification of the input shape. We describe and analyze a distributed algorithm for this capability and present simulation results. Making shapes by magnification can be viewed as a programming interface for creating objects by programming matter.

Our prior work presented a decentralized algorithm for coordinating the construction of truss shaped objects out of multiple components (rods and connectors). In this paper, we consider how to transfer the theory to practice, implementing the algorithm to create a decentralized multi robot construction system. The system is composed of mobile manipulators and smarts parts with an embedded communication device. We discuss the delivery and assembly algorithms that comprise this system and the assumptions behind them. We present data from extensive hardware experiments with 4 robots coordinating an assembly task.

This paper addresses the problem of deploying a group of networked robots on a non-planar surface embedded in 3D space. Two distributed coverage control algorithms are presented that both provide a solution to the problem by discrete coverage of a graph. The first method computes shortest paths and runs the Lloyd algorithm on the graph to obtain a centroidal Voronoi tessellation. The second method uses the Euclidean distance measure and locally exchanges mesh cells between approximated Voronoi regions to reach an optimal robot configuration. Both methods are compared and evaluated in simulations and in experiments with five robots on a curved surface.

Underwater vehicles are typically operated using a tether or a slow acoustic link. We present an underwater optical communication system that enables a high-throughput and low-latency link to an underwater robot. The optical link allows the robot to operate in cluttered environments without the need for a tether. We demonstrate the performance of the system in a number of experiments which characterize the optical link and demonstrate remote control of the robot using a human input device.

In this paper we present a method for automatically generating optimal robot trajectories satisfying high level mission specifications. The motion of the robot in the environment is modeled as a weighted transition system. The mission is specified by a general linear temporal logic formula. In addition, we require that an optimizing proposition must be repeatedly satisfied. The cost function that we seek to minimize is the maximum time between satisfying instances of the optimizing proposition. For every environment model, and for every formula, our method computes a robot trajectory which minimizes the cost function. The problem is motivated by robotic monitoring and data gathering. In this setting, the optimizing proposition is satisfied at locations where data can be uploaded, and the formula specifies a an infinite horizon data collection mission. Our method utilizes Büchi automata to produce an automaton (which can be thought of as a graph) whose runs satisfy the temporal logic formula. We then present a graph algorithm which computes a path corresponding to the optimal robot trajectory. We also present an implementation for a robot performing a data gathering mission.

Understanding the dynamics of bodies of water and their impact on the global environment requires sensing information over the full volume of water. We develop a gradient-based decentralized controller that dynamically adjusts the depth of a network of underwater sensors to optimize sensing for computing maximally detailed volumetric models. We prove that the controller converges to a local minimum. We implement the controller on an underwater sensor network capable of adjusting their depths. Through simulations and experiments, we verify the functionality and performance of the system and algorithm.

We study a fully distributed, reactive algorithm for deployment and maintenance of a mobile communication backbone that provides an area around a network gateway with wireless network access for higher-level agents. Possible applications of such a network are distributed sensor networks as well as communication support for disaster or military operations. The algorithm has minimalist requirements on the individual robotic node and does not require any localization. This makes the proposed solution suitable for deployment of large numbers of comparably cheap mobile communication nodes and as a backup solution for more capable systems in GPS-denied environments. Robots keep exploring the configuration space by random walk and stop only if their current location satisfies user-specified constraints on connectivity (number of neighbors). Resulting deployments are robust and convergence is analyzed using both kinematic simulation with a simplified collision and communication model as well as a probabilistic macroscopic model. The approach is validated on a team of 9 iRobot Create robots carrying wireless access points in an indoor environment.

This paper addresses the problem of coverage control of a network of mobile sensors. In the current literature, this is commonly formulated as a locational optimization problem under the assumption that sensing performance is independent of the locations of sensors. We extend this work to a more general framework where the sensor model is location-dependent. We propose a distributed control law and coordination algorithm. If the global sensing performance function is known a priori, we prove that the algorithm is guaranteed to converge. To validate this algorithm, we conduct experiments with indoor and outdoor deployments of Cyclops cameras and model its sensing performance. This model is used to simulate deployments on 1D pathways and study the coverage obtained. We also examine the coverage in the case when the global sensing function is not known and is estimated in an online fashion.

This paper presents a distributed control strategy for deploying hovering robots with multiple downward facing cameras to collectively monitor an environment. Information per pixel is proposed as an optimization criterion for multi-camera placement problems. This metric is used to derive a specific cost function for multiple downward facing cameras mounted on hovering robot platforms. The cost function leads to a gradient-based distributed controller for positioning the robots. A convergence proof using LaSalle's invariance principle is given to show that the robots converge to locally optimal positions. The controller is demonstrated in experiments with three flying quad-rotor robots.

In this paper we describe an optimal reconfiguration planning algorithm that morphs a grounded truss structure of known geometry into a new geometry. The plan consists of a sequence of paths to move truss elements to their new locations that generate the new truss geometry. The trusses are grounded and remain connected at all time. Intuitively, the algorithm grows gradually the new truss structure from the old one. The truss elements are rigid bars joined with 18-way connectors. The paper also introduces the design of a truss-climbing robot that can execute the plan.

This paper describes the architecture and implementation of a distributed autonomous gardening system. The garden is a mesh network of robots and plants. The gardening robots are mobile manipulators with an eye-in-hand camera. They are capable of locating plants in the garden, watering them, and locating and grasping fruit. The plants are potted cherry tomatoes enhanced with sensors and computation to monitor their well-being (e.g. soil humidity, state of fruits) and with networking to communicate servicing requests to the robots. Task allocation, sensing and manipulation are distributed in the system and de-centrally coordinated. We describe the architecture of this system and present experimental results for navigation, object recognition and manipulation.

A control strategy inspired by the hunting tactics of ladybugs is presented to simultaneously achieve sensor coverage and exploration of an area with a group of networked robots. The controller is distributed in that it requires only information local to each robot, and adaptive in that it modifies its behavior based on information in the environment. The ladybug controller is developed as a modification to a basic coverage control law, first for the non-adaptive case, then for the adaptive case. Stability is proven for both cases with a Lyapunov-type proof. Results of numerical simulations are presented.

A decentralized controller is presented that causes a network of robots to converge to a near optimal sensing configuration, while simultaneously learning the distribution of sensory information in the environment. A consensus (or flocking) term is introduced in the learning law to allow sharing of parameters among neighbors, greatly increasing learning convergence rates. Convergence and consensus is proven using a Lyapunov-type proof. The controller with parameter consensus is shown to perform better than the basic controller in numerical simulations.

We describe self-assembling robot arm systems composed of active modular robots and passive bars.We present a case study where the robotic module is the Shady3D robot and the passive component is a rigid bar with embedded IR LEDs. We propose algorithms that demonstrate the cooperative aggregation of modular robotic manipulators with greater capability and workspace out of these two types of elements. We present results from physical experiments in which two 3DOF Shady3D robots and one rigid bar coordinate to self-assemble into a 6DOF manipulator. We then demonstrate cooperative algorithms for forward and inverse kinematics, grasping, and mobility with this arm.

Precise robot positioning is important for many applications in indoor environments. Current solutions to the indoor localization problem are either both unreliable and inaccurate, or very expensive. In this paper we propose, design and build a low-cost, robust and accurate indoor localization system using laser light sources. The system calculates the coordinates of a robotic arm by using triangulation algorithms with precisely measured values of the angles of the receiver with respect to the three laser emitters. A system of three rotating lasers and receiver unit was built and deployed in the wing of an aircraft. Using this system, a robotic arm could be localized accurately within error margins defined approximately by Gaussian distributions centered at the object’s true coordinate values and with standard deviations of 0.19 mm, 0.11 mm and 0.34 mm in the x, y and z coordinate directions respectively. The system was also used to detect vertical drop in the robotic arm due to its weight as it extends to perform fitting operations on the skin of the wing. Feedback from the laser localization system was used to adjust the position of the tip of the robotic arm in order to perform a sequence of high precision docking tasks within the aircraft wing

We describe algorithms to build self-assembling robot systems composed of active modular robots and passive bars. The distributed algorithms are based on locally optimal matching. We demonstrate how to build an active structure by the cooperative aggregation and disassembly of modular robotic manipulators. A target structure is modeled as a dynamic graph. We prove that the same optimality - quadratic competitive ratio - as for the static graph can be achieved for the algorithms. We demonstrate how this algorithm can be used to build truss-like structures.

The Personal Commute Portal (PCP) is a Web-based traffic information system that provides a good driving direction and personalized route recommendation using historical and real-time traffic data obtained by a vehicular sensor network.

Predictive environmental sensor networks provide complex engineering and systems challenges. These systems must withstand the event of interest, remain functional over long time periods when no events occur, cover large geographical regions of interest to the event, and support the variety of sensor types needed to detect the phenomenon. Prediction of the phenomenon on the network complicates the system further, requiring additional computation on themicrocontrollers and utilizing prediction models that are not typically designed for sensor networks. This paper describes a system architecture and deployment to meet the design requirements and to allow model-driven control, thereby optimizing the prediction capability of the system. We explore the application of river flood prediction using this architecture, describing our work on a centralized form of the prediction model, network implementation, component testing and infrastructure development in Honduras, deployment on a river in Massachusetts, and results of the field experiments. Our system uses only a small number of nodes to cover basins of 1000-10000 square km2 using an unique heterogeneous communication structure to provide real-time sensed data, incorporating self-monitoring for failure, and adapting measurement schedules to capture events of interest.

This paper describes a novel experiment in which two very different methods of underwater robot localization are compared. The first method is based on a geometric approach in which a mobile node moves within a field of static nodes, and all nodes are capable of estimating the range to their neighbours acoustically. The second method uses visual odometry, from stereo cameras, by integrating scaled optical flow. The fundamental algorithmic principles of each localization technique is described. We also present experimental results comparing acoustic localization with GPS for surface operation, and a comparison of acoustic and visual methods for underwater operation.

We describe the design, implementation, and experimentation with a collection of robots that, starting from an amorphous arrangement, can be assembled into arbitrary shapes and then commanded to self-disassemble in an organized manner. Each of the 28 modules in the system is implemented as a 1.8-inch autonomous cube-shaped robot able to connect to and communicate with its immediate neighbors. Two cooperating microprocessors control each module's magnetic connection mechanisms and infrared communication interfaces. When assembled into a structure, the modules form a system that can be virtually sculpted using a computer interface. We report on the hardware design and experiments from hundreds of trials.

We describe an efficient, reliable, and robust four-rotor flying platform for indoor and outdoor navigation. Currently, similar platforms are controlled at low frequencies due to hardware and software limitations. This causes uncertainty in position control and unstable behavior during fast maneuvers. Our flying platform offers a 1 kHz control frequency and motor update rate, in combination with powerful brushless DC motors in a light-weight package. Following a minimalistic design approach this system is based on a small number of low-cost components. Its robust performance is achieved by using simple but reliable highly optimized algorithms. The robot is small, light, and can carry payloads of up to 350g

A decentralized, adaptive control law is presented to drive a network of mobile robots to a near-optimal sensing configuration. The control law is adaptive in that it integrates sensor measurements to provide a converging estimate of the distribution of sensory information in the environment. It is decentralized in that it requires only information local to each robot. A Lyapunov-type proof is used to show that the control law causes the network to converge to a near-optimal sensing configuration, and the controller is demonstrated in numerical simulations. This technique suggests a broader application of adaptive control methodologies to decentralized control problems in unknown dynamical environments.

This paper describes a truss climbing robot we designed and prototyped. The robot has a minimalist design with three motive degrees of freedom that enable movement along three-dimensional truss structures. This robot can form a six-degree-of-freedom structure by connecting to another identical module using a passive bar as a medium. We present the design and implementation of this robot, control algorithms for moving the robot in a 3D truss structure, and hardware experiments

In developing countries, flooding due to natural disasters such as hurricanes and earthquakes results in massive loss of life and property. Warning communities of the incoming flood provides an effective solution to this by giving people sufficient time to evacuate and protect their property. However, the range of early warning system solutions introduces a tangle of conflicting requirements including cost and reliability, and creates several interesting problems from factors as diverse as technological, social, and political. The complexity of these systems and need for autonomy within the context of a developing country while remaining maintainable and accessible by non-technical personnel provides a challenge not often solved within developed countries, much less the developing. After describing this problem, the paper discusses a proposed solution for the problem, initial experiments in implementing the solution, and lessons learned through that work.

This paper considers the problem of allocating tasks among robots that operate on a 3D truss. Each robot is commanded to navigate to a different location for work. When the information about the robots' initial and desired locations are centrally known, this problem reduced to a classical disjoint-path problem. In this paper we consider the distributed problem where each robot knows its own goals only and we wish to plan an optimal set of steps for each robot that minimizes energy while fulfilling the task requirements. The challenge is to cope with possible path collisions. We present and analyze a distributed algorithm. We describe a simulation of this algorithm and show data from a physical experiment.

This paper explores the space of active modular trusses, ranging from a passive truss with one independent active climbing module to fully self-reconfiguring dynamically controllable trusses comprised of active modules and passive struts. We describe a hardware design for truss climbing and present hierarchical algorithms for controlling hyper-redundant modular trusses

We present algorithms, systems, and experimental results for underwater data muling. In data muling a mobile agent interacts with static agents to upload, download, or transport data to a different physical location. We consider a system comprising an autonomous underwater vehicle (AUV) and many static underwater sensor nodes (USN) networked together optically and acoustically. The AUV can locate the static nodes using vision and hover above the static nodes for data upload. We describe the hardware and software architecture of this underwater system, as well as experimental data

We describe a system in which autonomous robots assemble two-dimensional structures out of square building blocks. A fixed set of local control rules is sufficient for a group of robots to collectively build arbitrary solid structures. We present and compare four versions in which blocks are (1) inert and indistinguishable, (2) uniquely labeled, (3) able to be relabeled by robots, (4) capable of some computation and local communication. Added block capabilities increase the availability of nonlocal structural knowledge, thereby increasing robustness and significantly speeding construction. In this way we extend the principle of stigmergy (storing information in the environment) used by social insects, by increasing the capabilities of the blocks that represent that environmental information. Finally, we describe hardware experiments using a prototype capable of building arbitrary solid 2-D structures

Global synchronization is important for many sensor network applications that require precise mapping of collected sensor data with the time of the events, for example, in tracking and surveillance. It also plays an important role in energy conservation in MAC layer protocols. This paper describes four methods to achieve global synchronization in a sensor network: a node-based approach, a hierarchical cluster-based method, a diffusion-based method, and a fault-tolerant diffusion-based method. The diffusion-based protocol is fully localized. We present two implementations of the diffusion-based protocol for synchronous and asynchronous systems and prove its convergence. Finally, we show that, by imposing some constraints on the sensor network, global clock synchronization can be achieved in the presence of malicious nodes that exhibit Byzantine failures.

Most reconfiguration planners for self-reconfiguring robots do not consider the placement of specific modules within the configuration. Recently, we have begun to investigate heterogeneous reconfiguration planning in lattice-based systems, in which there are various classes of modules. The start and goal configurations specify the class of each module, in addition to placement. Our previous work presents solutions for this problem with unrestricted free space available to the robot during reconfiguration, and also free space limited to a thin connected region over the entire surface of the configuration. In this paper, we further this restriction and define free space by an arbitrarily-shaped bounding region. This addresses the important problem of reconfiguration among obstacles, and reconfiguration over a rigid surface. Our algorithm plans module trajectories through the volume of the structure, and is divided into two phases: shape-forming, and sorting the goal configuration to correctly position modules by class. The worst-case running time for the first phase is O(n2) with O(n2) moves for an n-module robot, and a loose upper bound for the second phase is O(n4) time and moves. However, we show this bound to be Θ (n2)time and moves in common instances.

In this paper we describe a modular self-reconfiguring robot composed of Molecule robot modules. We present the architecture of this robot and discuss how self-reconfiguration can be used as a locomotion gait for this system. We present two types of locomotion algorithms for this robot: a statically stable tumbling algorithm and a dynamically stable algorithm that achieves locomotion by modifying the center of mass of the robot. For each algorithm we analyze the efficiency of the self-reconfiguration gait for locomotion. Finally we present experimental data for the tumbling algorithm implemented on a four-module Molecule robot.

We propose a novel modular underwater robot which can self-reconfigure by stacking and unstacking its component modules. Applications for this robot include underwater monitoring, exploration, and surveillance. Our current prototype is a single module which contains several subsystems that later will be segregated into different modules. This robot functions as a testbed for the subsystems which are needed in the modular implementation. We describe the module design and discuss the propulsion, docking, and optical ranging subsystems in detail. Experimental results demonstrate depth control, linear motion, target module detection, and docking capabilities.

We describe VIsolate (Voronoi Isolate), a system which performs geometric computations associated with toolpath planning for mechanical etch (also called isolation routing) of printed-circuit boards, including the computation of a novel Voronoi-based toolpath with some advantages over the current industry practice. We highlight how we use the 3D Graphics Processing Unit (GPU) to implement simple, intuitive algorithms in VIsolate, including polygon overlap detection, 2D offset, and constrained generalized Voronoi diagram computation, building on a method from [1]. Thus, this work also illustrates how we can employ the GPU as a rudimentary ‘ mind’s eye’ for the machine, allowing us to rapidly implement visually-intuitive geometric algorithms.

We consider the problem of deploying or repairing a sensor network to guarantee a specified level of multi-path connectivity (k-connectivity) between all nodes. Such a guarantee simultaneously provides fault tolerance against node failures and high capacity through multi-path routing. We design and analyze the first algorithms that place an almost-minimum number of additional sensors to augment an existing network into a k-connected network, for any desired parameter k. Our algorithms have provable guarantees on the quality of the solution. Specifically, we prove that the number of additional sensors is within a constant factor of the absolute minimum, for any fixed k. We have implemented greedy and distributed versions of this algorithm, and demonstrate in simulation that they produce high-quality placements for the additional sensors. We are also in the process of using our algorithms to deploy nodes in a physical sensor network using a mobile robot.

In this paper we present a novel platform for underwater sensor networks to be used for long-term monitoring of coral reefs and fisheries. The sensor network consists of static and mobile underwater sensor nodes. The nodes communicate point-to-point using a novel high-speed optical communication system integrated into the TinyOS stack, and they broadcast using an acoustic protocol integrated in the TinyOS stack. The nodes have a variety of sensing capabilities, including cameras, water temperature, and pressure. The mobile nodes can locate and hover above the static nodes for data muling, and they can perform network maintenance functions such as deployment, relocation, and recovery. In this paper we describe the hardware and software architecture of this underwater sensor network. We then describe the optical and acoustic networking protocols and present experimental networking and data collected in a pool, in rivers, and in the ocean. Finally, we describe our experiments with mobility for data muling in this network.

We develop distributed algorithms for adaptive sensor networks that respond to directing a target through a region of space. We model this problem as an online distributed motion planning problem. Each sensor node senses values in its perception space and has the ability to trigger exceptions events we call “danger” and model as “obstacles”. The danger/obstacle landscape changes over time. We present algorithms for computing distributed maps in perception space and for using these maps to compute adaptive paths for a mobile node that can interact with the sensor network. We give the analysis to the protocol and report on hardware experiments using a physical sensor network consisting of Mote sensors. We also show how to reduce searching space and communication cost using Voronoi diagram.

We describe a moving virtual fence algorithm for herding cows. Each animal in the herd is given a smart collar consisting of a GPS, PDA, wireless networking and a sound amplifier. Using the GPS, the animal's location can be verified relative to the fence boundary. When approaching the perimeter, the animal is presented with a sound stimulus whose effect is to move away. We have developed the virtual fence control algorithm for moving a herd. We present simulation results and data from experiments with 8 cows equipped with smart collars.

Sensor networks are systems of many small units that work together to monitor a given environment. Endowing such sensor units with mobility can allow them to reactively converge on more interesting portions of their environment. This enables the concentration of sensing resources where they are most useful and provides robustness by delivering redundancy at the point of interest. However, when converging, in general the sensors should not leave any portion of the environment unsensed. In this paper, we review distributed methods for controlling the sensors and describe a family of distributed methods for retaining coverage while allowing the convergence to proceed where possible. The coverage methods are based on the Voronoi diagram of the sensors' positions, and can use different amounts of communication and computation to produce complete coverage of the environment. We also describe extensions that serve to make coverage more uniform or allow specific areas to be left uncovered. We present implementations of these algorithms in simulation and describe results and avenues of future work.

We describe a sensor network deployment method using autonomous flying robots. Such networks are suitable for tasks such as large-scale environmental monitoring or for command and control in emergency situations. We describe in detail the algorithms used for deployment and for measuring network connectivity and provide experimental data we collected from field trials. A particular focus is on determining gaps in connectivity of the deployed network and generating a plan for a second, repair, pass to complete the connectivity. This project is the result of a collaboration between three robotics labs (CSIRO, USC, and Dartmouth.).

We develop distributed algorithms for sensor networks that respond by directing a target (robot or human) through a region. The sensor network models the event levels sensed across a geographical area, adapts to changes, and guides a moving object incrementally across the network. We describe a device we call a flashlight for interacting with the sensor field. This interaction includes collecting navigation information from the sensors in the local neighborhood, activating and deactivating specified areas of the sensor network, and detecting events in the sensor network. We report on hardware experiments using a physical sensor network consisting of Mote sensors.

Global synchronization is crucial to many sensor network applications that require precise mapping of the collected sensor data with the time of the events, for example in tracking and surveillance. It also plays an important role in energy conservation in MAC layer protocols. This paper discusses three methods to achieve global synchronization in a sensor network: a node-based approach, a hierarchical cluster-based method, and a fully localized diffusion-based method. We also give the synchronous and asynchronous implementations of the diffusion-based protocols.

In this paper we present generic distributed algorithms for assembling and repairing shapes using modular self-reconfiguring robots. The algorithms work in the sliding cube model. Each module independently evaluates a set of local rules using different evaluation models. Two methods are used to determine the correctness of the algorithms - a graph analysis technique which can prove the rule set is correct for specific instances of the algorithm, and a statistical technique which can produce arbitrary bounds on the likelihood that the rule set functions correctly. An extension of the assembly algorithm can be used to produce arbitrary non-cantilevered convex shapes without holes. The algorithms have been implemented and evaluated in simulation.

We propose to automate controller design for distributed modular robots. In this paper, we present some initial experiments with learning distributed controllers for synthesizing compliant locomotion gaits for modular, self-reconfigurable robots. We use both centralized and distributed policy search and find that the learning approach is promising, as locomotion tasks are learnt well. We also find that the additive nature of the robotic platforms can help speed up learning if we increase the robot size incrementally.