Uncertainty in perception, actuation, and the environment often require multiple attempts for a robotic task to be successful. We study a class of problems providing (1) low-entropy indicators of terminal success / failure, and (2) unreliable (high-entropy) data to predict the final outcome of an ongoing task. Examples include a robot trying to connect with a charging station, parallel parking, or assembling a tightly-fitting part. The ability to restart after predicting a failure early, versus simply running to failure, can significantly decrease the makespan, that is, the total time to completion, with the drawback of potentially short-cutting an otherwise successful operation. Assuming task running times to be Poisson distributed, and using a Markov Jump Process to capture the dynamics of the underlying Markov Decision Process, we derive a closed-form solution that predicts makespan based on the confusion matrix of the failure predictor. This allows the robot to learn failure prediction in a production environment, and only adopt a preemptive policy when it actually saves time. We demonstrate this approach using a robotic peg-in-hole assembly problem. Failures are predicted by a dilated convolutional network based on force-torque data, showing an average makespan reduction from 101s to 81s ($\mathrm{N}=120,\ \mathrm{p} < 0.05$). We posit that the proposed algorithm generalizes to any robotic behavior with an unambiguous terminal reward, with wide ranging applications on how robots can learn and improve their behaviors in the wild.

The need to create more viable soft sensors is increasing in tandem with the growing interest in soft robots. Several sensing methods, like capacitive stretch sensing and intrinsic capacitive self-sensing, have proven to be useful when controlling soft electro-hydraulic actuators, but are still problematic. This is due to challenges around high-voltage electronic interference or the inability to accurately sense the actuator at higher actuation frequencies. These issues are compounded when trying to sense and control the movement of a multiactuator system. To address these shortcomings, we describe a two-part magnetic sensing mechanism to measure the changes in displacement of an electro-hydraulic (HASEL) actuator. Our magnetic sensing mechanism can achieve high accuracy and precision for the HASEL actuator displacement range, and accurately tracks motion at actuation frequencies up to 30 Hz, while being robust to changes in ambient temperature and relative humidity. The high accuracy of the magnetic sensing mechanism is also further emphasized in the gripper demonstration. Using this sensing mechanism, we can detect submillimeter difference in the diameters of three tomatoes. Finally, we successfully perform closed-loop control of one folded HASEL actuator using the sensor, which is then scaled into a deformable tilting platform of six units (one HASEL actuator and one sensor) that control a desired end effector position in 3D space. This work demonstrates the first instance of sensing electro-hydraulic deformation using a magnetic sensing mechanism. The ability to more accurately and precisely sense and control HASEL actuators and similar soft actuators is necessary to improve the abilities of soft, robotic platforms.

Locomotion through rolling is attractive compared to other forms of locomotion thanks to uniform designs, high degree of mobility, dynamic stability, and self-recovery from collision. Despite previous efforts to design rolling soft systems, pneumatic and other soft actuators are often limited in terms of high-speed dynamics, system integration, and/or functionalities. Furthermore, mathematical description of the rolling dynamics for this type of robot and how the models can be used for speed control are often not mentioned. This article introduces a cylindrical-shaped shell-bulging rolling soft wheel that employs an array of 16 folded-HASEL actuators as a mean for improved rolling performance. The actuators represent the soft components with discrete forces that propel the wheel, whereas the wheel's frame is rigid but allows for smooth, continuous change in position and speed. We discuss the interplay between the electrical and mechanical design choices, the modeling of the wheel's hybrid (continuous and discrete) dynamic behavior, and the implementation of a model predictive controller (MPC) for the robot's speed. With the balance of several design factors, we show the wheel's ability to carry integrated hardware with a maximum rolling speed at 0.7 m/s (or 2.2 body lengths per second), despite its total weight of 979 g, allowing the wheel to outperform the existing rolling soft wheels with comparable weights and sizes. We also show that the MPC enables the wheel to accelerate and leverage its inherent braking capability to reach desired speeds—a critical function that did not exist in previous rolling soft systems.

This paper describes a system identification method and the development of a closed-loop controller for a Hydraulically Amplified Self-healing Electrostatic (HASEL) actuator. Our efforts focus on developing a reliable and consistent way to identify system models for these soft robotic actuators using high-speed videography based motion tracking. Utilizing a mass-spring-damper model we are able to accurately capture the behavior of a HASEL actuator. We use the resulting plant model to design a Proportional-Integral controller that demonstrates improved closed-loop tracking and steady-state error performance.

This paper presents an overview of the inaugural Amazon Picking Challenge along with a summary of a survey conducted among the 26 participating teams. The challenge goal was to design an autonomous robot to pick items from a warehouse shelf. This task is currently performed by human workers, and there is hope that robots can someday help increase efficiency and throughput while lowering cost. We report on a 28-question survey posed to the teams to learn about each team's background, mechanism design, perception apparatus, planning, and control approach. We identify trends in this data, correlate it with each team's success in the competition, and discuss observations and lessons learned based on survey results and the authors' personal experiences during the challenge.

We use a conductive fabric substrate as a building material for a soft sensor to extend the functionality of soft actuators. We use PCB etching techniques to apply a pattern to the fabric, yielding distinct conductive surfaces within the same textile. We connect these via flexible wire bus embedded in silicone, terminating in a flexible PCB. We show how touch and metal objects can be localized along the length of the composite fabric strip. We demonstrate an example soft robotic application, by replacing the constraint layer component in a PneuFlex-style soft actuator with the self contained sensing strip. We show that the augmented composite actuator is able to interact with conductive objects in the environment using a capacitive touch sensing with applications in grasping and human-robot interaction.

Deep learning approaches have been used to perform classification in several applications with high-dimensional input data. In this paper, we investigate the potential for deep learning for classifying affective touch on robotic skin in a social setting. Three models are considered, a convolutional neural network, a convolutional-recurrent neural network and an autoencoder-recurrent neural network. These models are evaluated on two publicly available affective touch datasets, and compared with models built to classify the same datasets. The deep learning approaches provide a similar level of accuracy, and allows gestures to be predicted in real-time at a rate of 6 to 9 Hertz. The memory requirements of the models demonstrate that they can be implemented on small, inexpensive microcontrollers, demonstrating that classification can be performed in the skin itself by collocating computing elements with the sensor array.

We describe opportunities and challenges with wireless robotic materials. Robotic materials are multi-functional composites that tightly integrate sensing, actuation, computation and communication to create smart composites that can sense their environment and change their physical properties in an arbitrary programmable manner. Computation and communication in such materials are based on miniature, possibly wireless, devices that are scattered in the material and interface with sensors and actuators inside the material. Whereas routing and processing of information within the material build upon results from the field of sensor networks, robotic materials are pushing the limits of sensor networks in both size (down to the order of microns) and numbers of devices (up to the order of millions). In order to solve the algorithmic and systems challenges of such an approach, which will involve not only computer scientists, but also roboticists, chemists and material scientists, the community requires a common platform --- much like the "Mote" that bootstrapped the widespread adoption of the field of sensor networks --- that is small, provides ample of computation, is equipped with basic networking functionalities, and preferably can be powered wirelessly.

While several sensing modalities and transduction approaches have been developed for tactile sensing in robotic skins, there has been much less work towards extracting features for or identifying high-level gestures performed on the skin. In this paper, we investigate using deep neural networks with hidden Markov models (DNN-HMMs), geometric moments and gesture level features to identify a set of gestures performed on robotic skins. We demonstrate that these features are useful for identifying gestures, and predict a set of gestures from a 14-class dataset with 56% accuracy, and a 7-class dataset with 71% accuracy.

We present an experience-based planning framework called Thunder that learns to reduce computation time required to solve high-dimensional planning problems in varying environments. The approach is especially suited for large configuration spaces that include many invariant constraints, such as those found with whole body humanoid motion planning. Experiences are generated using probabilistic sampling and stored in a sparse roadmap spanner (SPARS), which provides asymptotically near-optimal coverage of the configuration space, making storing, retrieving, and repairing past experiences very efficient with respect to memory and time. The Thunder framework improves upon past experience-based planners by storing experiences in a graph rather than in individual paths, eliminating redundant information, providing more opportunities for path reuse, and providing a theoretical limit to the size of the experience graph. These properties also lead to improved handling of dynamically changing environments, reasoning about optimal paths, and reducing query resolution time. The approach is demonstrated on a 30 degrees of freedom humanoid robot and compared with the Lightning framework, an experience-based planner that uses individual paths to store past experiences. In environments with variable obstacles and stability constraints, experiments show that Thunder is on average an order of magnitude faster than Lightning and planning from scratch. Thunder also uses 98.8% less memory to store its experiences after 10,000 trials when compared to Lightning. Our framework is implemented and freely available in the Open Motion Planning Library.

We present an unsupervised framework for simultaneous appearance-based object discovery, detection, tracking and reconstruction using RGBD cameras and a robot manipulator. The system performs dense 3D simultaneous localization and mapping concurrently with unsupervised object discovery. Putative objects that are spatially and visually coherent are manipulated by the robot to gain additional motion-cues. The robot uses appearance alone, followed by structure and motion cues, to jointly discover, verify, learn and improve models of objects. Induced motion segmentation reinforces learned models which are represented implicitly as 2D and 3D level sets to capture both shape and appearance. We compare three different approaches for appearance-based object discovery and find that a novel form of spatio-temporal super-pixels gives the highest quality candidate object models in terms of precision and recall. Live experiments with a Baxter robot demonstrate a holistic pipeline capable of automatic discovery, verification, detection, tracking and reconstruction of unknown objects.

We present a fiber reinforced soft pneumatic actuator with integrated strain and pressure sensors. We demonstrate that combining these sensors into the same actuator enables proprioception of both actuator curvature and environmental contact forces. We describe the manufacture and integration of a simple liquid metal strain sensor, a pressure sensor, and electrical circuits used for the sensors. We derive a constant curvature model for the actuator which predicts actuator curvature from the air pressure, and other constants of manufacture. The utility of the sensor integration is demonstrated by using the actuator to distinguish successful grasps amongst a set of common cylindrical objects with varying diameter. The grasp radius is estimated from the relationship between the sensor pair. Contact forces with the environment (touch) may be inferred from sensor readings which deviate from unconstrained motion.

Assistive technology (AT) has the ability to improve the standard of living of those with disabilities, however, it can often be abandoned for aesthetic or stigmatizing reasons. Garment-based AT offers novel opportunities to address these issues as it can stay with the user to continuously monitor and convey relevant information, is non-invasive, and can provide aesthetically pleasing alternatives. In an effort to overcome traditional AT and wearable computing challenges including, cumbersome hardware constraints and social acceptability, we present Flutter, a fashion-oriented wearable AT. Flutter seamlessly embeds low-profile networked sensing, computation, and actuation to facilitate sensory augmentation for those with hearing loss. The miniaturized distributed hardware enables both textile integration and new methods to pair fashion with function, as embellishments are functionally leveraged to complement technology integration. Finally, we discuss future applications and broader implications of using such computationally-enabled textile wearables to support sensory augmentation beyond the realm of AT.

We present algorithms for gesture recognition using in-network processing in distributed sensor arrays embedded within systems such as tactile input devices, sensing skins for robotic applications, and smart walls. We describe three distributed gesture-recognition algorithms that are designed to function on sensor arrays with minimal computational power, limited memory, limited bandwidth, and possibly unreliable communication. These constraints cause storage of gesture templates within the system and distributed consensus algorithms for recognizing gestures to be difficult. Building up on a chain vector encoding algorithm commonly used for gesture recognition on a central computer, we approach this problem by dividing the gesture dataset between nodes such that each node has access to the complete dataset via its neighbors. Nodes share gesture information among each other, then each node tries to identify the gesture. In order to distribute the computational load among all nodes, we also investigate an alternative algorithm, in which each node that detects a motion will apply a recognition algorithm to part of the input gesture, then share its data with all other motion nodes. Next, we show that a hybrid algorithm that distributes both computation and template storage can address trade-offs between memory and computational efficiency.

We present an algorithm, analysis, and implementation of a six-channel range and bearing system for swarm robot systems with sizes in the order of centimeters. The proposed approach relies on a custom sensor and receiver model, and collection of intensity signals from all possible sensor/emitter pairs. This allows us to improve range calculation by accounting for orientation-dependent variations in the transmitted intensity, as well as to determine the orientation of the emitting robot. We show how the algorithm and analysis generalize to other range and bearing systems, and evaluate its performance experimentally using two ping-pong ball-sized “Droplets” mounted on a precise gantry system.

We present a soft, amorphous skin that can sense and localize textures. The skin consists of a series of sensing and computing elements that are networked with their local neighbors and mimic the function of the Pacinian corpuscle in human skin. Each sensor node samples a vibration signal at 1 KHz, transforms the signal into the frequency domain, and classifies up to 15 textures using logistic regression. By measuring the power spectrum of the signal and comparing it with its local neighbors, computing elements can then collaboratively estimate the location of the stimulus. The resulting low-bandwidth information, consisting of the texture probability distribution and its location are then routed to a sink anywhere in the skin in a multi-hop fashion. We describe the design, manufacturing, classification, localization and networking algorithms and experimentally validate the proposed approach. In particular, we demonstrate texture classification with 71% accuracy and centimeter accuracy in localization over an area of approximately three square feet using ten networked sensor nodes.

We present a method for system identification of flexible objects by measuring forces and displacement during interaction with a manipulating arm. We model the object's structure and flexibility by a chain of rigid bodies connected by torsional springs. Unlike previous work, the proposed optimal control approach using variational integrators allows identification of closed loops, which include the robot arm itself. This allows using the resulting models for planning in configuration space of the robot. In order to solve the resulting problem efficiently, we develop a novel method for fast discrete-time adjoint-based gradient calculation. The feasibility of the approach is demonstrated using full physics simulation in trep and using data recorded from a 7-DOF series elastic robot arm.

We present an omnidirectional powertrain for swarm robotic platforms that relies on low-cost vibration motors.We describe a mechanism and controller to achieve full 3-DoF motion on the plane. The proposed approach does not require the motors to be in phase, and overcomes differences in manufacturing by a hardware-in-the-loop auto-calibration routine based on the Nelder-Mead algorithm, which issues motion commands via infrared and records the resulting trajectories using an off-the-shelf webcam. We show convergence results of the calibration routine and sample trajectories of the swarm robotic platform “Droplet” demonstrating turning and omnidirectional drive.

We describe user interactions with the self-organized amorphous wall, a modular, fully distributed system of computational building blocks that communicate locally for creating smart surfaces and functional room dividers. We describe a menu and a widget-based approach in which functions are color-coded and can be selected by dragging them from module to module on the surface of the wall. We also propose an on-off switch gesture and a dial gesture each spanning multiple units as canonical input mechanisms that are realized in a fully distributed way.

We propose and study the navigation with foraging problem, where an agent with a limited sensor range must simultaneously: (1) navigate to a global goal and (2) forage en route as opportunities to forage are detected. Each foraging act causes a deviation from the shortest path to the long-term goal, with consequences for path length, mission duration, and fuel usage. We analytically calculate and/or bound the expected distance the robot actually travels, given the initial distance to the the global goal. In particular, for either of two non-trivial greedy strategies: (A) forage the point that minimizes goal-heading deviation. (B) forage the closest point ahead of the robot. Our results generalize to problems in higher dimensions.

We present the "Honeycomb", a microcontroller platform that can easily be networked into hexagonal lattices of hundreds of nodes to create novel materials that tightly integrate sensing, actuation, computation and communication. The tool-chain consists of the platforms, a viral boot-loader to virally disseminate programs into the network, a software library that facilitates sensing, control, and communication, and software tools that allow interacting with the network from a host computer. After a brief tutorial, participants will have an opportunity to experiment with the Honeycomb hardware, which will be made available during the studio, write code for distributed processing of sensor information, and drive various actuators ranging from multi-color lights to servo motors, with the goal to construct an interactive installation to be displayed at the conference. All materials, including open source hardware and software will be made available on the web prior to the studio.

C-FOREST is a parallelization framework for single-query sampling-based shortest path-planning algorithms. Multiple search trees are grown in parallel (e.g., 1 per CPU). Each time a better path is found, it is exchanged between trees so that all trees can benefit from its data. Specifically, the path's nodes increase the other trees' configuration space visibility, while the length of the path is used to prune irrelevant nodes and to avoid sampling from irrelevant portions of the configuration space. Experiments with a robotic team, a manipulator arm, and the alpha benchmark demonstrate that C-FOREST achieves significant superlinear speedup in practice for shortest path-planning problems (team and arm), but not for feasible path panning (alpha).

We present a suite of algorithms that enable a team of mobile robots to repair connectivity in a wireless mesh network. Each robot carries a wireless router and can act as a mobile access point. The algorithms are distributed, with each robot computing it's trajectory using its position, the positions of its neighbors within communication range, and the position of a gateway node. The algorithms are validated via an analytical model as well as field experiments with 7 Create robots.

We wish to design decentralized algorithms for self-assembly of robotic modules that have 100% yield even if the number of available building blocks is limited, and specifically when the number of available building blocks is identical to the number of blocks required by the structure. In contrast to self-assembly at the nano and micro scales where abundant building blocks are available, modular robotic systems need to self-assemble from a limited number of modules. In particular, when self-assembly is used for reconfiguration, it is desirable that the new conformation includes all of the available modules. We propose a suite of algorithms that (1) generate a reversible graph grammar, i.e., generates rules for a desired structure that allow the structure not only to assemble, but also to disassemble, and (2) have a set of structures that are growing in parallel converge to a single structure using broadcast communication. We show that by omitting a reversal rule for the last attached module, self-assembly eventually completes, and that communication can drastically speed up this process.

This paper introduces the Object Interaction Language (OIL) that allows programming and coordination of distributed, heterogeneous sensor-actuator networks, such as sensor networks and multi-robot systems. OIL is an interpreted, object oriented language and is contained in an OIL environment. An OIL environment provides communication between agents and allows agents to exchange code snippets among each other. Possible implementations of OIL environments can be - in the simplest case - a sheet of paper with OIL code literally printed on it, or a computational agent endowed with sensors, actuators and wireless communication. The atomic primitive in OIL is the intent for which implementation is resolved during runtime, potentially using code from other OIL environments and leading to distributed execution. We develop the structure of the language and demonstrate its key properties using a distributed computation task that is parallelized via an OIL environment. We evaluate the algorithm empirically by running OIL code on a team of six computational agents that communicate wirelessly. We then show experimentally how OIL can be used to allocate sensing and mobility in a multi-robot system using a case study in navigation, where one robot dynamically provides laser range data to another robot which is blind to its environment.

We study a fully distributed, reactive algorithm for deployment and maintenance of a mobile communication backbone that provides an area around a network gateway with wireless network access for higher-level agents. Possible applications of such a network are distributed sensor networks as well as communication support for disaster or military operations. The algorithm has minimalist requirements on the individual robotic node and does not require any localization. This makes the proposed solution suitable for deployment of large numbers of comparably cheap mobile communication nodes and as a backup solution for more capable systems in GPS-denied environments. Robots keep exploring the configuration space by random walk and stop only if their current location satisfies user-specified constraints on connectivity (number of neighbors). Resulting deployments are robust and convergence is analyzed using both kinematic simulation with a simplified collision and communication model as well as a probabilistic macroscopic model. The approach is validated on a team of 9 iRobot Create robots carrying wireless access points in an indoor environment.

This paper describes the architecture and implementation of a distributed autonomous gardening system. The garden is a mesh network of robots and plants. The gardening robots are mobile manipulators with an eye-in-hand camera. They are capable of locating plants in the garden, watering them, and locating and grasping fruit. The plants are potted cherry tomatoes enhanced with sensors and computation to monitor their well-being (e.g. soil humidity, state of fruits) and with networking to communicate servicing requests to the robots. Task allocation, sensing and manipulation are distributed in the system and de-centrally coordinated. We describe the architecture of this system and present experimental results for navigation, object recognition and manipulation.

This paper presents a methodology for finding optimal control parameters as well as optimal system parameters for robot swarm controllers using probabilistic, population dynamic models. With distributed task allocation as a case study, we show how optimal control parameters leading to a desired steady-state task distribution for two fully-distributed algorithms can be found even if the parameters of the system are unknown. First, a reactive algorithm in which robots change states independently from each other and which leads to a linear macroscopic model describing the dynamics of the system is considered. Second, a threshold-based algorithm where robots change states based on the number of other robots in this state and which leads to a non-linear model is investigated. Whereas analytical results can be obtained for the linear system, the optimization of the non-linear controller is performed numerically. Finally, we show using stochastic simulations that whereas the presented methodology and models work best if the swarm size is large, useful results can already be obtained for team-sizes below a hundred robots. The methodology presented can be applied to scenarios involving the control of large numbers of entities with limited computational and communication abilities as well as a tight energy budget, such as swarms of robots from the centimeter to nanometer range or sensor networks.

Vision-based tracking is used in nearly all robotic laboratories for monitoring and extracting of agent positions, orientations, and trajectories. However, there is currently no accepted standard software solution available, so many research groups resort to developing and using their own custom software. In this paper, we present version 4 of SwisTrack, an open source project for simultaneous tracking of multiple agents. While its broad range of pre-implemented algorithmic components allows it to be used in a variety of experimental applications, its novelty stands in its highly modular architecture. Advanced users can therefore also implement additional customized modules which extend the functionality of the existing components within the provided interface. This paper introduces SwisTrack and shows experiments with both marked and marker-less agents.

The Personal Commute Portal (PCP) is a Web-based traffic information system that provides a good driving direction and personalized route recommendation using historical and real-time traffic data obtained by a vehicular sensor network.

For the multi-robot coverage problem deterministic deliberative as well as probabilistic approaches have been proposed. Whereas deterministic approaches usually provide provable completeness and promise good performance under perfect conditions, probabilistic approaches are more robust to sensor and actuator noise, but completion cannot be guaranteed and performance is sub-optimal in terms of time to completion. In reality, however, almost all deterministic algorithms for robot coordination can be considered probabilistic when considering the unpredictability of real world factors. This paper investigates experimentally and analytically how probabilistic and deterministic algorithms can be combined for maintaining the robustness of probabilistic approaches, and explicitly model the reliability of a robotic platform. Using realistic simulation and data from real robot experiments, we study system performance of a swarm-robotic inspection system at different levels of noise (wheel-slip). The prediction error of a purely deterministic model increases when the assumption of perfect sensors and actuators is violated, whereas a combination of probabilistic and deterministic models provides a better match with experimental data.

Tracking of miniature robotic platforms involves major challenges in image recognition and data association. We present our 2.5 years effort into developing a platform-independent, easy to use, and robust tracking software SwisTrack, which is tailored to research in swarm robotics and behavioral biology. We demonstrate the software and algorithm's abilities using two case studies, tracking of a swarm of cockroaches, and a swarm-robotic inspection task, while outlining hard problems in tracking and data-association of marker-less objects. Its open, platform-independent architecture, and easy-to-use interfaces (Matlab, Java, and C++), allowing for (distributed) post-processing of trajectory data online, make the software highly adaptive to particular research projects without changes to the source code. SwisTrack will be publicly available shortly under the OSI Adaptive License via SourceForge.net.

Tracking of miniature robotic platforms involves major challenges in image recognition and data association. We present our 3-year effort into developing the platform-independent, easy-to-use, and robust tracking software SwisTrack, which is tailored to research in swarm robotics and behavioral biology. We demonstrate the software and algorithms abilities using two case studies, tracking of a swarm of cockroaches, and a swarm-robotic inspection task, while outlining hard problems in tracking and data-association of marker-less objects. Tracking accuracy of a moving robot with respect to camera noise and the calibration model are calculated experimentally. Its open, platform-independent architecture, and easy-to-use interfaces (Matlabtrade, Javatrade, and C++), allowing for (distributed) post-processing of trajectory data online, make the software highly adaptive to particular research projects without changes to the source code. SwisTrack is publicly available on Sourceforge.net under the OSI Adaptive License and contributions from the robotics and biology community are encouraged

We are developing a swarm-intelligent inspection system based on a swarm of autonomous, miniature robots, using only on-board, local sensors. To estimate intrinsic advantages and limitations of the proposed possible distributed control solution, we capture the dynamic of the system at a higher abstraction level using non-spatial probabilistic microscopic and macroscopic models. In a previous publication, we showed that we are able to predict quantitatively the performances of the swarm of robots for a given metric and a beaconless policy. In this paper, after briefly reviewing our modeling methodology, we explore the effect of adding an additional state to the individual robot controller, which allow robots to serve as a beacon for teammates and therefore bias their inspection routes. Results show that this additional complexity helps the swarm of robots to be more efficient in terms of energy consumption but not necessarily in terms of time required to complete the inspection. We also demonstrate that a beacon-based policy introduces a strong coupling among the behavior of robots, coupling which in turn results in nonlinearities at the macroscopic model level.