Deep reinforcement learning (DRL) methods can solve complex continuous control tasks in simulated environments by taking actions based solely on state observations at each decision point. Because of the dynamics involved, individual snapshots of real-world sensor measurements afford only partial state observability, so it is typical to use a history of observations to improve training and policy performance. Such intertemporal information can be further exploited using a recurrent neural network (RNN) to reduce the dimensionality of the dynamic state representation. However, using RNNs as an internal part of a DRL network presents challenges of its own; and even then, the improvements in resulting policies are usually limited. To address these shortcomings, we propose using gated feature extraction to improve DRL training of real-world robots. Specifically, we use an untrained gated recurrent unit (GRU) to encode a low-dimension representation of the state observation sequence before passing it to the DRL training procedure. In addition to dimensionality reduction, this allows us to unroll the RNN by encoding the observations cumulatively as they are collected, thereby avoiding same-length input requirements, and train the RL network on the raw observations at the current step combined with the GRU-encoding of the preceding steps. Our simulation experiments employ gated feature extraction with the TD3 algorithm. Our results show that the GRU-encoded state observations improve the training speed and execution performance of the TD3 algorithm, improving the learned policies in all 19 test cases, exceeding the maximum achieved reward by over 38% in 8 and doubling the maximum achieved reward in three, while also outperforming a baseline implementation of SAC in 17 out of 19 environments. Moreover, the greatest improvement is seen in real-world experiments, where our approach successfully learns to balance a pendulum as well as a complex quadrupedal locomotion task. In contrast, the standard TD3 algorithm not only does not show any learning progress at all, but also repeatedly damages the hardware.

Deep Reinforcement Learning (DRL) faces challenges bridging the sim-to-real gap to enable real-world applications. In contrast to the simulated environments used in conventional DRL training, real-world systems are non-linear and evolve in an asynchronous fashion; sensors and actuators have limited precision; communication channels are noisy; and many components introduce variable delays. While these issues are known to many researchers, published methods for systematically tackling the problem of DRL training under these conditions without using simulation are sparse in the field. To this end, this paper proposes a non-blocking and asynchronous DRL training architecture for non-linear, real-time dynamical systems tailored to handling variable delays. Compared to conventional DRL training, we: (i) decouple the RL loop into separate processes run independently at their own frequencies, (ii) further decouple collection of transition tuples$(s_{t}, at_{t}, s_{t+1})$via asynchronous and independent streaming of both actions and observations, and (iii) mitigate the effects of delays and increase sample efficiency by providing delay-length measurements to the training loop and regular retraining of the DRL network. This allows the action step time to be tuned to find an optimal control frequency for a given system, and handles streamed observations that arrive with random delays and independently of action timing. We demonstrate the efficacy of this architecture with a physical implementations of a commodity-grade swing-up pendulum and a quadrupedal robot. Our architecture achieves the best results balancing the pendulum for almost entire length of the episode, compared to conventional blocking approaches which fail to learn effective policies. Our results show that these techniques scale to more complex tasks such as quadrupedal locomotion.

Aerial Modular Reconfigurable Robots (AMRRs) are scalable systems consisting of rotor modules capable of rearrangement during flight. The potential to dynamically change any shape for a given task poses the question: what arrangements offer the most aerodynamic benefit for the task of flying? Answering this requires understanding how adjacent rotors in various configurations influence each another. Intuitively, aerodynamic models such as momentum theory suggest that close rotor proximity decreases performance due to the upstream rotor flow fields interacting. However, effects such as vortex interaction or viscous flow entrainment (used by the Dyson bladeless fan) may offer benefits not captured by the modelling assumptions of computational analysis or simulation. Thus, this work takes an experimental approach, testing thrust performance of rotors in independent configurations of lines, square lattices, and hexagons with various inter-rotor spacings. It was found that inter-rotor spacing did not significantly change thrust performance, but that hexagonal arrangements outperformed line and grid lattices. Smoke tests indicated that hexagon configurations entrained air in the central cavity resulting in a thrust improvement. An inter-rotor spacing of 1.51 rotor diameters gave the best performance increase, roughly equal to that of an additional rotor. This suggests that by placing rotors in an array of six hollow hexagonal honeycombs, thrust performance could theoretically be increased by up to 27.3 per cent, for no additional mass.

The use of robots, as a social stimulus, provides several advantages over using another animal. In particular, for rat-robot studies, robots can produce social behaviour that is reproducible across trials. In the current work, we outline a framework for rat-robot interaction studies, that consists of a novel rat-sized robot (PiRat), models of robotic behavior, and a position tracking system for both robot and rat. We present the design of the framework, including constraints on autonomy, latency, and control. We pilot tested our framework by individually running the robot rat with eight different rats, first through a habituation stage, and then with PiRat performing two different types of behaviour - avoiding and frequently approaching. We evaluate the performance of the framework on latency and autonomy, and on the ability to influence the behaviour of individual rats. We find that the framework performs well on its constraints, engages some of the rats (according to the number of meetings), and features a control scheme that produces reproducible behaviour in rats. These features represent a first demonstration of a closed-loop rat-robot framework.

Motion is a critical aspect of communication, required to create natural interactions between humans and robots. Robots for the classroom pose several constraints on motion, which make them challenging to design, including maintaining the safety of the child and the robot, responding in a timely fashion, and creating motions that are expressive and not scary. In this paper we present the mechanical design of a social robot and demonstrate that it is capable of safe motion within the proximity of children through analysis and empirical testing of the arms. The robot has a novel mechanical design for its two arms, which include torso-mounted, back-drivable, torque-limited stepper motors. The results suggest that our design succeeds at increasing safety levels while enabling the use of socially acceptable speeds of motion during the interaction. This study implies that the design of robotic agents for social interaction with children should consider the design of mechanical features that enable safe contact between the human and the robot while not limiting the robot to slow motions that would impair the timing of the interaction.

Robots that facilitate touch by children have special requirements in terms of safety and robustness, but little is known about how and when children actually use touch with robots. Tools and techniques are required to sense the variety of children's touch and to interpret the volumes of data generated. This explorative user study investigated children's patterns of touch during game play with a robot. We examined where the children touch the robot and their patterns of touch over time, using a raster-based visualisation of each child's time series of touches, recording patterns of touch across different games and children. We found that children readily engage with the robot, in particular spontaneously touching the robot's hands more than any other area. This user study and the tools developed may aid future designs of robots to autonomously detect when they have been touched.

Pretotyping is a set of techniques, tools, and metrics for gauging the interest in a product, prior to full-scale development [1]. This late breaking report describes a pretotyping case study of an ethnodroid - a robot that functions as an ethnographer - intended to engage with young children and record their learning progress. The central requirement for the project is that the robot will be able to interact socially with children aged 1-6 years in tablet-based tasks. We developed a simple robot made of MDF (thick cardboard), added tablets for the face and torso, and controlled a scripted interaction using Wizard of Oz (WoZ). Children's engagement with the robot was tested in an early learning centre which provided a relatively structured environment (“in the lab”) and at a science fair which provided a relatively unconstrained setting (“in the wild”). The rapid testing revealed distinct effects in the children's attitudes and behaviors in the two user contexts and provided insights into form, sensors and analyses for the design process.

We describe a new configuration of fixed-pitch miniature robot rotorcraft that combines the energetic efficiency of a helicopter and the mechanical simplicity of a quadrotor. The large power required to hover is proportional to the inverse of the rotor radius; thus, for a given diameter footprint, a single large rotor will energetically outperform several smaller rotors within the same boundary. However, smaller rotors are able to respond more quickly than large rotors, which require complex actuation to provide control. Our “triangular quadrotor” configuration uses a single large rotor for lift and three small rotors for control, gaining the benefits of both. The small rotors are canted slightly to also provide the same service as a conventional helicopter's tail rotor. Momentum theory analysis shows that a triangular quadrotor may provide a 20% reduction in required hover power, compared with a quadrotor of the same mass and footprint. This is particularly valuable for flying robots working indoors where maximum rotor size is constrained. Using conventional quadrotor and a triangular quadrotors constructed to be a similar as possible, we demonstrate that the triangular quadrotor uses 15% less power, without optimization. A power efficiency budget is provided, and the influence of drive system efficiency is explored. We present a dynamic model and demonstrate experimentally that the aircraft can be stabilized in flight with simple PID control.

The proposed research effort explores the development of active cells - simple contractile electromechanical units that can be used as the material basis for larger articulable structures. Each cell, which might be considered a “muscle unit”, consists of a contractile Nitinol SMA core with conductive terminals. Large numbers of these cells might be combined and externally powered to change phase, contracting to either articulate with a large strain or increase the stiffness of the ensemble, depending on the cell design. Unlike traditional work in modular robotics, the approach presented here focuses on cells that have a simplistic design and function, are inexpensive to fabricate, and are eventually scalable to sub-millimeter sizes, working towards our vision of robot structures that can be custom-fabricated from large numbers of general cell units, similar to biological structures.

Energetic efficiency is a key limiting factor of hovering UAVs. Equipping a quadrotor with low-friction wheels allows it to exploit efficient rolling locomotion to travel long distances on smooth surfaces - common in human environments. A novel feature of this approach is the use of skateboard steering trucks that use lateral tilt to affect steering. This allows existing quadrotor flight controls for driving without modification to the avionics. In this paper we present turning mechanics for driving along the ground and performance curves for the vehicle rolling over different surfaces. We show experimentally that the rolling range of a commercial off-the-shelf quadrotor greatly exceeds its flying range, despite the small added mass of rolling wheels.

Aerial vehicles are difficult to stabilize, especially when acted upon by external forces. A hovering vehicle interacting with objects and surfaces must be robust to contact forces and torques transmitted to the airframe. These produce coupled dynamics that are distinctly different from those of free flight. While external contact is generally avoided, extending aerial robot functionality to include contact with the environment during flight opens up new and useful areas such as perching, object grasping, and manipulation. These mechanics may be modeled as elastic couplings between the aircraft and the ground, represented by springs in R3×SO(3). We show that proportional derivative and proportional integral derivative (PID) attitude and position controllers that stabilize a rotorcraft in free flight will also stabilize the aircraft during contact for a range of contact displacements and stiffnesses. Simulation of the coupled aircraft dynamics demonstrates stable and unstable modes of the system. We find analytical measures that predict the stability of these systems and consider, in particular, the planar system in which the contact point is directly beneath the rotor. We show through explicit solution of the linearized system that the planar dynamics of the object-helicopter system in vertical, horizontal, and pitch motion around equilibrium remain stable, within a range of contact stiffnesses, under unmodified PID attitude control. Flight experiments with a small-scale PID-stabilized helicopter fitted with a compliant gripper for capturing objects affirm our model's stability predictions.

This paper presents a novel approach for constructing light-weight, low-cost air-deployable sensor modules and miniature unmanned aerial vehicles which consist of printed circuit boards as integrated electronic, structural and aerodynamic lift-producing devices. In this way, each aircraft can be a single-piece, single-manufacturing step assembly, reducing materials, processing and labour costs. This approach allows the fabrication of devices that are sufficiently inexpensive so as to be economic for single-use or disposable applications, such as long-range sensor deployment for environmental monitoring. We present two example low-cost, proof-of-concept aircraft employing electro-aeromechanical structures - rotary-wing and fixed-wing - and demonstrate their flight performance.

The small rotor sizes of quadrotors and multirotors makes them intrinsically less energy efficient than a traditional helicopter with a large single rotor. However, the quadrotor configuration's innate simplicity and inexpensive construction recommends its use in many aerial robotics applications. We present a four-rotor configuration that merges the simplicity of a quadrotor with the energy efficiency of a helicopter, while improving manoeuvering rotor bandwidth. This class of aircraft, called a `Y4' or `triangular quadrotor', consists of a single fixed-pitch main rotor with three smaller rotors on booms that provide both counter-torque and manoeuvering control. Our analysis indicates that a Y4 may provide a 20 per cent reduction in hovering power required, compared with a similarly sized conventional quadrotor. Using a matched pair of quadrotor/triangular quadrotor aircraft, our preliminary experiments show that the test-bed Y4 used 15 per cent less power, without optimisation. We present a dynamic model and demonstrate experimentally that the aircraft can be stabilised in flight with PID control.

This paper reports recent research efforts to advance the functionality of Unmanned Aerial Vehicles (UAVs) beyond passive observation to active interaction with and manipulation of objects. The archetypical aerial manipulation task - grasping objects during flight - is difficult due to the unstable dynamics of rotorcraft and coupled object-aircraft motion. In this paper, we analyze key challenges encountered when lifting a grasped object and transitioning into laden free-flight. We demonstrate that dynamic load disturbances introduced by the load mass will be rejected by a helicopter with PID flight control. We determine stability bounds in which the changing mass-inertia parameters of the system due to the grasped object will not destabilize this flight controller. The conditions under which transient partial contact mechanics of objects resting on a surface will not induce instability are identified. We demonstrate grasping and retrieval of a variety of objects while hovering, without touching the ground, using the Yale Aerial Manipulator testbed.

This video demonstrates a helicopter Unmanned Aerial Vehicle (UAV) research platform for grasping objects while in flight. Typically, helicopters avoid interacting with objects in their surroundings due to the unstable flight dynamics of rotorcraft and coupled mechanics encountered during contact. We introduce the Yale Aerial Manipulator and demonstrate stable grasping of a range of objects both when landed and while hovering. We discuss the platform's underactuated gripper and its contribution to aircraft stability while grasping.

A hovering vehicle interacting with objects and surfaces must be robust to contact forces and torques transmitted to the airframe, which produce coupled dynamics distinctly different from those of free flight. These mechanics may be modeled as elastic couplings between the aircraft and the ground, represented by a 6-DOF spring in ℝ3×SO(3).We show that Proportional Derivative attitude and position controllers that stabilize a rotorcraft in free flight will also stabilize the aircraft during contact for a range of contact displacements and stiffnesses. Simulation of the coupled aircraft dynamics demonstrates stable and unstable modes of the system.

Virtually all quadrotors used in research weigh less than 2 kg, and carry payload measured in hundreds of grams. To be useful platforms for expanded operations, these vehicles must be capable of carrying greater weight. Several obstacles in aerodynamics, design and control must be overcome to enable the construction of larger craft with payloads in excess of 1 kg. We report the key design considerations essential for the construction of heavy quadrotor MAVs and demonstrate a 4 kg quadrotor with 1 kg payload.