Sim2real for robotic manipulation is difficult due to the challenges of simulating complex contacts and generating realistic task distributions. To tackle the latter problem, we introduce ManipGen, which leverages a new class of policies for sim2real transfer: local policies. Locality enables a variety of appealing properties including invariances to absolute robot and object pose, skill ordering, and global scene configuration. We combine these policies with foundation models for vision, language and motion planning and demonstrate SOTA zero-shot performance of our method to Robosuite benchmark tasks in simulation (97 %). We transfer our local policies from simulation to reality and observe they can solve unseen long-horizon manipulation tasks with up to 8 stages with significant pose, object and scene configuration variation. ManipGen outperforms SOTA approaches such as SayCan, Open VLA, LLMTrajGen and VoxPoser across 50 real-world manipulation tasks by 36%, 76%, 62% and 60% respectively. Video results at mihdalal.github.io/manipgen

In the field of deep point cloud understanding, KP-Conv is a unique architecture that uses kernel points to locate convolutional weights in space, instead of relying on Multi-Layer Perceptron (MLP) encodings. While it initially achieved success, it has since been surpassed by recent MLP networks that employ updated designs and training strategies. Building upon the kernel point principle, we present two novel designs: KPConvD (depthwise KP-Conv), a lighter design that enables the use of deeper architectures, and KPConvX, an innovative design that scales the depthwise convolutional weights of KPConvD with kernel attention values. Using KPConvX with a modern architecture and training strategy, we are able to outperform current state-of-the-art approaches on the ScanObjectNN, Scannetv2, and S3DIS datasets. We validate our design choices through ablation studies and release our code and models.

A household robot should be able to navigate to target objects without requiring users to first annotate everything in their home. Most current approaches to object navigation do not test on real robots and rely solely on reconstructed scans of houses and their expensively labeled semantic 3D meshes. In this work, our goal is to build an agent that builds self-supervised models of the world via exploration, the same as a child might - thus we (1) eschew the expense of labeled 3D mesh and (2) enable self-supervised in-situ finetuning in the real world. We identify a strong source of self-supervision (Location Consistency - LocCon) that can train all components of an ObjectNav agent, using unannotated simulated houses. Our key insight is that embodied agents can leverage location consistency as a self-supervision signal - collecting images from different views/angles and applying contrastive learning. We show that our agent can perform competitively in the real world and simulation. Our results also indicate that supervised training with 3D mesh annotations causes models to learn simulation artifacts, which are not transferrable to the real world. In contrast, our LocCon shows the most robust transfer in the real world among the set of models we compare to, and that the real-world performance of all models can be further improved with self-supervised LocCon in-situ training.

We present a method for generating, predicting, and using spatiotemporal occupancy grid maps (SOGM), which embed future semantic information of real dynamic scenes. We present an autolabeling process that creates SOGMs from noisy real navigation data. We use a 3-D–2-D feedforward architecture, trained to predict the future time steps of SOGMs, given 3-D Lidar frames as input. Our pipeline is entirely self-supervised, thus enabling lifelong learning for real robots. The network is composed of a 3-D back-end that extracts rich features and enables the semantic segmentation of the lidar frames, and a 2-D front-end that predicts the future information embedded in the SOGM representation, potentially capturing the complexities and uncertainties of real-world multiagent interactions. We also design a navigation system that uses these predicted SOGMs within planning, after they have been transformed into spatiotemporal risk maps. We verify our navigation system's abilities in simulation, validate it on a real robot, study SOGM predictions on real data in various circumstances, and provide a novel indoor 3-D lidar dataset, collected during our experiments, which includes our automated annotations.

We present a novel method for generating, predicting, and using Spatiotemporal Occupancy Grid Maps (SOGM), which embed future information of dynamic scenes. Our au-tomated generation process creates groundtruth SOGMs from previous navigation data. We build on prior work to annotate lidar points based on their dynamic properties, which are then projected on time-stamped 2D grids: SOGMs. We design a 3D-2D feedforward architecture, trained to predict the future time steps of SOGMs, given 3D lidar frames as input. Our pipeline is entirely self-supervised, thus enabling lifelong learning for robots. The network is composed of a 3D back-end that extracts rich features and enables the semantic segmentation of the lidar frames, and a 2D front-end that predicts the future information embedded in the SOGMs within planning. We also design a navigation pipeline that uses these predicted SOGMs. We provide both quantitative and qualitative insights into the predictions and validate our choices of network design with a comparison to the state of the art and ablation studies.

Offline Reinforcement Learning promises to learn effective policies from previously-collected, static datasets without the need for exploration. However, existing Q-learning and actor-critic based off-policy RL algorithms fail when bootstrapping from out-of-distribution (OOD) actions or states. We hypothesize that a key missing ingredient from the existing methods is a proper treatment of uncertainty in the offline setting. We propose Uncertainty Weighted Actor-Critic (UWAC), an algorithm that detects OOD state-action pairs and down-weights their contribution in the training objectives accordingly. Implementation-wise, we adopt a practical and effective dropout-based uncertainty estimation method that introduces very little overhead over existing RL algorithms. Empirically, we observe that UWAC substantially improves model stability during training. In addition, UWAC out-performs existing offline RL methods on a variety of competitive tasks, and achieves significant performance gains over the state-of-the-art baseline on datasets with sparse demonstrations collected from human experts.

We present a self-supervised learning approach for the semantic segmentation of lidar frames. Our method is used to train a deep point cloud segmentation architecture without any human annotation. The annotation process is automated with the combination of simultaneous localization and mapping (SLAM) and ray-tracing algorithms. By performing multiple navigation sessions in the same environment, we are able to identify permanent structures, such as walls, and disentangle short-term and long-term movable objects, such as people and tables, respectively. New sessions can then be performed using a network trained to predict these semantic labels. We demonstrate the ability of our approach to improve itself over time, from one session to the next. With semantically filtered point clouds, our robot can navigate through more complex scenarios, which, when added to the training pool, help to improve our network predictions. We provide insights into our network predictions and show that our approach can also improve the performances of common localization techniques.

Designing a hummingbird-inspired, at-scale, tail-less flapping-wing micro aerial vehicle (FWMAV) is a challenging task in order to achieve animallike flight performance under the constraints of size, weight, power, and actuation limitations. It is even more challenging to design such a vehicle with a pair of independently controlled wings equipped with a total of only two actuators. This article details a systematic solution for the design optimization and prototyping of such FWMAVs. The proposed solution covers the complete system models and analysis of wing-actuation dynamics, control authorities, body dynamics, mechanical limitations, and electrical constraints. Each subsystem, as well as the overall system, is experimentally validated. This comprehensive approach can facilitate the design of such FWMAVs with different optimization goals. To demonstrate the effectiveness of the proposed approach, in this article, we conduct three different design optimization tasks, which yield three different prototype systems: optimizing the lift-to-weight ratio, optimizing control bandwidth, and optimizing control authority. We first construct a vehicle with an optimized lift-to-weight ratio. Based on it, the other two platforms with different design optimization goals are presented for better flight performance. Flight tests were performed on each prototype to validate their flight performance and design goals. Compared to optimized control bandwidth, we demonstrate that the vehicle with optimized control authorities is significantly more capable in terms of flight performance. It shows sustained stable flight in both hovering and heavy load carrying (more than 60% of the vehicle's total weight), which is hard for the other two platforms to achieve.

Insects and hummingbirds exhibit extraordinary flight performance and can simultaneously master seemingly conflicting goals: stable hovering and aggressive maneuvering, which are unmatched by conventional small scale man-made vehicles. Flapping Wing Micro Air Vehicles (FWMAVs) hold great promise for closing this performance gap. However, design and control of such systems remain challenging. Here, we present an open source high fidelity dynamic simulation for FWMAVs. The simulator serves as a testbed for the design, optimization and flight control of FWMAVs. To validate the simulation, we recreated the at-scale hummingbird robot developed in our lab in the simulation. System identification was performed to obtain the model parameters. Force generation and dynamic response of open-loop and closed loop systems between simulated and experimental flights were compared. The unsteady aerodynamics and the highly nonlinear flight dynamics present challenging control problems for conventional and learning control algorithms such as Reinforcement Learning. The interface of the simulation is fully compatible with OpenAI Gym environment. As a benchmark study, we present a linear controller for hovering stabilization and a Deep Reinforcement Learning control policy for goal-directed maneuvering. Finally, we demonstrate direct simulation-to-real transfer of both control policies onto the physical robot, further demonstrating the fidelity of the simulation.

Biological studies show that hummingbirds can perform extreme aerobatic maneuvers during fast escape. Given a sudden looming visual stimulus at hover, a hummingbird initiates a fast backward translation coupled with a 180-degree yaw turn, which is followed by instant posture stabilization in just under 10 wingbeats. Consider the wingbeat frequency of 40Hz, this aggressive maneuver is carried out in just 0.2 seconds. Inspired by the hummingbirds' near-maximal performance during such extreme maneuvers, we developed a flight control strategy and experimentally demonstrated that such maneuverability can be achieved by an at-scale 12-gram hummingbird robot equipped with just two actuators driving a pair of flapping wings up to 40Hz. The proposed hybrid control policy combines model-based nonlinear control with model-free reinforcement learning. We used the model-based nonlinear control for nominal flight conditions where dynamic models are relatively accurate. During extreme maneuvers when the modeling error becomes unmanageable, we use a model-free reinforcement learning policy trained and optimized in simulation to 'destabilize' the system for peak performance during maneuvering. The hybrid policy manifests a maneuver that is close to that observed in hummingbirds. Direct simulation-to-real transfer is achieved, demonstrating the hummingbird-like fast evasive maneuvers on the at-scale hummingbird robot.

Wings of flying animals can not only generate lift and control torques but also can sense their surroundings. Such dual functions of sensing and actuation coupled in one element are particularly useful for small sized bio-inspired robotic flyers, whose weight, size, and power are under stringent constraint. In this work, we present the first flapping-wing robot using its flapping wings for environmental perception and navigation in tight space, without the need for any visual feedback. As the test platform, we introduce the Purdu Hummingbird, a flapping-wing robot with 17cm wingspan and 12 grams weight, with a pair of 30-40Hz flapping wings driven by only two actuators. By interpreting the wing loading feedback and its variations, the vehicle can detect the presence of environmental changes such as grounds, walls, stairs, obstacles and wind gust. The instantaneous wing loading can be obtained through the measurements and interpretation of the current feedback by the motors that actuate the wings. The effectiveness of the proposed approach is experimentally demonstrated on several challenging flight tasks without vision: terrain following, wall following and going through a narrow corridor. To ensure flight stability, a robust controller was designed for handling unforeseen disturbances during the flight. Sensing and navigating one's environment through actuator loading is a promising method for mobile robots, and it can serve as an alternative or complementary method to visual perception.

In recent years, Deep Reinforcement Learning has made impressive advances in solving several important benchmark problems for sequential decision making. Many control applications use a generic multilayer perceptron (MLP) for non-vision parts of the policy network. In this work, we propose a new neural network architecture for the policy network representation that is simple yet effective. The proposed Structured Control Net (SCN) splits the generic MLP into two separate sub-modules: a nonlinear control module and a linear control module. Intuitively, the nonlinear control is for forward-looking and global control, while the linear control stabilizes the local dynamics around the residual of global control. We hypothesize that this will bring together the benefits of both linear and nonlinear policies: improve training sample efficiency, final episodic reward, and generalization of learned policy, while requiring a smaller network and being generally applicable to different training methods. We validated our hypothesis with competitive results on simulations from OpenAI MuJoCo, Roboschool, Atari, and a custom urban driving environment, with various ablation and generalization tests, trained with multiple black-box and policy gradient training methods. The proposed architecture has the potential to improve upon broader control tasks by incorporating problem specific priors into the architecture. As a case study, we demonstrate much improved performance for locomotion tasks by emulating the biological central pattern generators (CPGs) as the nonlinear part of the architecture.

Unlike conventional aerial vehicles of fixed or rotary wings, realtime on-board attitude estimation of insect or hummingbird scale Flapping Wing Micro Aerial Vehicles (FWMAVs) is very challenging due to the severe instantaneous oscillations (approximately ten times of gravity on our platform) induced by high-frequency wing flapping. In this work, we present a novel sensor fusion algorithm for realtime on-board attitude estimation of FWMAVs. The algorithm is proposed with adaptive model-based compensation for both sensing drift and aerodynamic forces induced by flapping wings. We validated our approach on a 12.5 grams hummingbird robot. The experimental results demonstrated the accuracy, convergence, and robustness of the proposed algorithm.

Flying animals with flapping wings may best exemplify the astonishing ability of natural selection on design optimization by excelling both stability and maneuverability at insect/hummingbird scale. Flapping Wing Micro Air Vehicle (FWMAV) holds great promise in bridging the performance gap between engineering system and their natural counterparts. Designing and constructing such a system is a challenging problem under stringent size, weight and power (SWaP) constraints. In this work, we presented a systematic approach for design optimization and integration for a hummingbird inspired FWMAV. Our formulation covers aspects of actuation, dynamics, flight stability and control, which was validated by experimental data for both rigid and flexible wings, ranging from low to high wing loading. The optimization yields prototypes with onboard sensors, electronics, and computation units. The prototype flaps at 30Hz to 40Hz, with 7.5 to 12 grams of system weight and 12 to 20 grams of maximum lift. Liftoff was demonstrated with added payloads. Flapping wing platforms with different requirements and scales can now be designed and optimized with minor modifications of proposed formulation.

Controlled hovering of motor driven flapping wing micro aerial vehicles (FWMAVs) is challenging due to its limited control authority, large inertia, vibration produced by wing strokes, and limited components accuracy due to fabrication methods. In this work, we present a hummingbird inspired FWMAV with 12 grams of weight and 20 grams of maximum lift. We present its full non-linear dynamic model including the full inertia tensor, non-linear input mapping, and damping effect from flapping counter torques (FCTs) and flapping counter forces (FCFs). We also present a geometric flight controller to ensure exponentially stable and globally exponential attractive properties. We experimentally demonstrated the vehicle lifting off and hover with attitude stabilization.

Achieving resonance in flapping wings has been recognized as one of the most important principles to enhance power efficiency, lift generation, and flight control performance of high-frequency flapping wing micro air vehicles (MAVs). Most work on the development of such vehicles have attempted to achieve wing flapping resonance. However, the theoretical understanding of its effects on the response and energetics of flapping motion has lagged behind, leading to suboptimal design decisions and misinterpretations of experimental results. In this work, we systematically model the dynamics of flapping wing as a forced nonlinear resonant system, using both nonlinear perturbation method and linear approximation approach. We derived an analytic solution for steady-state flapping amplitude, energetics, and characteristic frequencies including natural frequency, damped natural frequency, and peak frequency. Our results showed that both aerodynamic lift and power efficiency are maximized by driving the wing at natural frequency, instead of other frequencies. Interestingly, the flapping velocity is maximized at natural frequency as well, which can lead to an easy experimental approach to identify natural frequency and validate the resonance design. Our models and analysis were validated with both simulations and experiments on ten different wings mounted a direct-motor-drive flapping wing MAV. The result can serve as a systematic design principle and guidance in the interpretations of empirical results.

Achieving resonance in flapping wings has been recognized as one of the most important principles to enhance power efficiency, lift generation, and flight control performance of high-frequency flapping wing micro air vehicles (MAVs). Most of work on the development of such vehicles have attempted to achieve wing flapping resonance. However, the theoretical understanding of its effects on the response and energetics of flapping motion has lagged behind, leading to sub-optimal design decisions and misinterpretations of experimental results. In this work, we systematically model the dynamics of flapping wing as a forced nonlinear resonant system. Using linear approximation approach, we derived analytic solution for steady-state flapping amplitude, energetics, and characteristic frequencies including natural frequency, damped natural frequency, and peak frequency. Our results showed that both aerodynamic lift and power efficiency are maximized by driving the wing at natural frequency, instead of other frequencies. Interestingly, the flapping velocity is maximized at natural frequency as well, which can lead to an easy experimental approach to identify natural frequency and validate the resonance design. Our models and analysis were validated with both simulations and experiments on ten different wings mounted a direct-motor-drive flapping wing MAV. The result can serve as a systematic design principle and guidance in the interpretations of empirical results.

The prominent maneuverability of flapping flight is enabled by rapid and significant changes in aerodynamic forces, which is a result of surprisingly subtle and precise changes of wing kinematics. The high sensitivity of aerodynamic forces to wing kinematic changes demands precise and instantaneous control of the flapping wing trajectories, especially in the presence of various types of uncertainties. In this work, we first present a dynamic model of a pair of direct-motor-driven flapping wings while taking into consideration the parameter uncertainties and external disturbances. We then present an Adaptive Robust Controller (ARC) to achieve robust performance of high-frequency (over 30Hz) instantaneous wing trajectory tracking with onboard feedback. The proposed control algorithm was experimentally validated on a 7.5 gram flapping-wing MAV which showed excellent tracking of various wing trajectories with different amplitude, bias, frequency, and split-cycles. Experimental results on various model wings demonstrated that the ARC can adapt to unknown parameters and show no performance degradation across wings of different geometries. The results of ARC were also compared with those of open-loop and classical PID controllers.

In this study, we present a motor-driven flapping-wing actuator, designed to operate at its resonant frequency using a torsion spring. The wing is driven by a DC motor directly through gear transmission. Linear torsion springs mounted on the load shaft creates restoring torque when the wing is displaced from its mid-stroke position. The actuator dynamics is obtained using system identification. The flapping motion of the wing is achieved by closed-loop motor control, for example, tracking a sinusoidal wave with the frequency tuned to match the resonant frequency of the system. PID and LQR controllers are applied for instantaneous wing kinematics tracking: A PID controller is able to precisely track the trajectory with relatively large control input; on the other hand, a linear quadratic regulator (LQR) achieves large flapping amplitude with small input effort. We show that the mechanism is able to track sinusoidal motions with different amplitude, bias and frequencies with relatively large range by changing the springs, therefore generating roll and pitch torques that can be used for flight control. Then a Hopf oscillator based central pattern generator is also shown to be an alternative trajectory to track. The proposed wing actuation mechanism provides an at-scale wing testing platform for flapping wing micro aerial vehicles.