Navigating and understanding complex environments over extended periods of time is a significant challenge for robots. People interacting with the robot may want to ask questions like where something happened, when it occurred, or how long ago it took place, which would require the robot to reason over a long history of their deployment. To address this problem, we introduce a Retrieval-augmented Memory for Embodied Robots, or ReMEmbR, a system designed for long-horizon video question answering for robot navigation. To evaluate ReMEmbR, we introduce the NaVQA dataset where we annotate spatial, temporal, and descriptive questions to long-horizon robot navigation videos. ReMEmbR employs a structured approach involving a memory building and a querying phase, leveraging temporal information, spatial information, and images to efficiently handle continuously growing robot histories. Our experiments demonstrate that ReMEmbR outperforms LLM and VLM baselines, allowing ReMEmbR to achieve effective long-horizon reasoning with low latency. Additionally, we deploy ReMEmbR on a robot and show that our approach can handle diverse queries. The dataset, code, videos, and other material can be found at the following link: https://nvidia-ai-iot.github.io/remembr.

General-purpose navigation in challenging environments remains a significant problem in robotics, with current state-of-the-art approaches facing myriad limitations. Classical approaches struggle with cluttered settings and require extensive tuning, while learning-based methods face difficulties generalizing to out-of-distribution environments. This paper introduces X-Mobility, an end-to-end generalizable navigation model that overcomes existing challenges by leveraging three key ideas. First, X-Mobility employs an auto-regressive world modeling architecture with a latent state space to capture world dynamics. Second, a diverse set of multi-head decoders enables the model to learn a rich state representation that correlates strongly with effective navigation skills. Third, by decoupling world modeling from action policy, our architecture can train effectively on a variety of data sources, both with and without expert policies-off-policy data allows the model to learn world dynamics, while on-policy data with supervisory control enables optimal action policy learning. Through extensive experiments, we demonstrate that X-Mobility not only generalizes effectively but also surpasses current state-of-the-art navigation approaches. Additionally, X-Mobility also achieves zero-shot Sim2Real transferability and shows strong potential for crossembodiment generalization. Project page: https://nvlabs.github.io/X-MOBILITY.

In this paper, we propose a novel walking method for torque controlled robots. The method is able to produce a wide range of speeds without requiring off-line optimizations and re-tuning of parameters. We use a quadratic whole-body optimization method running online which generates joint torques, given desired Cartesian accelerations of center of mass and feet. Using a dynamics model of the robot inside this optimizer, we ensure both compliance and tracking, required for fast locomotion. We have designed a foot-step planner that uses a linear inverted pendulum as simplified robot internal model. This planner is formulated as a quadratic convex problem which optimizes future steps of the robot. Fast libraries help us performing these calculations online. With very few parameters to tune and no perception, our method shows notable robustness against strong external pushes, relatively large terrain variations, internal noises, model errors and also delayed communication.

We present a modular controller for quadruped locomotion over unperceived rough terrain. Our approach is based on a computational Central Pattern Generator (CPG) model implemented as coupled nonlinear oscillators. Stumbling correction reflex is implemented as a sensory feedback mechanism affecting the CPG. We augment the outputs of the CPG with virtual model control torques responsible for posture control. The control strategy is validated on a 3D forward dynamics simulated quadruped robot platform of about the size and weight of a cat. To demonstrate the capabilities of the proposed approach, we perform locomotion over unperceived uneven terrain and slopes, as well as situations facing external pushes.

In this paper, a method is proposed for controlling a hopping monopod. It takes dynamics of the robot into account to have better nominal tracking of desired trajectories and more compliant environmental interactions at the same time. We have incorporated also natural dynamics of the robot into the system by using off-line gaits extracted from optimizations on energy. The main control loop consists of the projected inverse dynamics that generates actuator torques given desired trajectories and also a feedback loop designed and tuned specifically for the structure of the robot. A trajectory generator uses known optimal trajectories together with some stabilizing control laws that modify these trajectories to have better robustness in different situations. The average speed of the robot is also regulated by means of a self-organizing controller. We apply soft transitions in trajectories from phase to phase to avoid sharp actuator input profiles. Our method is successfully tested on a monopod hopper robot in simulation. It can handle slightly rough or sloped terrains while maintaining a given average speed. Simulation results suggest that our method is a promising candidate to control a real robot under construction.

The design of efficient locomotion gaits for robots with many degrees of freedom is challenging and time consuming even if optimization techniques are applied. Control parameters can be found through optimization in two ways: (i) through online optimization where the performance of a robot is measured while trying different control parameters on the actual hardware and (ii) through offline optimization by simulating the robot's behavior with the help of models of the robot and its environment. In this paper, we present a hybrid optimization method that combines the best properties of online and offline optimization to efficiently find locomotion gaits for arbitrary structures. In comparison to pure online optimization, both the number of experiments using robotic hardware as well as the total time required for finding efficient locomotion gaits get highly reduced by running the major part of the optimization process in simulation using a cluster of processors. The presented example shows that even for robots with a low number of degrees of freedom the time required for optimization can be reduced by a factor of 2.5 to 30, at least, depending on how extensive the search for optimized control parameters should be. Time for hardware experiments becomes minimal. More importantly, gaits that can possibly damage the robotic hardware can be filtered before being tried in hardware. Yet in contrast to pure offline optimization, we reach well matched behavior that allows a direct transfer of locomotion gaits from simulation to hardware. This is because through a meta-optimization we adapt not only the locomotion parameters but also the parameters for simulation models of the robot and environment allowing for a good matching of the robot behavior in simulation and hardware. We validate the proposed hybrid optimization method on a structure composed of two Roombots modules with a total number of six degrees of freedom. Roombots are self-reconfigurable modular robots that can form arbitrary structures with many degrees of freedom through an integrated active connection mechanism.

Modular robots offer the possibility to quickly design robots with a high diversity of shapes and functionalities. This nice feature also brings an important challenge: namely how to design efficient locomotion gaits for arbitrary robot structures with many degrees of freedom. In this paper, we present a framework that allows one to explore and identify highly different gaits for a given arbitrary-shaped modular robot. We use simulated robots made of several Roombots modules that have three degrees of freedom each. These modules have the interesting feature that they can produce both oscillatory movements (i.e. periodic movements around a rest position) and rotational movements (i.e. with continuously increasing angle), leading to rich locomotion patterns. Here we ask ourselves which types of movements - purely oscillatory, purely rotational, or a combination of both- lead to the fastest gaits. To address this question we designed a control architecture based on a distributed system of coupled phase oscillators that can produce synchronized rotations and oscillations in many degrees of freedom. We also designed a specific optimization algorithm that can automatically design hybrid controllers, i.e. controllers that use oscillations in some joints and rotations in others. The proposed framework is verified by multiple simulations for several robot morphologies. The results show that (i) the question whether it is better to oscillate or to rotate depends on the morphology of the robot, and that in general it is best to do both, (ii) the optimization framework can successfully generate hybrid controllers that outperform purely oscillatory and purely rotational ones, and (iii) the resulting gaits are fast, innovative, and would have been hard to design by hand.