Navigation is a fundamental skill of agents with visual-motor capabilities. We introduce a Navigation World Model (NWM), a controllable video generation model that predicts future visual observations based on past observations and navigation actions. To capture complex environment dynamics, NWM employs a Conditional Diffusion Transformer (CDiT), trained on a diverse collection of egocentric videos of both human and robotic agents, and scaled up to 1 billion parameters. In familiar environments, NWM can plan navigation trajectories by simulating them and evaluating whether they achieve the desired goal. Unlike supervised navigation policies with fixed behavior, NWM can dynamically incorporate constraints during planning. Experiments demonstrate its effectiveness in planning trajectories from scratch or by ranking trajectories sampled from an external policy. Furthermore, NWM leverages its learned visual priors to imagine trajectories in unfamiliar environments from a single input image, making it a flexible and powerful tool for next-generation navigation systems1.

We train models to predict ego-centric video from human actions (PEVA), given the past video and an action represented by the relative 3D body pose. By conditioning on kinematic pose trajectories, structured by the joint hierarchy of the body, our model learns to simulate how physical human actions shape the environment from a first-person point of view. We train an auto-regressive conditional diffusion transformer on Nymeria, a large-scale dataset of real-world egocentric video and body pose capture. We further design a hierarchical evaluation protocol with increasingly challenging tasks, enabling a comprehensive analysis of the model’s embodied prediction and control abilities. Our work represents an initial attempt to tackle the challenges of modeling complex real-world environments and embodied agent behaviors with video prediction from the perspective of a human.

This paper describes the work being performed under a NASA Earth Science Technology Office (ESTO) Advanced Information System Technology (AIST) grant to develop a modular sensor web architecture which enables discovery and generic tasking capability for sensors, both space-based and in-situ. This effort seeks to demonstrate methods to facilitate interoperability and ease of use of a diverse set of sensors by hiding the details required to obtain sensor data, process the science data and deliver the science products. In particular, Web 2.0 and open geospatial consortium (OGC) Sensor Web Enablement (SWE) standard services are used. Thus, these capabilities serve to facilitate a user-centric approach to Global Earth Observing System of Systems (GEOSS). This work builds on previous sensor web efforts conducted at NASA/GSFC using the Earth Observing 1 (EO-1) and other satellites.