We present a benchmark for Planning And Reasoning Tasks in humaN-Robot collaboration (PARTNR) designed to study human-robot coordination in household activities. PARTNR tasks exhibit characteristics of everyday tasks, such as spatial, temporal, and heterogeneous agent capability constraints. We employ a semi-automated task generation pipeline using Large Language Models (LLMs), incorporating simulation-in-the-loop for the grounding and verification. PARTNR stands as the largest benchmark of its kind, comprising 100,000 natural language tasks, spanning 60 houses and 5,819 unique objects. We analyze state-of-the-art LLMs on PARTNR tasks, across the axes of planning, perception and skill execution. The analysis reveals significant limitations in SoTA models, such as poor coordination and failures in task tracking and recovery from errors. When LLMs are paired with 'real' humans, they require 1.5x as many steps as two humans collaborating and 1.1x more steps than a single human, underscoring the potential for improvement in these models. We further show that fine-tuning smaller LLMs with planning data can achieve performance on par with models 9 times larger, while being 8.6x faster at inference. Overall, PARTNR highlights significant challenges facing collaborative embodied agents and aims to drive research in this direction.

Imaging spectrometers are valuable instruments for space exploration, but their large data volumes limit the number of scenes that can be downlinked. Missions could improve science yield by acquiring surplus images and analyzing them onboard the spacecraft. This onboard analysis could generate surficial maps, summarizing scenes in a bandwidth-efficient manner to indicate data cubes that warrant a complete downlink. Additionally, onboard analysis could detect targets of opportunity and trigger immediate automated follow-up measurements by the spacecraft. Here, we report a first step toward these goals with demonstrations of fully automatic hyperspectral scene analysis, feature discovery, and mapping onboard the Earth Observing One (EO-1) spacecraft. We describe a series of overflights in which the spacecraft analyzes a scene and produces summary maps along with lists of salient features for prioritized downlink. The onboard system uses a superpixel endmember detection approach to identify compositionally distinctive features in each image. This procedure suits the limited computing resources of the EO-1 flight processor. It requires very little advance information about the anticipated spectral features, but the resulting surface composition maps agree well with canonical human interpretations. Identical spacecraft commands detect outlier spectral features in multiple scenarios having different constituents and imaging conditions.

Orbital remote sensing provides a powerful way to efficiently survey targets such as the Earth and other planets and moons for features of interest. One such feature of astrobiological relevance is the presence of surface sulfur deposits. These deposits have been observed to be associated with microbial activity at the Borup Fiord glacial springs in Canada, a location that may provide an analogue to other icy environments such as Europa. This article evaluates automated classifiers for detecting sulfur in remote sensing observations by the hyperion spectrometer on the EO-1 spacecraft. We determined that a data-driven machine learning solution was needed because the sulfur could not be detected by simply matching observations to sulfur lab spectra. We also evaluated several methods (manual and automated) for identifying the most relevant attributes (spectral wavelengths) needed for successful sulfur detection. Our findings include (1) the Borup Fiord sulfur deposits were best modeled as containing two sub-populations: sulfur on ice and sulfur on rock; (2) as expected, classifiers using Gaussian kernels outperformed those based on linear kernels, and should be adopted when onboard computational constraints permit; and (3) Recursive Feature Elimination selected sensible and effective features for use in the computationally constrained environment onboard EO-1. This study helped guide the selection of algorithm parameters and configuration for the classification system currently operational on EO-1. Finally, we discuss implications for a similar onboard classification system for a future Europa orbiter.

We describe efforts to apply sensorweb technologies to the monitoring of forest fires in Thailand. In this approach, satellite data and ground reports are assimilated to assess the current state of the forest system in terms of forest fire risk, active fires, and likely progression of fires and smoke plumes. This current and projected assessment can then be used to actively direct sensors and assets to best acquire further information. This process operates continually with new data updating models of fire activity leading to further sensing and updating of models. As the fire activity is tracked, products such as active fire maps, burn scar severity maps, and alerts are automatically delivered to relevant parties. We describe the current state of the Thailand Fire Sensorweb which utilizes the MODIS-based FIRMS system to track active fires and trigger Earth Observing One / Advanced Land Imager to acquire imagery and produce active fire maps, burn scar severity maps, and alerts. We describe ongoing work to integrate additional sensor sources and generate additional products.

We describe efforts to integrate in-situ sensing, space-borne sensing, hydrological modeling, active control of sensing, and automatic data product generation to enhance monitoring and management of flooding. In our approach, broad coverage sensors and missions such as MODIS, TRMM, and weather satellite information and in-situ weather and river gauging information are all inputs to track flooding via river basin and sub-basin hydrological models. While these inputs can provide significant information as to the major flooding, targetable space measurements can provide better spatial resolution measurements of flooding extent. In order to leverage such assets we automatically task observations in response to automated analysis indications of major flooding. These new measurements are automatically processed and assimilated with the other flooding data. We describe our ongoing efforts to deploy this system to track major flooding events in Thailand.

Autonomous onboard processing of data allows rapid response to detections of dynamic, changing processes. Software that can detect volcanic eruptions from thermal emission has been used to retask the Earth Observing 1 spacecraft to obtain additional data of the eruption. Rapid transmission of these data to the ground, and the automatic processing of the data to generated images, estimates of eruption parameters and maps of thermal structure, has allowed these products to be delivered rapidly to volcanologists to aid them in assessing eruption risk and hazard. Such applications will enhance science return from future Earth-orbiting spacecraft and also from spacecraft exploring the Solar System, or beyond, which hope to image dynamic processes. Especially in the latter case, long communication times between the spacecraft and Earth exclude a rapid response to what may be a transient process - only using onboard autonomy can the spacecraft react quickly to such an event.

This paper describes work being performed under a NASA Earth Science Technology Office grant to develop a modular Sensor Web architecture based on Open Geospatial Consortium (OGC) standards, which enables discovery and generic tasking capability for sensors, both space-based and insitu. A series of increasingly complex demonstrations have been developed to prototype this architecture. Recent demonstrations have made use of the Hyperion and Advanced Land Imager instruments on the Earth Observing 1 (EO-1) satellite, the Moderate Resolution Imaging Spectroradiometer (MODIS) on the Terra and Aqua, the Advanced Space-borne Thermal Emission and Reflection Radiometer (ASTER) on the Terra satellite and the Wildfire sensor on the Ikhana Unmanned Aerial System (UAS). This Sensor Web was used in the recent Southern California fires during October 2007 to deliver key wildfire imagery to the San Diego county Emergency Operations Center (EOC) to assist emergency workers with situational awareness. Presently the team is in the process of prototyping the use of this sensor web for floods in collaboration with the International Federations of the Red Cross/Red Crescent for better flood disaster management. The paper will also describe a general overview of the modular architecture that has thus far been built and capabilities still needed to realize the full vision.

We describe a network of sensors linked by software and the Internet to an autonomous satellite observation response capability. This system of systems is designed with a flexible, modular, architecture to facilitate expansion in sensors, customization of trigger conditions, and customization of responses. This system has been used to implement a global surveillance program of science phenomena including: volcanoes, flooding, cryosphere events, and atmospheric phenomena. In this paper we describe the importance of the earth observing sensorweb application as well as overall architecture for the system of systems.

An Autonomous Science Agent has been flying onboard the Earth Observing One spacecraft since 2003. This software enables the spacecraft to autonomously detect and responds to science events occurring on the Earth such as volcanoes, flooding, and snow melt. This agent includes artificial intelligence software systems that perform science data analysis, deliberative planning, and run-time robust execution. This software is in routine use to fly the EO-1 mission. In this paper we discuss the architecture used to integrate these systems and lessons learned from its multi-year flight on EO-1.