This paper addresses rock hazard detection for in-situ resource utilization (ISRU) robotic navigation in the challenging visual environment of the lunar south pole (LSP). We evaluate three state-of-the-art instance segmentation mod-els-Mask R-CNN, YOLOv8, and SAM-using a novel, synthetically generated dataset that simulates LSP-specific illumination challenges at sun angles of$2.5^{\circ}, 5^{\circ}$, and 7.5°. Additionally, we evaluate these approaches in both up and downsun driving with low solar angle light. This study highlights the potential of deep learning-based approaches for improving ISRU operations by reliably identifying visual surface hazards, such as rocks, which may impede robotic navigation and excavation in future lunar missions.

To support sustainable infrastructure on the Moon, NASA is developing the In-Situ Resource Utilization (ISRU) Pilot Excavator (IPEx) to extract and transport lunar regolith for processing and construction. During its mission, IPEx will execute various driving patterns, primarily cycling between excavation and unloading sites, with additional ma-neuvers such as circular traverses around the lander and raster scans for environmental mapping. In this work, dynamic move-ment primitives (DMPs) are used to represent these patterns. We augment the DMPs with a vision-based real-time obstacle avoidance system to navigate surface hazards, such as rocks, encountered during traversal. Our approach is evaluated in a high-fidelity simulation replicating the challenging environment of the lunar south pole to demonstrate IPEx's ability to adapt to surface hazards while fulfilling its operational tasks.

As robots become increasingly prominent in di-verse industrial settings, the desire for an accessible and reliable system has correspondingly increased. Yet, the task of meaningfully assessing the feasibility of introducing a new robotic component, or adding more robots into an existing infrastructure, remains a challenge. This is due to both the logistics of acquiring a robot and the need for expert knowledge in setting it up. In this paper, we address these concerns by developing a purely virtual simulation of a robotic system. Our proposed framework enables natural human-robot interaction through a visually immersive representation of the workspace. The main advantages of our approach are the following: (i) independence from a physical system, (ii) flexibility in defining the workspace and robotic tasks, and (iii) an intuitive interaction between the operator and the simulated environment. Not only does our system provide an enhanced understanding of 3D space to the operator, but it also encourages a hands-on way to perform robot programming. We evaluate the effectiveness of our method in applying novel automation assignments by training a robot in virtual reality and then executing the task on a real robot.

To support sustainable infrastructure on the Moon, NASA must leverage robots to extract lunar resources for in-situ processing and construction. As part of this effort, NASA is launching the in-situ resource utilization (ISRU) Pilot Excavator later this decade to validate a robotic regolith excavator based on the Regolith Advanced Surface Systems Operations Robot (RASSOR). RASSOR is designed to extract and transport regolith to meet the needs of ISRU architectures. During its mission, Pilot Excavator will be tasked with driving in test patterns to demonstrate the operational concept. One possible test pattern is a circular trajectory around the lander while avoiding surface hazards such as lunar rocks. To this end, we utilize dynamic movement primitives to represent navigation sequences as primitive trajectories. We introduce a novel obstacle avoidance parameter, which is configured to avoid rocks throughout testing exercises. We demonstrate the effectiveness our method in a newly developed simulation tool called the Simulated Excavation Environment for Lunar Operations (SEELO) using models based on the NASA RASSOR 2.0 excavator. Our results show that the robot is able to safety and robustly navigate the lunar surface with densely populated rock obstacles while retaining the desired circle pattern behavior.

In this paper, we develop a novel and safe control design approach that takes demonstrations provided by a human teacher to enable a robot to accomplish complex manipulation scenarios in dynamic environments. First, an overall task is divided into multiple simpler subtasks that are more appropriate for learning and control objectives. Then, by collecting human demonstrations, the subtasks that require robot movement are modeled by probabilistic movement primitives (ProMPs). We also study two strategies for modifying the ProMPs to avoid collisions with environmental obstacles. Finally, we introduce a rule-base control technique by utilizing a finite-state machine along with a unique means of control design for ProMPs. For the ProMP controller, we propose control barrier and Lyapunov functions to guide the system along a trajectory within the distribution defined by a ProMP while guaranteeing that the system state never leaves more than a desired distance from the distribution mean. This allows for better performance on nonlinear systems and offers solid stability and known bounds on the system state. A series of simulations and experimental studies demonstrate the efficacy of our approach and show that it can run in real time.Note to Practitioners—This paper is motivated by the need to create a teach-by-demonstration framework that captures the strengths of movement primitives and verifiable, safe control. We provide a framework that learns safe control laws from a probability distribution of robot trajectories through the use of advanced nonlinear control that incorporates safety constraints. Typically, such distributions are stochastic, making it difficult to offer any guarantees on safe operation. Our approach ensures that the distribution of allowed robot trajectories is within an envelope of safety and allows for robust operation of a robot. Furthermore, using our framework various probability distributions can be combined to represent complex scenarios in the environment. It will benefit practitioners by making it substantially easier to test and deploy accurate, efficient, and safe robots in complex real-world scenarios. The approach is currently limited to scenarios involving static obstacles, with dynamic obstacle avoidance an avenue of future effort.