Imitation learning by watching humans offers a promising path to learning general-purpose robot skills with intuitive task specifications. While prior approaches for video-conditioned imitation learning directly extract skill embeddings from unstructured human videos and follow demonstrations step-by-step, such paradigm usually falls short in generalizing to unseen long-horizon tasks with a single human prompt video due to the significant embodiment and environment gap. To this end, our key insight is to infer local intentions from videos in order to retrieve robot skill memories from prior experience, and conversely select the feasible video clip to follow based on robot observations. Motivated by this, we introduce AdaMimic, a hierarchical imitation learning method that learns the bidirectional mapping of cross-domain sensorimotor skills and derives skill-based policy conditioned on adaptable latent plans. To enable generalization to unseen tasks given cross-domain human videos, AdaMimic leverages task-agnostic play data for interaction-aware skill embedding extraction and video-robot trajectory pairs for semantic and temporal human-to-robot skill alignment. In addition, our method exploits a skill adapter for robot-to-human alignment to adaptively align the robot with the skill intentions. We systematically evaluate AdaMimic on both simulated and real-world kitchen domains, demonstrating AdaMimic’s superiority over prior imitation learning methods in generalizing to novel long-horizon tasks with a single human prompt video.

For a general-purpose robot, it is desirable to imitate human demonstration videos that can effectively solve long-horizon tasks and perform novel ones. Recent advances in skill-based imitation learning have shown that extracting skill embedding from raw human videos is a promising paradigm to enable robots to cope with long-horizon tasks. However, generalization to unseen tasks in a different domain with a human prompt video poses a significant challenge due to the big embodiment and environment difference. To this end, we present Hierarchical Human-to-Robot Imitation Learning (H2RIL) that learns the mapping of cross-domain sensorimotor skills and utilizes it to generalize to unseen tasks given a human video in a different environment. To allow for generalizing zero-shot across environments and embodiments, H2RIL leverages task-agnostic play data for low-level policy training and paired human-robot data for both semantic and temporal skill embedding alignment. Extensive experiments in a simulated kitchen environment demonstrate that H2RIL significantly outperforms other prior baselines and is capable of generalizing to composable new tasks and adapting to Out-of-Distribution (OOD) tasks.