This work is motivated by the complexity of conventional robot programming methods, which require a deep understanding of both robotics and software development. This situation presents significant challenges for nontechnical end-users who need to program a new robotic task. To address this issue, we propose a new robot programming method that enables humans to specify robot motion tasks using simple and intuitive body-language commands. To this end, we propose a triadic task decomposition framework to decompose complex robot motion tasks into three basic task units, i.e., location, operation, and motion. These units are paired with a set of body-language cues that enable end-users to specify a variety of actions and motions, providing an intuitive alternative to traditional robot programming methods. To program these units, we develop a multimodal fusion-based instruction-parsing algorithm with a dual-phase visual architecture, enabling temporally aware body-language recognition and high-accuracy cross-modal object localization. Furthermore, a semantics-adaptive task execution system is proposed, which incorporates an intent reasoning module that dynamically handles end-users' nonstandard programming sequences and incomplete task logic, enhancing programming success rates and system usability. We evaluate the proposed method on a dual-arm robotic platform and a humanoid robot through various pick-and-place tasks, and compare its performance with that of several baselines from the literature.

We introduce a novel framework with strong generalization capabilities for enabling humanoid robots to perform semantically grounded, context-aware, and physically realizable expressive behaviors, with the goal of enhancing end-user engagement in open-world human–robot interaction (HRI). To achieve this goal, we develop a new large language model–based human cognition module that interprets user dialogs to infer latent intent and generates high-level multi-modal behavior descriptions. These semantic representations are mapped to speech and motion outputs through a dual-stage embodied behavior generation pipeline. The pipeline consists of a shape adaptation module that maps human body motions into the robot’s kinematic space, followed by a motion retargeting module that generates executable joint trajectories under physical constraints. Additionally, the modular architecture enables seamless integration of state-of-the-art generative models and serves as a practical testbed for evaluating expressive behavior generation in real-world settings. We validate this system on a 58-DoF humanoid platform through both controlled video-based studies and live HRI experiments. The results show significant improvements over rule-based and handcrafted baselines in terms of expressiveness, behavioral appeal, and user engagement. This work helps bridging the gap between high-level language understanding and low-level robot control, thereby enabling scalable and human-aligned expressive behavior generation for embodied agents.

Enabling humanoid robots to perform long-horizon mobile manipulation planning in real-world environments based on embodied perception and comprehension abilities has been a longstanding challenge. With the recent rise of large language models (LLMs), there has been a notable increase in the development of LLM-based planners. These approaches either utilize human-provided textual representations of the real world or heavily depend on prompt engineering to extract such representations, lacking the capability to quantitatively understand the environment, such as determining the feasibility of manipulating objects. To address these limitations, we present the Instruction-Augmented Long-Horizon Planning (IALP) system, a novel framework that employs LLMs to generate feasible and optimal actions based on real-time sensor feedback, including grounded knowledge of the environment, in a closed-loop interaction. Distinct from prior works, our approach augments user instructions into PDDL problems by leveraging both the abstract reasoning capabilities of LLMs and grounding mechanisms. By conducting various real-world long-horizon tasks, each consisting of seven distinct manipulatory skills, our results demonstrate that the IALP system can efficiently solve these tasks with an average success rate exceeding 80%. Our proposed method can operate as a high-level planner, equipping robots with substantial autonomy in unstructured environments through the utilization of multi-modal sensor inputs.