Chengkun Cai, Xu Zhao, Haoliang Liu, Zhongyu Jiang, Tianfang Zhang, Zongkai Wu, Jenq-Neng Hwang, Lei Li. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.

This paper presents LLaMo (Large Language and Human Motion Assistant), a multimodal framework for human motion instruction tuning. In contrast to conventional instruction-tuning approaches that convert non-linguistic inputs, such as video or motion sequences, into language tokens, LLaMo retains motion in its native form for instruction tuning. This method preserves motion-specific details that are often diminished in tokenization, thereby improving the model’s ability to interpret complex human behaviors. By processing both video and motion data alongside textual inputs, LLaMo enables a flexible, human-centric analysis. Experimental evaluations across high-complexity domains, including human behaviors and professional activities, indicate that LLaMo effectively captures domain-specific knowledge, enhancing comprehension and prediction in motion-intensive scenarios. We hope LLaMo offers a foundation for future multimodal AI systems with broad applications, from sports analytics to behavioral prediction.

Spatial intelligence is fundamental to AI systems that interact with the physical world, particularly in 3D scene generation and spatial comprehension. Current layout generation in 3D scene synthesis remains highly complex, often constrained by predefined datasets and limited dynamic adaptation to changing spatial relationships. In this paper, we propose GraphCanvas3D, a flexible, query-driven framework for controllable 3D scene generation. Unlike traditional methods that require retraining and predefined input masks for modifications, GraphCanvas3D provides a training-free solution supporting the generation of diverse scenes-both indoor and outdoor-through free manipulation of objects and scene elements. Our framework employs hierarchical, graph-driven scene descriptions, representing spatial elements as graph nodes and establishing coherent relationships among objects in 3D environments. The decoupled object representation enables flexible, on-the-fly scene adjustments and dynamic, customizable scene creation. Experimental results and user studies demonstrate that GraphCanvas3D improves usability, adaptability, and generalization across various 3D scene generation tasks, offering a powerful tool for scalable and diverse scene synthesis.

Estimating 3D full-body pose from sparse sensor data is a pivotal technique employed for the reconstruction of realistic human motions in Augmented Reality and Virtual Reality. However, translating sparse sensor signals into comprehensive human motion remains a challenge since the sparsely distributed sensors in common VR systems fail to capture the motion of full human body. In this paper, we use well-designed Body Pose Graph (BPG) to represent the human body and translate the challenge into a prediction problem of graph missing nodes. Then, we propose a novel full-body motion reconstruction framework based on BPG. To establish BPG, nodes are initially endowed with features extracted from sparse sensor signals. Features from identifiable joint nodes across diverse sensors are amalgamated and processed from both temporal and spatial perspectives. Temporal dynamics are captured using the Temporal Pyramid Structure, while spatial relations in joint movements inform the spatial attributes. The resultant features serve as the foundational elements of the BPG nodes. To further refine the BPG, node features are updated through a graph neural network that incorporates edge reflecting varying joint relations. Our method's effectiveness is evidenced by the attained state-of-the-art performance, particularly in lower body motion, outperforming other baseline methods. Additionally, an ablation study validates the efficacy of each module in our proposed framework.

The Vision-and-Language Navigation in Continuous Environments (VLN-CE) task requires an agent to follow a language instruction in a realistic environment. Understanding the environment is crucial, yet current methods are relatively simple and direct, without delving into the interplay between language instructions and visual context. Therefore, we propose a novel environment representation. First, we propose an Environment Representation Graph (ERG) through object detection to express the environment in semantic level. Then, relational representations of object-object and object-agent in the ERG are learned through Graph Convolution Network (GCN), so as to obtain a continuous ERG expression. Sequentially, we combine the ERG expression with object label embeddings to obtain the environment representation. Finally, a new cross-modal attention navigation framework is proposed, incorporating our environment representation and a specialized loss function for ERG training. Experimental results demonstrate the effectiveness of our approach in achieving commendable performance on VLN-CE tasks.

Vision-and-language navigation (VLN) is a challenging task that requires an agent to navigate in real-world environments by understanding natural language instructions and visual information received in real time. Prior works have implemented VLN tasks on continuous environments or physical robots, all of which use a fixed-camera configuration due to the limitations of datasets, such as 1.5-m height, 90° horizontal field of view (HFOV), and so on. However, real-life robots with different purposes have multiple camera configurations, and the huge gap in visual information makes it difficult to directly transfer the learned navigation skills between various robots. In this brief, we propose a visual perception generalization strategy based on meta-learning, which enables the agent to fast adapt to a new camera configuration. In the training phase, we first locate the generalization problem to the visual perception module and then compare two meta-learning algorithms for better generalization in seen and unseen environments. One of them uses the model-agnostic meta-learning (MAML) algorithm that requires few-shot adaptation, and the other refers to a metric-based meta-learning method with a feature-wise affine transformation (AT) layer. The experimental results on the VLN-CE dataset demonstrate that our strategy successfully adapts the learned navigation skills to new camera configurations, and the two algorithms show their advantages in seen and unseen environments respectively.

The vision-and-language navigation (VLN) task requires the agent to navigate by following both natural language instructions and vision. Prior works focus on improving the navigation model or adopting data augmentation to enhance the accuracy and efficiency of navigation. However, few attempts have been made to optimize the training process. In this paper, we propose a cluster-based curriculum learning (CCL) method to schedule appropriate curricula to acquire a more rational learning process. Our CCL dynamically segments the tasks in training dataset into three subsets: easy, medium and hard, according to whether tasks can be successfully completed by the present navigation model. Meanwhile, we propose a dynamic data extraction method based on the volume of subsets to adjust the priority of training from easy to hard. On the other hand, the existing metrics are hard to show the possibility that the agent will complete the task exactly along the expected path. We propose a new metric named path success rate (PSR), where a path is defined as success when the agent passes by all locations described by the instruction. Extensive experiments are conducted to demonstrate the effectiveness and superiority of our proposed CCL on VLN tasks.

Vision-and-language navigation (VLN) based on human-robot interaction has become increasingly attractive in recent years for flexible robot navigation. As key metrics, the success rate and path length are equally important in VLN, where the former represents the accuracy of navigation and the latter indicates the efficiency. However, the most current approaches aim to improve the accuracy of navigation while rarely consider the path efficiency. For this situation, we propose a progress predictor module, where the progress predictor is used to predict the future navigation progress to be attained by taking action. Using the progress predictor, we design a multi-grounding navigator (MGN) to generate the distribution of actions by adding the predicted future progress for improving the accuracy and efficiency simultaneously. After action distribution is given, a path-shortening (PS) method is presented to further improve the path efficiency. On the other hand, by considering an extra progress monitor, we design a self-supervised (SS) module to further improve the navigation performance. Experiments using Room-to-Room (R2R) dataset are conducted to demonstrate the effectiveness of our MGN with SS and/or PS. Experiment results show that our scheme enhances accuracy and efficiency than baselines, especially in unseen environments.