Enabling robotic systems to perform long-horizon manipulation planning in real-world environments based on multimodal embodied perception and comprehension remains a longstanding challenge. Recent advancements in large language models (LLMs) have spurred the development of LLM-based planners; however, these approaches often rely on human-provided textual representations or extensive prompt engineering, lacking the ability to quantitatively interpret the environment. To overcome these limitations, we propose a novel framework that leverages LLMs and vision-language models (VLMs) to perform abstract reasoning and extract task-relevant representations from the environment using grounding mechanisms. To further enhance robotic capabilities, we introduce a systematic approach to constructing robotic skill libraries, enabling efficient generation of feasible and optimal actions. Unlike prior work, our LLM-based task planner reformulates user instructions into Planning Domain Description Language (PDDL) problems and employs Behavior Trees to represent the hierarchical structure of tasks, offering interpretable and modular task execution. Extensive evaluations on diverse real-world long-horizon manipulation tasks demonstrate the effectiveness of the proposed method, achieving an average success rate exceeding 80%. Furthermore, the framework functions as a high-level planner, empowering robots with substantial autonomy in unstructured environments by leveraging multimodal sensor inputs.

In the traditional marine equipment construction process, the material transportation vehicle scheduling method dominated by manual experience has shown great limitations, which is inefficient, costly, wasteful of human resources, and unable to cope with complex and changing scheduling scenarios. The existing scheduling system cannot realize the information interaction and collaborative integration between the physical world and the virtual world, while the digital twin (DT) technology can effectively solve the problem of real-time information interaction and the reinforcement learning (RL) method can cope with dynamic scenarios. Therefore, this paper proposed a DT-driven RL method to solve the marine equipment vehicle scheduling problem. Given the dynamic nature of transportation tasks, the diversity of transported goods, and the optimization characteristics of transportation requirements, a framework for scheduling transportation vehicle operations based on DT is constructed, and a RL-based vehicle scheduling method in a dynamic task environment is proposed. A Markov decision process (MDP) model of the vehicle scheduling process is established to realize one-to-one mapping between information and physical elements. An improved RL method based on Q-learning is proposed to solve the MDP model, and the value function approximation and convergence enhancement methods are applied to optimize the solving process. Finally, a case study is used for example verification to prove the superiority and effectiveness of the proposed method in this paper.Note to Practitioners—The motivation of this paper is to optimize material transportation vehicle scheduling in dynamic task environments and to improve logistics transportation efficiency. Therefore, a DT-based vehicle scheduling method for marine equipment is proposed. Firstly, a framework of vehicle scheduling based on DT is designed to establish a MDP model of the vehicle scheduling process, and the dynamic task characteristics are described by mathematical methods in the design of the elements of the model. A RL-based vehicle scheduling method is proposed. The value function approximation method and the convergence enhancement method of the algorithm are investigated for the characteristics of continuous dynamic action features leading to huge state space and non-convergence of the algorithm. The algorithm performance is verified and analyzed through data validation of actual cases.

Human–robot collaborative assembly is required to comprehensively perceive the working scenarios for the most possible assembly collaborations. Nevertheless, existing works have paid much attention to physical entities (i.e., object detection, pose estimation), while ignores the weight of interactive relationships. This research gap makes it difficult to become aware of the cues for decision-making, especially in a complicated assembly task. Furthermore, inadequate relative position characteristics and indescribable object influence remain quite challenging for visual relationship representation. To overcome these abovementioned gaps, a multimodality scene graph generation approach is proposed to more robustly describe the abstract visual relationships. A novel heat modality is presented to better represent the relative spatial characteristic. Three strategies are developed for adapting different baselines in the multimodality feature encoder module. Experimental results show the generality and superb performance for multimodality scene graph generation tasks in human–robot collaborative assembly scenarios.

In this article, we propose a Swin Transformer and multilevel Feature Consistency based Network (STFC-Net), which is a multilevel cascade stereo matching method to predict the disparity in a coarse-to-fine manner. 1) To alleviate the problem of the limited receptive field of existing convolutional neural network (CNN)-based methods, inspired by the capability of modeling the large-scale dependence of transformer, we adopt a multilevel feature extraction module combining CNN and Swin Transformer to capture long-range context information; a multiscale cascaded cost aggregation module is used to cover different image regions with less memory consumption. 2) To make full use of the hierarchical features, we checked the multilevel left-right feature consistency in an unsupervised manner to improve the disparity accuracy. The experimental results show that our method outperforms some previous CNN methods on the Scene Flow and KITTI datasets with lower computational time complexity. Moreover, it generalizes well in some unknown and challenging real-world scenarios.

Multi-fisheye System has the advantages of sufficient overlap and the ability to capture a complete 360° scene, which is beneficial for the omnidirectional depth estimation task. However, due to the severe distortion of the fisheye images, it is hard for such systems to extract and match features to predict an accurate depth. In this work, on the basis of a multi-fisheye system, we present a novel end-to-end deep learning architecture for omnidirectional depth estimation: 1) to capture the reliable features of the distorted fisheye image, a multi-scale feature extraction and aggregation module is improved, which can adaptively obtain the global context information to represent the features; 2) to leverage more aligned features, especially those in the overlap between multi-fisheye images, we construct a fusion cost volume to combine similarity and semantic information, which can enhance the feature discriminability; and 3) to refine the omnidirectional depth map efficiently, a cascaded cost regularization architecture is proposed. Instead of several costly 3D convolutions, the 3D BSConv based on intra-kernel correlations is introduced to regularize the cost. The proposed method can incrementally predict the depth map from coarse to fine, and reduce the network computational complexity significantly. The experiments in several public indoor and outdoor synthetic datasets demonstrate that the proposed method outperforms some state-of-the-art methods in terms of a synthesis of accuracy and speed, with fewer model parameters.

Fraction of absorbed photosynthetically active radiation (FAPAR) is one of key variables required in modeling primary production and global climate. FAPAR can be derived from satellite images, for example the MODIS FAPAR. However, validation of the product is a prerequisite for using it to estimate local net primary production (NPP). For this purpose, we carried out in situ measurements of FAPAR in two 2km×2km areas within the temperate meadow-steppe grassland in Hulunber during growing season in 2008. The MODIS FAPAR product reflected very well the seasonal dynamics of in situ FAPAR, but tended to overestimate the value with averaged relative error of 13.7% in the Stipa baicalensis site and 18.7% in the Leymus Chinensis site. More fieldwork for various types of the grasslands is necessary.