Witnessing the rapid progress and development in instance-level object pose estimation, increasing attention has shifted to the more challenging problem for unseen objects, which is in great demand for various robotic applications. In this paper, we propose the GPD, a novel framework for unseen object pose estimation, including both category-level and cross-category objects. The key innovation of the GPD model is the effective utilization of geometric primitives in target reconstruction and pose estimation, as it can generalize the learned primitive deformation across intra-class and inter-class instances. Additionally, we also design an advanced scheme for representative object feature extraction, including attention-aware excitation, multi-scale fusion, and semantic feature encoding. Extensive evaluations validate the effectiveness of individual innovation modules and the overall superior performance of the GPD. It not only achieves the SOTA results on category-level benchmarks CAMERA25 and REAL275, but also demonstrates impressive generalization ability across novel objects on the GraspNet-1Billion dataset. Furthermore, we deploy the trained GPD model for vision-guided robotic grasping experiments in simulation and real-world settings, again exhibiting its outstanding robustness and practicability in robotic manipulations. Note to Practitioners—This paper is motivated by the problem of unseen object pose estimation and robotic manipulation in unstructured environments. For intelligent robots expected to interact with their surroundings, rather than just passively perceiving them like surveillance cameras, 6Dof pose estimation is a critical capability. However, existing approaches generally face two key challenges. On the one hand, robots are likely to encounter unseen objects in real-world applications. Without the availability of prior models or specific training data for these unseen objects, instance-level and category-level methods may become ineffective or even fail to work. On the other hand, the error tolerance of precise tabletop robotic manipulation is very tight, and the varying lighting conditions and background noise impose higher robustness requirements on pose estimation algorithms. To address these difficulties, we propose a novel network that learns geometric primitive deformation for pose estimation. This model is less dependent on object prior information, thereby enhancing the generalization ability. Additionally, by incorporating cross-modal excitation and multi-scale fusion during feature extraction, our model can capture representative appearance and geometric information of objects for accurate pose estimation. Extensive experimental results on benchmark datasets quantitatively validate the superior performance of our approach. We also demonstrate its effectiveness in robotic applications through unseen object grasping experiments on Kinova and Franka Emika robot platforms. In the future, we plan to explore primitive combination schemes for compound object representation, enabling pose estimation for more complex-shaped objects.

Recent planning methods based on Large Language Models typically employ the In-Context Learning paradigm. Complex long-horizon planning tasks require more context(including instructions and demonstrations) to guarantee that the generated plan can be executed correctly. However, in such conditions, LLMs may overlook(unfaithful) the rules in the given context, resulting in the generated plans being invalid or even leading to dangerous actions. In this paper, we investigate the faithfulness of LLMs for complex long-horizon tasks. Inspired by human intelligence, we introduce a novel framework named FLTRNN. FLTRNN employs a language-based RNN structure to integrate task decomposition and memory management into LLM planning inference, which could effectively improve the faithfulness of LLMs and make the planner more reliable. We conducted experiments in VirtualHome household tasks. Results show that our model significantly improves faithfulness and success rates for complex long-horizon tasks. Website at https://tannl.github.io/FLTRNN.github.io/

Scene graph generation (SGG) methods have historically suffered from long-tail bias and slow inference speed. In this paper, we notice that humans can analyze relationships between objects relying solely on context descriptions, and this abstract cognitive process may be guided by experience. For example, given descriptions of cup and table with their spatial locations, humans can speculate possible relationshipsor. Even without visual appearance information, some impossible predicates like flying in and looking at can be empirically excluded. Accordingly, we propose a contextual scene graph generation (C-SGG) method without using visual information and introduce a context augmentation method. We propose that slight perturbations in the position and size of objects do not essentially affect the relationship between objects. Therefore, at the context level, we can produce diverse context descriptions by using a context augmentation method based on the original dataset. These diverse context descriptions can be used for unbiased training of C-SGG to alleviate long-tail bias. In addition, we also introduce a context guided visual scene graph generation (CV-SGG) method, which leverages the C-SGG experience to guide vision to focus on possible predicates. Through extensive experiments on the publicly available dataset, C-SGG alleviates long-tail bias and omits the huge computation of visual feature extraction to realize real-time SGG. CV-SGG achieves a great trade-off between common predicates and tail predicates.

Despite the giant leap made in object 6D pose estimation and robotic grasping under structured scenarios, most approaches depend heavily on the exact CAD models of target objects beforehand, thereby limiting their wide applications. To address this, we propose a novel knowledge-guided network - KGNet to estimate the pose and size of category-level unseen objects. This network includes three primary innovations: knowledge-guided categorical model generation, pointwise deformation probability matrix and synergetic RGBD feature fusion, with the former two leveraging categorical object knowledge for unseen object reconstruction and the latter one facilitating pose-sensitive feature extraction. Exten-sive experiments on CAMERA25 and REAL275 verify their effectiveness, and KGNet achieves the SOTA performance on these two acknowledged benchmarks. Additionally, a real-world robotic grasping experiment is conducted, and its results further qualitatively prove the practicability and robustness of KGNet.

In this paper, we propose a novel RGBD-based object 6DoF pose estimation network - RFFCE. It is a two-stage method that firstly leverages deep neural networks for feature extraction and object points matching, and then the geometric principles are utilized for final pose computation. Our approach consists of three primary innovations: residual feature fusion for representative RGBD feature extraction; confidence evaluation and confidence-based paired points offsets regression for self-evaluation and self-optimization respectively. Their effectiveness is verified through an ablation study, and our RFFCE achieves the SOTA performance on LineMOD, Occlusion-LineMOD and YCB-Video datasets. Additionally, we also conduct a real-world object grasping experiment for visualization and qualitative evaluation of the RFFCE.

Human Action Recognition plays a driving engine of many human-computer interaction applications. Most current researches focus on improving the model generalization by integrating multiple homogeneous modalities, including RGB images, human poses, and optical flows. Furthermore, contextual interactions and out-of-context sign languages have been validated to depend on scene category and human per se. Those attempts to integrate appearance features and human poses have shown positive results. However, with human poses' spatial errors and temporal ambiguities, existing methods are subject to poor scalability, limited robustness, and sub-optimal models. In this paper, inspired by the assumption that different modalities may maintain temporal consistency and spatial complementarity, we present a novel Bi-directional Co-temporal and Cross-spatial Attention Fusion Model (B2C-AFM). Our model is characterized by the asynchronous fusion strategy of multi-modal features along temporal and spatial dimensions. Besides, the novel explicit motion-oriented pose representations called Limb Flow Fields (Lff) are explored to alleviate the temporal ambiguity regarding human poses. Experiments on publicly available datasets validate our contributions. Abundant ablation studies experimentally show that B2C-AFM achieves robust performance across seen and unseen human actions. The codes are available at https://github.com/gftww/B2C.git.