This article proposes a long-term dynamic object relocalization (L-DOR) solution to address the challenge of mobile robots efficiently relocalize task-related objects in human–robot coexisting environments over extended periods. Existing methods mainly focus on one-time object localization, neglecting long-term relocalization amidst dynamic changes caused by human activities. To tackle this issue, a probabilistic model of object distribution based on spatio-temporal patterns is first established. Then, a robot-object perception interaction is introduced to achieve dynamic object category discrimination, enabling the robot to handle sporadic events caused by human activities. Besides, a cost-expectation balance-based object matching method is presented to determine the suitable task object and gradually infer its potential locations with probabilistic model. On this basis, we suggest a hierarchical global planning and local decision-making to prioritize search efforts to improve localization efficiency. Extensive comparisons and long-term experiments in real-world scenarios with a Fetch robot demonstrate the efficacy of L-DOR in terms of search performance, adaptability to different scenarios, and long-term effectiveness.

Object instance segmentation is a key prerequisite for service robots to perform daily chores in unstructured environments. Traditional supervised learning-based segmentation solutions rely on massive annotated datasets, which are impractical for the wide variety of objects in real-world scenarios. To this end, we propose a novel zero-shot instance segmentation approach (TransZSIS) that enables precise instance segmentation without relying on external semantic embeddings or auxiliary information to address the unseen object instance segmentation (UOIS) problem. First, the RGB and depth images are segmented into irregular patches based on a super-pixel segmentation algorithm to generate a unified segmentation map, and then the comprehensive feature vectors of each patch is extracted and paired. Further, a Transformer-based architecture is introduced to capture the correlation between different patch-pair and the intrinsic characteristics of each patch-pair. To predict patch-pair relationships, TransZSIS uses a four-layer fully connected neural network (FCNN) to classify the transformer-encoded features and refine them with a graph-based processing tactic to achieve object instance segmentation. Extensive evaluations on both synthetic and real datasets demonstrate that TransZSIS achieves superior performance compared with state-of-the-art baseline methods. Also, we implement real experiments to verify that our solution can achieve robot grasping by segmenting unseen objects.

Service robots operating in unstructured environments must effectively recognize and segment unknown objects to enhance their functionality. Traditional supervised learning-based segmentation techniques require extensive annotated datasets, which are impractical for the diversity of objects encountered in real-world scenarios. Unseen object instance segmentation (UOIS) methods aim to address this by training models on synthetic data to generalize to novel objects, but they often suffer from the simulation-to-reality gap. This article proposes a novel approach (ZISVFM) for solving UOIS by leveraging the powerful zero-shot capability of the segment anything model (SAM) and explicit visual representations from a self-supervised vision transformer (ViT). The proposed framework operates in the following three stages: generating object-agnostic mask proposals from colorized depth images using SAM, refining these proposals using attention-based features from the self-supervised ViT to filter nonobject masks, and applying K-Medoids clustering to generate point prompts that guide SAM toward precise object segmentation. Experimental validation on two benchmark datasets and a self-collected dataset demonstrates the superior performance of ZISVFM in complex environments, including hierarchical settings such as cabinets, drawers, and handheld objects.

Object recognition, one of the main goals of robot vision, is a vital prerequisite for service robots to perform domestic tasks. Thanks to the rich sense of information provided by RGB-D sensors, RGB-D-based object recognition has received increasing attention. However, the existing works focus on collaborative RGB and depth data for object recognition, while ignoring the influence of depth image quality on recognition performance. Moreover, in real-world scenarios, there are many objects with strong similarity from certain observation angles, which poses a challenge for the service robot to recognize objects accurately. In this paper, we propose CNN-TransNet, a novel end-to-end Transformer-based architecture with convolutional neural networks (CNNs) for RGB-D object recognition. In order to deal with the effect of high inter-class similarity, discriminative multi-modal feature representations are generated by learning and relating multi-modal features at multiple levels. Besides, we employ a multi-modal fusion and projection (MMFP) module to reweight the contribution of each modality to address the problem of poor-quality depth image. Our proposed approach achieves state-of-the-art performance on three datasets (including Washington RGB-D Object Dataset, JHUIT-50, and Object Clutter Indoor Dataset), with accuracy of 95.4%, 98.1%, and 94.7%, respectively. The results demonstrate the effectiveness and superiority of the proposed model in RGB-D object recognition task.