This paper introduces a robot active task cognition framework for Situation-Aware Task Planning (SATP), leveraging visual scene understanding to generate action sequences. By integrating object knowledge, user preferences, and Large Language Models (LLMs), SATP interprets the robot’s current visual perception, and creates procedural actions that align with what the robot “sees”. Diverging from conventional methods requiring explicit verbal commands, our SATP framework autonomously performs task cognition, actively formulating robot-executable action sequences directly from visual input. Initially, a novel approach for describing the visual scene is presented, enabling the robot to grasp detailed object-level properties and inter-object relationships based on its observations. Building on this, a knowledge base for active task cognition is constructed using ontology technology. Furthermore, we develop a two-stage dual-feedback task planner, ReProg+, powered by LLMs, specifically designed for situation-aware task planning grounded in visual data. The efficacy, reliability, and advantages of our solution are thoroughly validated in real-world visual scenarios. Additionally, SATP has been tested with a real robot, with results confirming the feasibility and effectiveness of our approach.

In this article, the high-order fully actuated (HOFA) system approach is applied to study the matrix threshold strategy dynamic event-triggered control problem of a single-link flexible joint robot system (SFJRS). First, the SFJRS is transformed into an HOFA system model by using the recursive "ascending dimension and descending order" method. On this basis, different from the traditional method based on the state-space model, a novel matrix threshold strategy dynamic event-triggered control scheme based on the HOFA system approach is proposed, which not only simplifies the control design but also greatly saves communication resources. It is proved that the closed-loop systems are asymptotically stable under the proposed control strategy. Finally, the superiority of the HOFA system approach and the matrix threshold strategy dynamic event-triggered control method is demonstrated through two different simulation experiments.

This article proposes a long-term dynamic object relocalization (L-DOR) solution to address the challenge of mobile robots efficiently relocalize task-related objects in human–robot coexisting environments over extended periods. Existing methods mainly focus on one-time object localization, neglecting long-term relocalization amidst dynamic changes caused by human activities. To tackle this issue, a probabilistic model of object distribution based on spatio-temporal patterns is first established. Then, a robot-object perception interaction is introduced to achieve dynamic object category discrimination, enabling the robot to handle sporadic events caused by human activities. Besides, a cost-expectation balance-based object matching method is presented to determine the suitable task object and gradually infer its potential locations with probabilistic model. On this basis, we suggest a hierarchical global planning and local decision-making to prioritize search efforts to improve localization efficiency. Extensive comparisons and long-term experiments in real-world scenarios with a Fetch robot demonstrate the efficacy of L-DOR in terms of search performance, adaptability to different scenarios, and long-term effectiveness.

Object instance segmentation is a key prerequisite for service robots to perform daily chores in unstructured environments. Traditional supervised learning-based segmentation solutions rely on massive annotated datasets, which are impractical for the wide variety of objects in real-world scenarios. To this end, we propose a novel zero-shot instance segmentation approach (TransZSIS) that enables precise instance segmentation without relying on external semantic embeddings or auxiliary information to address the unseen object instance segmentation (UOIS) problem. First, the RGB and depth images are segmented into irregular patches based on a super-pixel segmentation algorithm to generate a unified segmentation map, and then the comprehensive feature vectors of each patch is extracted and paired. Further, a Transformer-based architecture is introduced to capture the correlation between different patch-pair and the intrinsic characteristics of each patch-pair. To predict patch-pair relationships, TransZSIS uses a four-layer fully connected neural network (FCNN) to classify the transformer-encoded features and refine them with a graph-based processing tactic to achieve object instance segmentation. Extensive evaluations on both synthetic and real datasets demonstrate that TransZSIS achieves superior performance compared with state-of-the-art baseline methods. Also, we implement real experiments to verify that our solution can achieve robot grasping by segmenting unseen objects.

This article addresses the distributed global prescribed-performance control problem for uncertain Lagrangian dynamics, with a particular emphasis on minimizing steady-state error oscillations. A novel global distributed prescribed-performance control framework is proposed based on a dynamic funnel function and neural network design. Specifically, by integrating funnel barrier properties and derivative information, a new neural network learning law is developed. Furthermore, a projection operator is incorporated into the learning law to guarantee the boundedness of the weight estimates in the stability proof, ultimately avoiding potential constraint incompatibility problems caused by neural network integration. The established control framework ensures that the trajectory consensus error of robotic manipulators under distributed control satisfies global arbitrary convergence rates and steady-state error bounds while leveraging neural network approximation to mitigate the inherent uncertainties of controllers that do not require precise mathematical model, thereby effectively suppressing steady-state error oscillations. Unlike existing literature, this work pioneers the incorporation of neural networks into distributed funnel control, achieving global prescribed performance while significantly reducing steady-state error oscillations. Finally, simulation results validate the effectiveness of the proposed method.

This article addresses the problem of prescribed-time stabilization of nonlinear systems with the features of unknown control directions and time-varying uncertain parameters based on a time-transformation method. The basic ideology relies on a newly established time-transformation method that incorporates the classical adaptive technique, converting the prescribed-time stable control problem of the original system into an asymptotically stable problem of its time-transformed stretched form. Unlike the existing literature, the time transformation method in this article directly gives the adaptive laws before and after the time transformation, which greatly reduces the complexity of designing the adaptive prescribed-time controller due to the fact that the design of the adaptive law in the stretched time domain only needs to satisfy the asymptotic stability criterion. Finally, the proposed methodology is validated by a simulation example.

The adaptive event-triggered control problem for a class of uncertain high-order pure feedback nonlinear systems (HOPFNSs) is considered. Different from the traditional backstepping method, a new high-order backstepping method is proposed based on the high-order fully actuated (HOFA) system approaches to design the adaptive event-triggered control law, which has the significant advantages of simple structure, high degree of freedom, and easy to realize. The high-order backstepping method does not need to transform the HOPFNSs into the first-order systems, which is more efficient and significantly reduces design complexity. It is proved that the adaptive event-triggered controller makes all the signals of the system bounded and save the energy in signal transmission. A simulation example is performed to verify the effectiveness of the control strategy.

Service robots operating in unstructured environments must effectively recognize and segment unknown objects to enhance their functionality. Traditional supervised learning-based segmentation techniques require extensive annotated datasets, which are impractical for the diversity of objects encountered in real-world scenarios. Unseen object instance segmentation (UOIS) methods aim to address this by training models on synthetic data to generalize to novel objects, but they often suffer from the simulation-to-reality gap. This article proposes a novel approach (ZISVFM) for solving UOIS by leveraging the powerful zero-shot capability of the segment anything model (SAM) and explicit visual representations from a self-supervised vision transformer (ViT). The proposed framework operates in the following three stages: generating object-agnostic mask proposals from colorized depth images using SAM, refining these proposals using attention-based features from the self-supervised ViT to filter nonobject masks, and applying K-Medoids clustering to generate point prompts that guide SAM toward precise object segmentation. Experimental validation on two benchmark datasets and a self-collected dataset demonstrates the superior performance of ZISVFM in complex environments, including hierarchical settings such as cabinets, drawers, and handheld objects.

This article explores the issue of event-triggered prescribed-time tracking control for uncertain nonlinear systems with unknown parameters and external disturbances. A novel adaptive event-triggered proportional–integral (PI) funnel control protocol is proposed, where the switching threshold event-triggered strategy is adopted to conserve communication resources while guaranteeing the expected control performance. To obtain the global results, a new method combining adaptive techniques with the design of prescribed functions with infinite initial values is presented. Compared with the existing schemes, a more concise PI-funnel control method is provided to realize the global prescribed-time tracking control performance, which avoids the external disturbance estimation and calculation of derivatives at each step of the traditional backstepping method. Finally, the effectiveness of the proposed method is illustrated through two simulation demonstrations.

The high-order fully actuated (HOFA) system approaches are utilized in this article to solve the synthesis problem of a class of generalized pseudo pure-feedback nonlinear control systems (PPFNCSs). Contrary to the traditional state space method, a recursive design approach is proposed to transform the generalized PPFNCSs into HOFA systems by adopting the idea of “ascending order and descending dimension.” In this framework, a class of generalized first-order, second-order, and mixed-order PPFNCSs are transformed into HOFA models based on the newly proposed generalized inverse function lemma, which overcomes the problem that traditional backstepping design methods are not applicable to such systems without any restrictions. Based on this, the linear time-invariant systems with the desired characteristic structure are derived by designing control strategies for the transformed HOFA systems, which not only achieves the desired control performance but also successfully avoids the problem of “explosion of complexity.” To demonstrate the effect of the approach, two examples are performed at final.

This article investigates the problem of dynamic memory event-triggered (DMET) fixed-time tracking control within time-varying asymmetric constraints for nonaffine nonstrict-feedback uncertain nonlinear systems with unmodeled dynamics and unknown disturbances. The existing dynamic event-triggered control methods cannot handle the nonlinear systems with unmodeled dynamics and nonaffine inputs, which greatly limits the applicability of the strategy. To this end, a novel DMET adaptive fuzzy fixed-time control protocol is constructed based on the idea of command filtered backstepping, in which a new dynamic signal function is established to deal with the unmodeled dynamics and an improved DMET mechanism (DMETM) is designed to solve the problem of nonaffine inputs. It is proved that the newly DMET control strategy ensures the tracking error converges to an arbitrarily small compact set in a fixed time and all the signals of the closed-loop systems are bounded. The effectiveness of the proposed approach is demonstrated by two simulation examples.

To incorporate the cooking logic into ingredient recognition from food images is beneficial for food cognition. Compared with food categorization, ingredient recognition gives a better understanding on food cognition, by providing crucial information on food compositions. However, there exist situations in which different food are made of different ingredients, thus it is necessary to incorporate cooking logic into ingredient recognition to achieve a better food cognition. Based on this point, our paper proposes a sequential learning method to guide a neural network based (NN-based) model on producing ingredients following the corresponding cooking logic in recipes. Firstly, in order to make a maximum utilization of visual features from images, a double-flow feature fusion module (DFFF) is proposed to obtain features from two image-based, visual tasks (food name proposal and multi-label ingredient proposal). After that, fused features from DFFF, together with original image features, are feed into a bidirectional long short time memory (Bi-LSTM) based ingredient generator to produce sequential ingredients. To guide the sequential ingredient generation process, reinforcement learning is employed by designing a hybrid loss related to both the common and personality traits in ingredients for optimizing the model ability of associating images and sequential ingredients. In addition, sequential ingredients are utilized in a backward flow by reconstructing food images, so that sequential ingredient generation can be further optimized in a complementary manner. In experiments, the results demonstrate the superiority of our method on driving the model to allocate more attention to the correlation between images and sequential ingredients, and produced ingredients are comprehensive and logical.

Object recognition, one of the main goals of robot vision, is a vital prerequisite for service robots to perform domestic tasks. Thanks to the rich sense of information provided by RGB-D sensors, RGB-D-based object recognition has received increasing attention. However, the existing works focus on collaborative RGB and depth data for object recognition, while ignoring the influence of depth image quality on recognition performance. Moreover, in real-world scenarios, there are many objects with strong similarity from certain observation angles, which poses a challenge for the service robot to recognize objects accurately. In this paper, we propose CNN-TransNet, a novel end-to-end Transformer-based architecture with convolutional neural networks (CNNs) for RGB-D object recognition. In order to deal with the effect of high inter-class similarity, discriminative multi-modal feature representations are generated by learning and relating multi-modal features at multiple levels. Besides, we employ a multi-modal fusion and projection (MMFP) module to reweight the contribution of each modality to address the problem of poor-quality depth image. Our proposed approach achieves state-of-the-art performance on three datasets (including Washington RGB-D Object Dataset, JHUIT-50, and Object Clutter Indoor Dataset), with accuracy of 95.4%, 98.1%, and 94.7%, respectively. The results demonstrate the effectiveness and superiority of the proposed model in RGB-D object recognition task.

Human beings tend to learn unknown knowledge in a gradual process, from the basic to the complex. Based on this point, we propose a progressive learning method for producing service strategies according to requests, with a hierarchical priori knowledge and reinforcement learning. Service strategy aims to guide how to perform home services and takes into consideration the relationship between actions and objects in home environment. In this paper, strategy generation is regarded as a text generation problem in question answering (QA). Firstly, a hierarchical priori knowledge with service-object correlation at the bottom and action-object correlation at the top is constructed to assist the understanding on the relationship of objects and actions in service strategies. Service-object correlation guides how to select proper objects with the correct order, while action-object correlation associates actions in strategies according to selected objects. Based on the hierarchical priori knowledge, a progressive learning method is proposed to make the model produce effective strategies with a sequential cognition, from service-object correlation (objects) to action-object correlation (actions). After that, reinforcement learning is employed to enhance the progressive guidance, by designing rewards in terms of the hierarchical priori knowledge. Finally, the proposed method is tested with both comparative experiments and ablation studies, and the experimental results demonstrate the superiority in producing comprehensive and logical strategies, indicating that the progressive learning method in our paper can further improve the QA performance.

The key challenges of scene recognition for service robots in various family environments are the view shortage of holistic scenes and poor adaptation. To address these problems, a family scene recognition mechanism for the service robot is proposed in this paper. A comprehensive application of fish-eye, pinhole, and depth cameras is provided to guarantee the sufficient view of robot. A selective CNN features fusion for the recognition of fish-eye scene images is designed to improve the training efficiency and the recognition accuracy. The mechanism is deployed in a designed hybrid cloud including public and private clouds. The proposed family scene recognition model is trained by large-scale datasets in the public cloud and runs in the private cloud. Besides, the recognition skill can be reinforced and increased by matching human guidance and CNN features to help the robot learn new scenes and improve the adaptation in different family environments. Extensive experiments are implemented to evaluate the proposed method using real scene images from six families. The experiment results show the validity and good performance of our method for the service robot scene recognition in various family environments.

It is a challenging task to produce recipes from images, due to the difficulty in bridging the gap from intuitive, static images to sequential, dynamic recipes. In this paper, we propose a novel recipe generation system for producing effective recipes from images. As medium steps, ingredient generation is introduced to guide recipe generation in our system. With potential information in ingredient lists, ingredient selection and ingredient sequence, the system is taught to generate effective recipes. For information representation, a hierarchical attention mechanism is designed to extract effective features for ingredient production and recipe generation. In order to guarantee the comprehensiveness and logic in recipes, a specific and explicit criterion around ingredients is designed under the framework of reinforcement learning. In ingredient generation, the system is required to generate ingredients with correct sequence in cooking procedures. And in recipe generation, ingredients in recipes are required to be consistent with produced ingredients. In experiments, the proposed method is compared with state-of-the-art methods to evaluate the feasibility. The results indicate that the proposed system achieves a better performance than other methods on both aspects of producing proper ingredients and effective recipes.

The proven efficacy of safety strategies based on 2-D laser rangefinder (LRF) strongly stimulates their application to mobile robots operating in the home environment. However, it remains a challenge for the robot to avoid collisions with all obstacles in the environment. Since LRF can only scan a horizontal slice of the world, some objects cannot be fully observed, such as tables and chairs. In this article, an effective solution based on laser-visual fusion is presented to enhance the safety of the robot. First, a vision sensor is adopted to help detect obstacles that are not fully visible to LRF. Then we propose a method to convert the depth information of the visual image into 2-Dpseudo-laser datarepresentation. With this representation, a strategy for 2-D mapping is developed. On this basis, a novel map fusion algorithm is proposed to generate an improved grid map that amends the incorrect representation of obstacles on the traditional 2-D grid map. We further investigate a robot autonomous navigation strategy that considers LRF data and pseudo-laser data to avoid all obstacles. Experimental results show that the improved grid map together with the presented navigation strategy allows the robot not only to plan a “real” collision-free path, but also to navigate safely in both static and dynamic scenarios, and the proposed strategies can significantly enhance the performance of robot navigation in terms of safety, reliability and robustness.