This paper introduces a robot active task cognition framework for Situation-Aware Task Planning (SATP), leveraging visual scene understanding to generate action sequences. By integrating object knowledge, user preferences, and Large Language Models (LLMs), SATP interprets the robot’s current visual perception, and creates procedural actions that align with what the robot “sees”. Diverging from conventional methods requiring explicit verbal commands, our SATP framework autonomously performs task cognition, actively formulating robot-executable action sequences directly from visual input. Initially, a novel approach for describing the visual scene is presented, enabling the robot to grasp detailed object-level properties and inter-object relationships based on its observations. Building on this, a knowledge base for active task cognition is constructed using ontology technology. Furthermore, we develop a two-stage dual-feedback task planner, ReProg+, powered by LLMs, specifically designed for situation-aware task planning grounded in visual data. The efficacy, reliability, and advantages of our solution are thoroughly validated in real-world visual scenarios. Additionally, SATP has been tested with a real robot, with results confirming the feasibility and effectiveness of our approach.

In this article, the high-order fully actuated (HOFA) system approach is applied to study the matrix threshold strategy dynamic event-triggered control problem of a single-link flexible joint robot system (SFJRS). First, the SFJRS is transformed into an HOFA system model by using the recursive "ascending dimension and descending order" method. On this basis, different from the traditional method based on the state-space model, a novel matrix threshold strategy dynamic event-triggered control scheme based on the HOFA system approach is proposed, which not only simplifies the control design but also greatly saves communication resources. It is proved that the closed-loop systems are asymptotically stable under the proposed control strategy. Finally, the superiority of the HOFA system approach and the matrix threshold strategy dynamic event-triggered control method is demonstrated through two different simulation experiments.

This article proposes a long-term dynamic object relocalization (L-DOR) solution to address the challenge of mobile robots efficiently relocalize task-related objects in human–robot coexisting environments over extended periods. Existing methods mainly focus on one-time object localization, neglecting long-term relocalization amidst dynamic changes caused by human activities. To tackle this issue, a probabilistic model of object distribution based on spatio-temporal patterns is first established. Then, a robot-object perception interaction is introduced to achieve dynamic object category discrimination, enabling the robot to handle sporadic events caused by human activities. Besides, a cost-expectation balance-based object matching method is presented to determine the suitable task object and gradually infer its potential locations with probabilistic model. On this basis, we suggest a hierarchical global planning and local decision-making to prioritize search efforts to improve localization efficiency. Extensive comparisons and long-term experiments in real-world scenarios with a Fetch robot demonstrate the efficacy of L-DOR in terms of search performance, adaptability to different scenarios, and long-term effectiveness.

Object instance segmentation is a key prerequisite for service robots to perform daily chores in unstructured environments. Traditional supervised learning-based segmentation solutions rely on massive annotated datasets, which are impractical for the wide variety of objects in real-world scenarios. To this end, we propose a novel zero-shot instance segmentation approach (TransZSIS) that enables precise instance segmentation without relying on external semantic embeddings or auxiliary information to address the unseen object instance segmentation (UOIS) problem. First, the RGB and depth images are segmented into irregular patches based on a super-pixel segmentation algorithm to generate a unified segmentation map, and then the comprehensive feature vectors of each patch is extracted and paired. Further, a Transformer-based architecture is introduced to capture the correlation between different patch-pair and the intrinsic characteristics of each patch-pair. To predict patch-pair relationships, TransZSIS uses a four-layer fully connected neural network (FCNN) to classify the transformer-encoded features and refine them with a graph-based processing tactic to achieve object instance segmentation. Extensive evaluations on both synthetic and real datasets demonstrate that TransZSIS achieves superior performance compared with state-of-the-art baseline methods. Also, we implement real experiments to verify that our solution can achieve robot grasping by segmenting unseen objects.

This article addresses the distributed global prescribed-performance control problem for uncertain Lagrangian dynamics, with a particular emphasis on minimizing steady-state error oscillations. A novel global distributed prescribed-performance control framework is proposed based on a dynamic funnel function and neural network design. Specifically, by integrating funnel barrier properties and derivative information, a new neural network learning law is developed. Furthermore, a projection operator is incorporated into the learning law to guarantee the boundedness of the weight estimates in the stability proof, ultimately avoiding potential constraint incompatibility problems caused by neural network integration. The established control framework ensures that the trajectory consensus error of robotic manipulators under distributed control satisfies global arbitrary convergence rates and steady-state error bounds while leveraging neural network approximation to mitigate the inherent uncertainties of controllers that do not require precise mathematical model, thereby effectively suppressing steady-state error oscillations. Unlike existing literature, this work pioneers the incorporation of neural networks into distributed funnel control, achieving global prescribed performance while significantly reducing steady-state error oscillations. Finally, simulation results validate the effectiveness of the proposed method.

This article addresses the problem of prescribed-time stabilization of nonlinear systems with the features of unknown control directions and time-varying uncertain parameters based on a time-transformation method. The basic ideology relies on a newly established time-transformation method that incorporates the classical adaptive technique, converting the prescribed-time stable control problem of the original system into an asymptotically stable problem of its time-transformed stretched form. Unlike the existing literature, the time transformation method in this article directly gives the adaptive laws before and after the time transformation, which greatly reduces the complexity of designing the adaptive prescribed-time controller due to the fact that the design of the adaptive law in the stretched time domain only needs to satisfy the asymptotic stability criterion. Finally, the proposed methodology is validated by a simulation example.

This article considers the design of prescribed-time sensor fault estimators (PSFEs) and unknown input compensation for linear systems, i.e., estimators that estimate the sensor fault and controllers that stabilize the state of the system both at a prescribed time. With this intention, a filter is introduced to eliminate the limitation of the fault being differentiable. Then, the task of designing PSFEs can be converted into the task of designing prescribed-time unknown input observers for the augmented systems. Next, the generalized inverse is employed to convert the augmented systems into a form suitable for observer design. Then, the PSFEs are developed by exploiting periodic delayed output. Both full- and reduced-order PSFEs are taken into account. In addition, based on the established state observer and unknown input estimation, periodic delayed controllers are designed to completely compensate for the unknown input so that the closed system is T-prescribed-time stable (T-PS). Finally, an example is provided to demonstrate the efficacy of the proposed methods.

The adaptive event-triggered control problem for a class of uncertain high-order pure feedback nonlinear systems (HOPFNSs) is considered. Different from the traditional backstepping method, a new high-order backstepping method is proposed based on the high-order fully actuated (HOFA) system approaches to design the adaptive event-triggered control law, which has the significant advantages of simple structure, high degree of freedom, and easy to realize. The high-order backstepping method does not need to transform the HOPFNSs into the first-order systems, which is more efficient and significantly reduces design complexity. It is proved that the adaptive event-triggered controller makes all the signals of the system bounded and save the energy in signal transmission. A simulation example is performed to verify the effectiveness of the control strategy.

Service robots operating in unstructured environments must effectively recognize and segment unknown objects to enhance their functionality. Traditional supervised learning-based segmentation techniques require extensive annotated datasets, which are impractical for the diversity of objects encountered in real-world scenarios. Unseen object instance segmentation (UOIS) methods aim to address this by training models on synthetic data to generalize to novel objects, but they often suffer from the simulation-to-reality gap. This article proposes a novel approach (ZISVFM) for solving UOIS by leveraging the powerful zero-shot capability of the segment anything model (SAM) and explicit visual representations from a self-supervised vision transformer (ViT). The proposed framework operates in the following three stages: generating object-agnostic mask proposals from colorized depth images using SAM, refining these proposals using attention-based features from the self-supervised ViT to filter nonobject masks, and applying K-Medoids clustering to generate point prompts that guide SAM toward precise object segmentation. Experimental validation on two benchmark datasets and a self-collected dataset demonstrates the superior performance of ZISVFM in complex environments, including hierarchical settings such as cabinets, drawers, and handheld objects.

This article explores the issue of event-triggered prescribed-time tracking control for uncertain nonlinear systems with unknown parameters and external disturbances. A novel adaptive event-triggered proportional–integral (PI) funnel control protocol is proposed, where the switching threshold event-triggered strategy is adopted to conserve communication resources while guaranteeing the expected control performance. To obtain the global results, a new method combining adaptive techniques with the design of prescribed functions with infinite initial values is presented. Compared with the existing schemes, a more concise PI-funnel control method is provided to realize the global prescribed-time tracking control performance, which avoids the external disturbance estimation and calculation of derivatives at each step of the traditional backstepping method. Finally, the effectiveness of the proposed method is illustrated through two simulation demonstrations.

The high-order fully actuated (HOFA) system approaches are utilized in this article to solve the synthesis problem of a class of generalized pseudo pure-feedback nonlinear control systems (PPFNCSs). Contrary to the traditional state space method, a recursive design approach is proposed to transform the generalized PPFNCSs into HOFA systems by adopting the idea of “ascending order and descending dimension.” In this framework, a class of generalized first-order, second-order, and mixed-order PPFNCSs are transformed into HOFA models based on the newly proposed generalized inverse function lemma, which overcomes the problem that traditional backstepping design methods are not applicable to such systems without any restrictions. Based on this, the linear time-invariant systems with the desired characteristic structure are derived by designing control strategies for the transformed HOFA systems, which not only achieves the desired control performance but also successfully avoids the problem of “explosion of complexity.” To demonstrate the effect of the approach, two examples are performed at final.

This article investigates the problem of dynamic memory event-triggered (DMET) fixed-time tracking control within time-varying asymmetric constraints for nonaffine nonstrict-feedback uncertain nonlinear systems with unmodeled dynamics and unknown disturbances. The existing dynamic event-triggered control methods cannot handle the nonlinear systems with unmodeled dynamics and nonaffine inputs, which greatly limits the applicability of the strategy. To this end, a novel DMET adaptive fuzzy fixed-time control protocol is constructed based on the idea of command filtered backstepping, in which a new dynamic signal function is established to deal with the unmodeled dynamics and an improved DMET mechanism (DMETM) is designed to solve the problem of nonaffine inputs. It is proved that the newly DMET control strategy ensures the tracking error converges to an arbitrarily small compact set in a fixed time and all the signals of the closed-loop systems are bounded. The effectiveness of the proposed approach is demonstrated by two simulation examples.

This article is concerned with the problem of event-based remote state estimation for nonlinear/non-Gaussian systems on a wireless network with limited bandwidth. To reduce unnecessary data transmissions, a novel event-triggering mechanism is developed by using the least-square technique. Based on this, an event-triggered box particle filtering scheme is designed to realize the minimum mean-squared error estimation at the remote estimator end, in which the posterior probability density functions are calculated separately according to the information of the event-triggered indicator to avoid the problem of excessive estimation error. Different from the existing approaches, the proposed algorithm does not depend on any Gaussian assumptions and reduces the computational complexity under the premise of ensuring the estimation performance. Finally, two simulation examples are performed to demonstrate the validity of the proposed algorithm.

This paper investigates the event-triggered adaptive output feedback control problem for a class of uncertain nonlinear systems in the presence of actuator failures and unknown control direction. By utilizing the adaptive backstepping technique, an event-based output feedback controller is developed together with a time-variant event-triggered rule. In this design, the radial basis function neural network algorithms are first introduced to identify the unknown terms of the systems. Then, a new state observer with adaptive compensation is designed to estimate the state vector. The overall control strategy guarantees that the output signal tracks the reference signal and all the signals of the closed-loop systems are bounded. Unlike the existing methods, the proposed control scheme can handle the coupling term incurred by the loss of effectiveness fault of the actuator, the event-triggered rule, and unknown control direction. Finally, an example is performed to demonstrate the validity of the proposed strategy.