This paper researches the adaptive fixed-time global prescribed performance tracking control (GPPTC) problem for a robotic manipulator (RM) under unknown backlash-like hysteresis and disturbance. In the control design of the RM, a novel global fixed-time prescribed performance function (FTPPF) is designed to achieve output tracking with arbitrarily prescribed transient performance. Moreover, fuzzy logic systems (FLSs) are used to handle the unknown robotic dynamics, and a hyperbolic tangent activation function is applied to deal with the hysteresis nonlinearity. The proposed prescribed performance control (PPC) method based on the FTPPF ensures that the tracking error converges to a predetermined range in a given time and remains within that range thereafter. Unlike the existing PPC methods, our control algorithm removes the limitation requiring the initial tracking error to satisfy the initial value of the PPF. Based on the fixed-time command-filtered (FTCF) technique, an adaptive fixed-time controller is designed, which enables the joint position to track the desired signal in a fixed time. Finally, the effectiveness of the proposed controller is demonstrated via the simulation.

Visual Place Recognition (VPR) plays a vital role in mobile robotics and autonomous navigation by retrieving reference images from a pre-established database. However, VPR systems frequently encounter performance degradation due to environmental variations. To overcome these challenges, we propose a re-ranking based VPR framework incorporating two key components: (1) A Visual Mamba Embedding (VME) module that optimizes spatial-channel feature interactions to generate discriminative global descriptors; and (2) A Spatial Graph Attentional Network (SGAN) that replaces conventional RANSAC-based verification with an efficient graph attention mechanism, improving matching accuracy while reducing computation. Comprehensive evaluations across multiple benchmark datasets demonstrate that the proposed method achieves superior performance compared to existing state-of-the-art methods, while maintaining advantages in computational efficiency and storage requirements.

Dexterous Functional Grasping (DFG) is the crucial first step for humanoid robots to perform generalized manipulation tasks. However, enabling robots to learn language-guided DFG skills in real-world environments presents several challenges, including comprehending the complex relationship between task instructions and grasp functionality, generating feasible functional grasps of dexterous hands, and handling generalization for novel functional concepts. To tackle these challenges, we introduce SayFuncGrasp, a Large Language Model (LLM) based DFG framework that can synthesize versatile dexterous functional grasps from language instructions and achieve generalization on novel functional concepts. SayFuncGrasp first harnesses the open-ended manipulation knowledge from an LLM to infer grasp functionality based on language instructions. Subsequently, it employs the inferred grasp functionality to synthesize plausible DFG actions characterized by hand synergies. Simulation experiments show that SayFuncGrasp significantly outperforms the baseline method in open-set grasp functionality generalization. Real robot experiments demonstrate the effectiveness and generalizability of SayFuncGrasp for interactive humanoid manipulation tasks, achieving an overall grasp success rate of 64.66% and a manipulation success rate of 70.41%. Note to Practitioners—This research was motivated by the practical challenge of enabling humanoid robots with high-DoF dexterous hands to perform functional grasping based on verbal instructions. In industrial settings, such capabilities can significantly enhance the versatility and adaptability of humanoid assistants, allowing them to perform complex manipulations simply by being told what to do, thereby reducing programming complexity and increasing flexibility. Current dexterous functional grasping methods rely solely on visual input, without the ability to process language instructions. Furthermore, they are restricted to pre-defined functional concepts and cannot be generalized to novel object classes and manipulation tasks within natural language. Our newly proposed language-guided dexterous functional grasping system takes advantage of open-ended manipulation knowledge from LLMs to produce generalized functional grasps of dexterous robot hands according to verbal commands. Our experiment results demonstrate improved versatility and generalizability compared to the state-of-the-art.

The bundle adjustment (BA) algorithm is a widely used nonlinear optimization technique in simultaneous localization and mapping (SLAM) systems. By leveraging the co-view relationships of landmarks from multiple perspectives, the BA method constructs a joint estimation model for both poses and landmarks, enabling the system to generate refined maps and reduce front-end localization errors. However, exploring a robust LiDAR BA estimator and achieving accurate solutions is a challenge. In this work, firstly we propose a novel mean square group metric (MSGM) to build the optimization objective of the LiDAR BA algorithm. This metric applies a mean square transformation to uniformly process the measurements of plane landmarks during one sampling period. The transformed metric ensures scale interpretability and does not require a time-consuming point-by-point calculation. Secondly, by integrating a robust kernel function, the metrics involved in the BA algorithm are reweighted, thus enhancing the robustness of the solution process. Thirdly, based on the proposed robust LiDAR BA model, we derived an explicit second-order estimator (RSO-BA). This estimator employs analytical formulas for Hessian and gradient calculations, ensuring the precision of the BA solution. Finally, we verify the merits of the proposed RSO-BA estimator against existing implicit second-order and explicit approximate second-order estimators using publicly available datasets and physical experiments. The experimental results demonstrate that the RSO-BA estimator outperforms its counterparts in terms of registration accuracy and robustness, particularly in dynamic or complex unstructured environments. Note to Practitioners—The motivation of this paper is to develop a novel LiDAR bundle adjustment (BA) algorithm that ensures accurate and consistent 3D scene modeling. Currently, most LiDAR BA algorithms use “group” processing to construct cost metrics, aiming to reduce the computational complexity of point-by-point operations. However, the cost metrics of these approaches lack scale interpretability, which makes it difficult to incorporate robust kernel functions into the model design, ultimately weakening the system’s robustness and reducing estimation accuracy. To address these issues, we propose a mean square group metric (MSGM) that considers the number of measurement points, to construct the optimization objective for the LiDAR BA (RSO-BA) problem. In each optimization iteration, a robust kernel function reweights each metric to ensure robustness in the solution. Additionally, we derive the analytical Hessian matrix and gradient vector required for solving the RSO-BA, with the inclusion of second-order terms enhancing estimation accuracy. The proposed method can be directly applied to the mobile robot system for automatic driving, home service and unmanned security applications. Furthermore, the proposed algorithm can be extended to sensors such as depth cameras, which provide direct depth information, for broader practical applications.

This article studies the dynamic event-triggered adaptive finite-time tracking control issue for a robotic manipulator (RM) system with disturbances. First, a new global prescribed performance function (PPF) is designed based on a scaling function such that the tracking error evolves within the constrained bounds and the restriction related to the initial conditions is removed. Then, the finite-time command filter (FTCF) is used to avoid the direct derivations of virtual controllers and the singularity issue of the conventional backstepping technique. Moreover, the filtering errors caused by the FTCF are removed by the designed error compensation mechanism. A novel dynamic event-triggered mechanism (DETM) using the dynamic auxiliary variable is designed to save communication resources. The proposed control scheme can guarantee that all signals of the RM are globally bounded within a finite time, and the tracking error can asymptotically reach zero. Finally, a simulation example and several comparative simulations show the validity of the proposed scheme.

Motor skills are essential for robots to accomplish complicated and dexterous manipulation tasks, which are difficult to be mastered through traditional controller designs. Currently, robots learning from demonstrations enable them to learn control policies automatically from human motor demonstrations. However, the nonlinearity and instantaneousness of the demonstrated forces prohibit robots from fully mastering the motor skill features by simply exploiting force examples. Therefore, a self-improvement learning scheme is required to refine the control policy further until satisfactory motor skills are acquired. Hence, this paper combines learning from demonstrations and reinforcement learning to learn a controller for complex motor skills. The proposed method is validated on an IIWA KUKA robot, performing a specified nonprehensile manipulation task.Note to Practitioners—The motivation of this paper originates from the requirement to develop an efficient and fast learning algorithm that improves the robot skill learning efficiency. Specifically, our research focuses on the nonprehensile manipulation task, easily subject to environmental changes. Therefore, the robot must continuously interact with the environment to master the skill. To accelerate the skill learning process, learning from demonstrations initializes the control policies, and then the robot starts to practice the demonstrated skill. After each practice round, the robot receives a reward from the environment, and based on the reinforcement learning algorithm limited up to 100 trials, the robot masters the nonprehensile manipulation skill.

This paper focuses on addressing the visual tracking problem using learning-based methods for object tracking tasks. This problem contains a major difficulty, i.e., how to acquire a satisfactory generalization ability of the developed system? In this paper, firstly, the object state tracking system, including a camera-in-hand, a 3D camera and a Rethink Baxter robot, is introduced. The problem formulation is also presented. Secondly, we propose a Kalman-based estimation strategy to acquire the object’s state. In addition, a learning-based tracking controller is developed using the Gaussian mixture models (GMM) method to steer the robot end-effector to track the mobile object. Thirdly, to guarantee system stability (i.e., the position and velocity errors between the object and end-effector will always converge to zeros), the controller parameter constraints are derived, which is a theoretical contribution of this paper. The controller parameter adjustment is avoided by the proposed training process. Thus, the proposed method becomes easy to implement, which is a practical contribution. Finally, the effectiveness of the proposed method is demonstrated by simulation and experimental examples, and the proposed method has satisfactory generalization ability. Note to Practitioners—This paper studies object tracking problems for different practical applications, such as industrial cutting, grasping and dynamic monitoring. Different trajectory tracking methods have been widely applied in the industrial area. However, users always complain that when the object or trajectory is changed, the tracking controller more or less needs to be re-adjusted. This re-adjust process always requires professional knowledge and programming experience, and thus a factory must employ some professional engineers. In addition, since the objects may be diverse in shape, color and size, to acquire the accurate object position and velocity, an appropriate solution is necessary. Motivated by the above introductions, this paper aims to develop a learning-based controller to track different complex trajectories without frequent and specific parameter adjustment processes. Firstly, a visual measurement system is developed to quickly find and estimate the position and velocity of an object. Secondly, with the object’s information, the learning from demonstration method (GMM method) is applied for the control policy design. Thirdly, the detailed system stability analysis is presented, and the corresponding controller parameter constraints are derived and considered in the proposed control policy. Subsequently, with the demonstration data, the packaged learning algorithm will automatically compute the controller parameters, and users can change the controller performance only by providing the desired demonstrations. In summary, this paper proposes a systematic object tracking solution, and it may bring a new idea to develop a practical object tracking system, using both the learning-based methods to improve the ability of generalization for tracking different objects.

In this article, we address the trajectory tracking problem using the learning from demonstration (LFD) method. By using the LFD method, the parameter adjusting problem in the tracking controller is avoided. Consequently, a strategy can be provided to users with limited parameter adjusting experience. The kinematic tracking problem is formulated as a second-order system and the objective is to simultaneously reduce the errors in position and velocity. The extreme learning machines (ELM) algorithm is applied in the controller design. The velocity and position are utilized as the inputs and the output is the robot corrected kinematic movement. The controller parameters are learned from the desired human or programming demonstrations taking into consideration the stability constraints. In this work, we analyze the system local and global asymptotic stability in detail. The effectiveness of the proposed strategy is demonstrated by simulation comparisons and a practical experiment using a KUKA robot manipulator.

Low-light image enhancement is a complex and vital task including, recovering color and texture details from low-light images. For automated driving, low-light scenarios will have severe implications for vision-based applications. To address this problem, we propose a real-time unsupervised generative adversarial network (GAN) with multiple discriminators. It includes a multi-scale discriminator, a texture discriminator, and a color discriminator to evaluate images from different perspectives. Furthermore, considering the uneven illumination distribution of images and the different information contained in the channels, we adopte a feature fusion attention module to combine channel attention with pixel attention to extract image features. Experiments show that our method outperforms state-of-the-art methods in qualitative and quantitative evaluation and provides visible improvements in SLAM localization effects.

Learning from Demonstration (LfD) has been identified as an effective method for making robots adapt to a similar kind of tasks. In this work, a framework of learning from demonstration has been proposed for modelling robot motions. We present an approach based on dimension ascending to learn a dynamical system, so that the reproduced motions can closely follow the demonstrations. In addition, the reproductions can ultimately reach and stop at the target, which reflects the robustness of the method. Therefore, the system accuracy and stability can be better guaranteed simultaneously. The effectiveness of the proposed approach is verified by performing handwriting experiments on the LASA data set.

This letter investigates how to place the received-signal-strength (RSS) sensors to improve the static target localization accuracy in the three-dimensional (3-D) space. By using the A-optimality criterion, i.e., minimizing the trace of the inverse Fisher information matrix (FIM), a new optimal RSS sensor placement strategy is developed when sensors can be placed freely in the 3-D space. The smallest reachable trace of Cramér-Rao lower bound, i.e., the inverse FIM, is derived with the corresponding optimal sensor-target geometries. Besides, a resistor network method and a special configuration strategy are proposed to quickly determine the optimal geometries. The findings are concluded in three remarks, which are used to evaluate and improve the estimation accuracy. Simulation examples verified these findings.

Learning from demonstration (LfD) has been increasingly used to encode robot tasks such that robots can achieve reproduction more flexibly in unstructured environments (e.g., households or factories). It is an effective alternative to preprogramming methods owing to its capacity of enabling robots to generalize to different situations. In this paper, we focus on LfD in the point-to-point movement case, where the dilemma of stability and accuracy exists. To avoid such a dilemma, we propose a learning approach that guarantees accuracy and stability simultaneously by means of constructed manifold immersion and submersion. We evaluate the proposed approach on two libraries of human handwriting motions (the LASA data set and a self-made GREEK data set) and on a set of experiments on the Barrett WAM robot.

The approach of dynamical system (DS) is promising for modeling robot motion, and provides a flexible means of realizing robot learning and control. Accuracy, stability, and learning speed are the three main factors to be considered when learning robot movements from human demonstrations with DS. Some approaches yield stable dynamical systems, but these may result in a poor reproduction performance, while some approaches yield good reproduction performance but are quite complex and time-consuming. In this paper, we address the accuracy-stability-speed issues simultaneously. We present a learning method named the fast and stable modeling for dynamical systems, which is based on the extreme learning machine to efficiently and accurately learn the parameters of the DS as well as to ensure the asymptotic stability at the target. We confirm the proposed approach by performing both 2-D tasks of learning handwriting motions and a set of robot experiments.

Nowadays, the control technology of the robotic manipulator with flexible joints (RMFJ) is not mature enough. The flexible-joint manipulator dynamic system possesses many uncertainties, which brings a great challenge to the controller design. This paper is motivated by this problem. In order to deal with this and enhance the system robustness, the full-state feedback neural network (NN) control is proposed. Moreover, output constraints of the RMFJ are achieved, which improve the security of the robot. Through the Lyapunov stability analysis, we identify that the proposed controller can guarantee not only the stability of flexible-joint manipulator system but also the boundedness of system state variables by choosing appropriate control gains. Then, we make some necessary simulation experiments to verify the rationality of our controllers. Finally, a series of control experiments are conducted on the Baxter. By comparing with the proportional-derivative control and the NN control with the rigid manipulator model, the feasibility and the effectiveness of NN control based on flexible-joint manipulator model are verified.

In clinics, customized archwires are demanded for lingual orthodontic treatment. However, only very experienced orthodontists can handle the manual appliance preparation. This pattern not only occupies lots of the orthodontist's labor time, but also can not ensure the accuracy of the appliances. Therefore, a robotic system was developed for automatic and accurate orthodontic archwire bending in our study. First, a method for customized archwire parameterization was developed. Second, an adaptive sampling-based bending planner with collision checker in time-varying environment was designed. Finally, a bending control strategy was used to eliminate the springback effect of the archwires and bending point shift during the bending process. A self-developed simulation platform based on Robot Operating System with MoveIt was used for preliminary validation of the proposed method. Physical experiments for multi-functional orthodontic bends on the robotic system were conducted as well. The results have shown that the developed robotic system using the proposed planning and control method was able to accomplish the automatic and accurate orthodontic archwire bending.

Teaching robots manipulative skills through human demonstration is an important research problem and can be used to quickly program robots in future manufacturing industries. To understand human demonstration, manipulative actions need to be recognized. To improve the recognition performance, we use three kinds of sensors to capture the motion and force involved in the fine manipulative actions. In addition, by taking advantage of the action/object correlation, the recognition accuracy can be further improved. In the proposed approach, important features for individual actions are selected first. Hidden Markov Models (HMMs) are employed to characterize the temporal changes. Then, a Bayesian model is adopted to model the object/action dependency. Our approach was evaluated through experiments on assembly tasks. The experimental results show that the proposed approach can recognize manipulative actions effectively.

In this paper, we propose a new approach for anomaly detection in video surveillance. This approach is based on a nonparametric Bayesian regression model built upon Gaussian process priors. It establishes a set of basic vectors describing motion patterns from low-level features via online clustering, and then constructs a Gaussian process regression model to approximate the distribution of motion patterns in kernel space. We analyze different anomaly measure criterions derived from Gaussian process regression model and compare their performances. To reduce false detections caused by crowd occlusion, we utilize supplement information from previous frames to assist in anomaly detection for current frame. In addition, we address the problem of hyperparameter tuning and discuss the method of efficient calculation to reduce computation overhead. The approach is verified on published anomaly detection datasets and compared with other existing methods. The experiment results demonstrate that it can detect various anomalies efficiently and accurately.

The papers in this special section present the most recent research work that showcases the state-of-the-art of human-centered computing and its potential applications in developing truly smart home automation systems.

Acquiring robot assembly skills through human demonstration is a challenging problem. To achieve this goal, not only the actions and objects have to be shown to the robot, but also the effect of the action needs to be estimated. Recognizing the subtle assembly actions is a non-trivial task, and it is difficult to estimate the effect of the action on the assembly parts due to the small part sizes. In this paper, with a RGB-D camera, we build a Portable Assembly Demonstration (PAD) system which can recognize the part/tool used, the action applied and the assembly state characterizing the spatial relationship between the parts. The experiment results proved that this PAD system can generate an assembly script with good accuracy in object and action recognition as well as assembly state estimation.

This paper addresses the robust H∞filtering problem for a class of Markov jump Lur'e systems with time-varying transition probabilities in discrete-time domain. The time-varying character of transition probabilities is considered to be finite piecewise homogeneous. A full-order filter is designed such that the resulting closed-loop systems are stochastically stable and have a guaranteed H∞performance index in terms of linear matrix inequalities. The effectiveness and potential of the developed results are verified through an example about a class of economic systems.

In this paper, we propose and develop a framework for automatic switching of manual driving and autonomous driving based on driver drowsiness detection. We first present the scale-down intelligent transportation system (ITS) testbed. This testbed has four main parts: an arena; an indoor localization system; automated radio controlled (RC) cars; and roadside monitoring facilities. Second, we present the drowsiness detection algorithm which integrates facial expression and racing wheel motion to recognize driver drowsiness. Third, a manual and autonomous driving switching mechanism is developed, which is triggered by the detection of drowsiness. Finally, experiments were performed on the ITS testbed to demonstrate the effectiveness of the proposed framework.

This paper introduces a dynamic model of the driver control strategy of lane-change behavior and applies it to trajectory planning in driver-assistance systems. The proposed model reflects the driver control strategies of adjusting longitudinal and latitudinal acceleration during the lane-change process and can represent different driving styles (such as slow and careful, as well as sudden and aggressive) by using different model parameters. We also analyze the features of the dynamic model and present the methods for computing the maximum latitudinal position and arrival time. Furthermore, we put forward an extended dynamic model to represent evasive lane-change behavior. Compared with the fifth-order polynomial lane-change model, the dynamic models fit actual lane-change trajectories better and can generate more accurate lane-change trajectories. We apply the dynamic models in emulating different lane-change strategies and planning lane-change trajectories for collision prediction. In the simulation, we use the models to compute the percentage of safe trajectories in different scenarios. The simulation shows that the maximum latitudinal position and arrival time of the generated lane-change trajectories can be good indicators of safe lane-change trajectories. In the field test, the dynamic models can generate the feasible lane-change trajectories and efficiently obtain the percentage of safe trajectories by computing the minimum gap and time to collision. The proposed dynamic model and module can be combined with the human-machine interface to help the driver easily identify safe lane-change trajectories and area.

In this paper, we study the stability region for a set of intelligent controllers developed by learning human expert control skills using support vector machines (SVMs). Based on the discrete-time system Lyapunov theory, a Chebychev points based estimation approach is proposed to evaluate the stability region, a key property of this set of SVM-based human learning controllers. One of such learning controllers has been implemented in vertical balance control of a dynamically stable, statically unstable single wheel mobile robot - Gyrover. The experimental results validate the proposed scheme for estimation of the stability region.

This paper introduces a real-time video surveillance system which detects human abnormal behaviors. We present two approaches to such a problem. The first one employs principal component analysis for feature selection and support vector machine for classification of human behaviors. The proposed feature selection method is based on the border information of four consecutive blobs. The second approach computes optical flow to obtain the velocity of each pixel for determining whether a human behavior is normal or not. Both algorithms are successfully implemented in crowded environments for detecting the human abnormal behaviors, such as (1) running people in a crowded environment, (2) bending down movement while most are walking or standing, (3) a person carrying a long bar and (4) a person waving hand in the crowd. Experimental results demonstrate the two methods proposed are robust and efficient in detecting human abnormal behaviors.

This paper studied convergence conditions for a class of intelligent controllers. We formulated conditions to verify that the learned closed-form control system is strongly stable under perturbations (SSUP). We developed an approach to evaluate the convergence quality of this class of controllers with representation of support vector machine. It has been implemented in a balance control of a dynamically stable, statically unstable single wheel robot. The experimental results verified the proposed convergence conditions and the theory upon which it is based.

The purpose of this work is to design a piecewise human learning control strategy for the autonomous control of dynamically stable systems in the following two cases. One case is in a single control process, the learning model is built up by combining some local neural networks. The other is that a desirable control target consists of some small control tasks which can be realized by human learning controllers individually. By estimating the stability region, we can guarantee the successful switch between two connected control pieces.

In this paper, we discuss the problem of how human control strategy can be represented as a parametric model using a Support Vector Machine (SVM), and how an SVM-based controller can be used to effectively control a dynamically stable system. We formulate the learning problem as a support vector regression and develop a new SVM learning structure to better implement human control strategy learning in control. The approach is fundamentally valuable in dealing with problems that normally dynamically stable robots experience, such as small sample data and local minima, and therefore is extremely useful in abstracting human controller for dynamic systems. The experimental study on the SVM approach with respect to other approaches clearly demonstrated the superiority of the SVM approach in terms of fidelity, efficiency and effectiveness in implementation.

In this paper, we study the interpolation approach in reducing the problem of small training sample sizes severely affecting the learning control performance of artificial neural networks when the dimension of the input variables is high. We use the local polynomial fitting approach to individually rebuild the time-variant functions of system states. Based on these functions, we can effectively produce new unlabelled training samples. We show that by using additional unlabelled samples, the learning control performance can be improved and, therefore, the overfitting phenomenon can be mitigated. Furthermore, experimental results verified these claims.

In this paper, we study the input selection in reducing the problem of the high dimension of input variables severely affecting the learning control performance of artificial neural networks. We first locally transform a nonlinear mapping problem into a nearly linear one by using the first-order derivatives of it. Then, we performed a local measure of the sensitivity of each of the model inputs (state variables) with respect to model outputs (human control inputs) under the least square error standard. Finally, based on voting, we defined a determination-rule to decide the importance order of the system state variables globally. By abstracting a human expert skill for controlling a dynamically stabilized robot: Gyrover, we validated the proposed approach.

The single-wheel gyroscopically-stabilized robot, Gyrover, is dynamically stable but statically unstable, with both first-order and second-order nonholonomic constraints. In this paper, based on the dynamic model of the robot, we first study the two classes of nonholonomic constrains associated with the system. We then propose control laws for the stabilization in Cartesian space and tracking line segments, while keeping its balance laterally.

The single wheel, gyroscopically stabilized robot, Gyrover, is dynamically stable but statically unstable, with both first-order and second-order nonholonomic constraints. In this paper, based on the dynamic model of the robot, we first study the two classes of nonholonomic constraints associated with the system. We then propose control laws for balance control in different cases.

In this paper, we describe the design of the hardware and software for a virtual bronchoscope with force feedback. A haptic interface allows surgeons to feel the reaction force of virtual pneumonic surgery as if they were touching the area directly. We present novel algorithms for haptic force rendering, and examine its ability to display force. The rendering algorithms have been interfaced with a force-reflecting device. This virtual haptic bronchoscope is of significance in training inexperienced doctors in pneumonic diagnosis and surgery.