Antagonistic pneumatic muscle (PM)-actuated wrist robots have great potential in rehabilitation and industrial applications. The antagonistic connection of PMs, which mimics the agonist-antagonist muscle pairs in human joints, provides substantial advantages such as improved joint stability and a better balance of torque disturbances. However, PM-actuated robots exhibit complex nonlinearities, such as hysteresis, creep, input delay, and time-varying parameters, while also confronting challenges such as external disturbances and coupling effects. In this paper, a switching non-singular terminal sliding mode control (NTSMC) method with a double-loop fuzzy neural network (DLFNN) is developed. This method enables the antagonistic PM-actuated wrist robots to achieve fast and precise tracking performance. Specifically, the lumped disturbances are estimated online using the DLFNN, which can adaptively adjust the weight of the inner and outer layers, achieving accurate approximation and robustness. Based on the estimated value of disturbances, a switching NTSMC is implemented to ensure that tracking errors converge to the origin within the fixed time. Switching functions guarantee fast convergence when the sliding surface errors are large. Meanwhile, switching functions ensure non-singularity as the sliding surface errors converge to the origin. Furthermore, joint angles and angular velocities are limited within the specific ranges by designing exponential constraint terms as time-varying proportional-differential gains, rather than traditional barrier functions that may induce excessive control inputs. Both detailed stability analysis and experimental validation demonstrate the effectiveness and adaptability of the proposed method.

This work is motivated by the complexity of conventional robot programming methods, which require a deep understanding of both robotics and software development. This situation presents significant challenges for nontechnical end-users who need to program a new robotic task. To address this issue, we propose a new robot programming method that enables humans to specify robot motion tasks using simple and intuitive body-language commands. To this end, we propose a triadic task decomposition framework to decompose complex robot motion tasks into three basic task units, i.e., location, operation, and motion. These units are paired with a set of body-language cues that enable end-users to specify a variety of actions and motions, providing an intuitive alternative to traditional robot programming methods. To program these units, we develop a multimodal fusion-based instruction-parsing algorithm with a dual-phase visual architecture, enabling temporally aware body-language recognition and high-accuracy cross-modal object localization. Furthermore, a semantics-adaptive task execution system is proposed, which incorporates an intent reasoning module that dynamically handles end-users' nonstandard programming sequences and incomplete task logic, enhancing programming success rates and system usability. We evaluate the proposed method on a dual-arm robotic platform and a humanoid robot through various pick-and-place tasks, and compare its performance with that of several baselines from the literature.

We introduce a novel framework with strong generalization capabilities for enabling humanoid robots to perform semantically grounded, context-aware, and physically realizable expressive behaviors, with the goal of enhancing end-user engagement in open-world human–robot interaction (HRI). To achieve this goal, we develop a new large language model–based human cognition module that interprets user dialogs to infer latent intent and generates high-level multi-modal behavior descriptions. These semantic representations are mapped to speech and motion outputs through a dual-stage embodied behavior generation pipeline. The pipeline consists of a shape adaptation module that maps human body motions into the robot’s kinematic space, followed by a motion retargeting module that generates executable joint trajectories under physical constraints. Additionally, the modular architecture enables seamless integration of state-of-the-art generative models and serves as a practical testbed for evaluating expressive behavior generation in real-world settings. We validate this system on a 58-DoF humanoid platform through both controlled video-based studies and live HRI experiments. The results show significant improvements over rule-based and handcrafted baselines in terms of expressiveness, behavioral appeal, and user engagement. This work helps bridging the gap between high-level language understanding and low-level robot control, thereby enabling scalable and human-aligned expressive behavior generation for embodied agents.

Safe and efficient operation of Autonomous Delivery Vehicles (ADVs) in dynamic multi-agent environments, such as university campuses and industrial parks, necessitates accurate trajectory prediction of interacting agents. Conventional autonomous navigation systems, however, rely heavily on reactive real-time perception and often fail to predict complex spatio-temporal interactions among heterogeneous agents. This limitation frequently leads to suboptimal motion planning outcomes and operational inefficiencies, including deadlock situations, in congested scenarios. This paper introduces Knowledge-Interaction-Temporal Network (KITNet), a novel trajectory prediction framework specifically designed for ADVs operating in such complex, dynamic settings. KITNet employs a hierarchical Graph Neural Network (GNN) architecture to model intricate interaction dynamics, incorporating a novel attention mechanism based on set theory for enhanced spatio-temporal feature extraction and prediction of diverse behavior patterns. We evaluate KITNet on several trajectory prediction benchmarks according to the different tailored behavior modes under the defined mode space, including the ETH/UCY pedestrian dataset, the NGSIM highway driving dataset, and the Argoverse 2 urban driving dataset. Our results demonstrate state-of-the-art prediction accuracy, outperforming or matching existing graph-based and recurrent approaches. Furthermore, we discuss the integration of KITNet’s predictive outputs into local motion planning modules, showing potential for significantly reducing conflict scenarios and optimizing trajectory execution for ADVs. These findings establish KITNet as a highly effective trajectory predictor for autonomous transport systems, critically advancing predictive navigation and bridging the gap between perception and robust intelligent decision-making in complex urban and campus environments.

Enabling humanoid robots to perform long-horizon mobile manipulation planning in real-world environments based on embodied perception and comprehension abilities has been a longstanding challenge. With the recent rise of large language models (LLMs), there has been a notable increase in the development of LLM-based planners. These approaches either utilize human-provided textual representations of the real world or heavily depend on prompt engineering to extract such representations, lacking the capability to quantitatively understand the environment, such as determining the feasibility of manipulating objects. To address these limitations, we present the Instruction-Augmented Long-Horizon Planning (IALP) system, a novel framework that employs LLMs to generate feasible and optimal actions based on real-time sensor feedback, including grounded knowledge of the environment, in a closed-loop interaction. Distinct from prior works, our approach augments user instructions into PDDL problems by leveraging both the abstract reasoning capabilities of LLMs and grounding mechanisms. By conducting various real-world long-horizon tasks, each consisting of seven distinct manipulatory skills, our results demonstrate that the IALP system can efficiently solve these tasks with an average success rate exceeding 80%. Our proposed method can operate as a high-level planner, equipping robots with substantial autonomy in unstructured environments through the utilization of multi-modal sensor inputs.

This paper presents a novel approach for visual servoing using neuromorphic event-based cameras. We extend the photometric Gaussian mixture model framework from frame-based to event-based vision by developing a mathematical formulation that bridges conventional models with the sparse, temporally-precise nature of event data. Our method transforms raw event streams into effective visual features through time surface representations, enabling visual servoing that leverages the microsecond temporal resolution and high dynamic range of event cameras. Evaluation using the N-Caltech101 dataset demonstrates excellent convergence characteristics and a high success rate (96.9%) across diverse object categories. Results confirm that our event-based photometric Gaussian mixture approach effectively exploits the temporal precision of event cameras while providing reliable performance for robot control tasks.

Drawing inspiration from the ability of fish to maintain efficient swimming over a wide range of speeds by tuning the stiffness of their tails, researchers have explored stiffness adjustment mechanisms in fish-like robots. Typically, existing mechanisms require extra actuators or power sources only for tuning stiffness, resulting in additional energy consumption and more complex structures. To address this, our study introduces an innovative fishtail featuring an online stiffness modulation mechanism that does not require additional actuators or power sources solely for stiffness adjustment. Through model-based simulations and experimental testing, we evaluated the effectiveness of the proposed method. The results demonstrate that the designed mechanism enables efficient swimming across a broader frequency range (0–4 Hz) compared to most servo-actuated platforms with adjustable stiffness reported in existing studies. The robot achieves a maximum average speed of 1.4 BL/s and a minimum cost of transport of 9.5 J/(m•kg).

Enabling robotic systems to perform long-horizon manipulation planning in real-world environments based on multimodal embodied perception and comprehension remains a longstanding challenge. Recent advancements in large language models (LLMs) have spurred the development of LLM-based planners; however, these approaches often rely on human-provided textual representations or extensive prompt engineering, lacking the ability to quantitatively interpret the environment. To overcome these limitations, we propose a novel framework that leverages LLMs and vision-language models (VLMs) to perform abstract reasoning and extract task-relevant representations from the environment using grounding mechanisms. To further enhance robotic capabilities, we introduce a systematic approach to constructing robotic skill libraries, enabling efficient generation of feasible and optimal actions. Unlike prior work, our LLM-based task planner reformulates user instructions into Planning Domain Description Language (PDDL) problems and employs Behavior Trees to represent the hierarchical structure of tasks, offering interpretable and modular task execution. Extensive evaluations on diverse real-world long-horizon manipulation tasks demonstrate the effectiveness of the proposed method, achieving an average success rate exceeding 80%. Furthermore, the framework functions as a high-level planner, empowering robots with substantial autonomy in unstructured environments by leveraging multimodal sensor inputs.

Accurately recognizing the structural regions of targeted objects is crucial for successful manipulation. In this study, we concentrate on the task of hanging crumpled garments on a rack, a common scenario in household environments. This context presents two primary challenges: (1) perceiving and grasping the structural regions of garments that exhibit severe deformations and self-occlusions; (2) adjusting the configuration of garments to fit the supporting components of the rack. To address these challenges, we propose a confidence-guided grasping strategy that actively seeks garment collars through handovers between dual robotic arms. In particular, we develop an autonomous data collection procedure in real-world settings to train the collar detection network. The exact grasping pose is determined through depth-aware contour extraction, and its success is evaluated based on a specially designed metric. Furthermore, we formulate the hanging task as one-shot imitation learning with an egocentric view. To precisely align the collar with the supporting item, we propose a two-step hanging strategy that involves coarse approaching followed by fine transformation. We perform comprehensive experiments and show that our framework notably enhances the success rate compared to existing methods.

Robot learning has attracted an ever-increasing attention by automating complex tasks, reducing errors, and increasing production speed and flexibility, which leads to significant advancements in manufacturing intelligence. However, its low training efficiency, limited real-time feedback, and challenges in adapting to untrained scenarios hinder its applications in smart manufacturing. Introducing a human role in the training loop, a practice known as human-in-the-loop (HITL) robot learning, can improve the performance of robots by leveraging human prior knowledge. Nonetheless, the exploration of HITL robot learning within the context of human-centric smart manufacturing remains in its infancy. This study provides a holistic literature review for understanding HITL robot learning within an industrial context from a human-centric perspective. A united structure is presented to encompass different aspects of human intelligence in HITL robot learning, highlighting perception, cognition, behavior, and notably, empathy. Then, the typical applications in manufacturing scenarios are analyzed to expand the research landscape for smart manufacturing. Finally, it introduces the empirical challenges and future directions for HITL robot learning in the next industrial revolution era.

Ultrasound-based medical examination usually requires establishing proper contact between an ultrasound probe and a human body that ensures the quality of ultrasound images. The scanning skills are quite challenging for a robot to learn primarily due to the complex coupling between the applied force profile and the resulting ultrasound image quality. While reinforcement learning appears as a powerful tool for learning complex robot skills, the deployment of these algorithms in medical robots demands special attention due to the evident safety concerns that arise from physical probe-tissue interactions. In this paper, we explicitly consider external constraints on the force magnitude when searching for the optimal policy parameters to enhance safety during ultrasound-guided robotic interventions. In particular, we study policy optimization under the framework of a constrained Markov decision process. The resulting gradient-based policy update is then subject to the involved constraints, which can be readily addressed by the primal-dual interior-point technique. In addition, upon the observation that policy update requires consecutive policies to be close to each other to have stable and robust performance with reinforcement learning algorithms, we design the learning rate of policy gradient from an imitation perspective. The performance of the proposed constraint-aware policy optimization method is validated with experiments of robotic ultrasound imaging for spinal diagnosisNote to Practitioners—This paper was motivated by the problem of safely learning the optimal interaction force strategy to facilitate robotic ultrasound imaging. Existing approaches to robotic ultrasound imaging usually empirically set a constant value for the scanning force, despite the fact the force strategy plays an important role in the quality of the ultrasound images. This paper suggests the usage of reinforcement learning to identify the optimal interaction force due to the complex acoustic coupling between the force and the ultrasound image quality. Specifically, we propose constraint-aware reinforcement learning in view of the safety-critical issues as a result of physical human-probe interaction. We then conduct a theoretical analysis of the proposed safe reinforcement learning, including monotonic improvement and policy value bound under mild assumptions. Preliminary real experiments on ultrasound imaging of the spine of a phantom for scoliosis assessment suggest that the proposed approach can safely learn the optimal scanning force without violating the prescribed force threshold. In the future, we would like to apply our approach to learning the optimal scanning force on different organs of interest of human subjects.

Fabric alignment is essential to key production processes such as cutting, sewing, and fusing in garment manufacturing. Traditionally, this task has relied heavily on the dexterity and expertise of skilled human workers. Although automated systems have been introduced, they often lack the flexibility required for complex alignment tasks. In this paper, we present a novel robotic fabric alignment framework that fully automates the process with high precision and adaptability. First, we propose a coarse-to-fine alignment strategy, where an initial imprecise target position is roughly computed based on a basic perception module and eye-to-hand calibration. This is followed by a sliding mode control (SMC)-based visual servoing approach (in an eye-in-hand configuration) to ensure a close-up view of feedback features for the fine alignment process. Additionally, we consider system disturbances estimated by a fuzzy logic system (FLS) and combine it with the controller to further enhance the system’s robustness. Finally, we developed an advanced end-effector equipped with force/torque (F/T) sensors and air-powered needle grippers for gentle fabric manipulation using admittance control. We validate our framework through a series of experiments that demonstrate its effectiveness in fabric alignment tasks.

The challenges inherent in long-horizon tasks in robotics persist due to the typical inefficient exploration and sparse rewards in traditional reinforcement learning approaches. To address these challenges, we have developed a novel algorithm, termed hlexplicit-implicit subgoal planning (EISP), designed to tackle long-horizon tasks through a divide-and-conquer approach. We utilize two primary criteria, feasibility and optimality, to ensure the quality of the generated subgoals. EISP consists of three components: a hybrid subgoal generator, a hindsight sampler, and a value selector. The hybrid subgoal generator uses an explicit model to infer subgoals and an implicit model to predict the final goal, inspired by way of human thinking that infers subgoals by using the current state and final goal as well as reason about the final goal conditioned on the current state and given subgoals. Additionally, the hindsight sampler selects valid subgoals from an offline dataset to enhance the feasibility of the generated subgoals. While the value selector utilizes the value function in reinforcement learning to filter the optimal subgoals from subgoal candidates. To validate our method, we conduct four long-horizon tasks in both simulation and the real world. The obtained quantitative and qualitative data indicate that our approach achieves promising performance compared to other baseline methods. These experimental results can be seen on the website https://sites.google.com/view/vaesi.

The recent emergence of Industry 5.0 underscores the need for increased autonomy in human–robot interaction (HRI), presenting both motivation and challenges in achieving resilient and energy-efficient production systems. To address this, in this article, we introduce a strategy for seamless collaboration between humans and robots in manufacturing and maintenance tasks. Our method enables smooth switching between temporary HRI (human-aware mode) and long-horizon automated manufacturing (fully automatic mode), effectively solving the human–robot coexistence problem. We develop a task progress monitor that decomposes complex tasks into robot-centric action sequences, further divided into three-phase subtasks. A trigger signal orchestrates mode switches based on detected human actions and their contribution to the task. In addition, we introduce a human agent coefficient matrix, computed using selected environmental features, to determine cut-points for reactive execution by each robot. To validate our approach, we conducted extensive experiments involving robotic manipulators performing representative manufacturing tasks in collaboration with humans. The results show promise for advancing HRI, offering pathways to enhancing sustainability within Industry 5.0. Our work lays the foundation for intelligent manufacturing processes in future societies, marking a pivotal step toward realizing the full potential of human–robot collaboration.

The increasing deployment of robots has significantly enhanced the automation levels across a wide and diverse range of industries. This article investigates the automation challenges of laser-based dermatology procedures in the beauty industry. This group of related manipulation tasks involves delivering energy from a cosmetic laser onto the skin with repetitive patterns. To automate this procedure, we propose to use a robotic manipulator and endow it with the dexterity of a skilled dermatology practitioner through a learning-from-demonstration framework. To ensure that the cosmetic laser can properly deliver the energy onto the skin surface of an individual, we develop a novel structured prediction-based imitation learning algorithm with the merit of handling geometric constraints. Notably, our proposed algorithm effectively tackles the imitation challenges associated with quasi-periodic motions, a common feature of many laser-based cosmetic tasks. The conducted real-world experiments illustrate the performance of our robotic beautician in mimicking realistic dermatological procedures. Our new method is shown to not only replicate the rhythmic movements from the provided demonstrations but also to adapt the acquired skills to previously unseen scenarios and subjects.

The exploration of energy-efficient active suspension control strategies for high-performance vibration suppression remains a critical challenge, particularly under partial-state measurements, uncertain dynamics, and external disturbances. This article proposes an innovative predefined-time output feedback control scheme for active vehicle suspension systems that achieves superior vibration mitigation with reduced energy consumption. By employing the time-varying scaling function technique, a predetermined-time extended state observer is developed to estimate unmeasurable velocities and lumped disturbance, while a second-order predefined-time filter is designed to avoid the explosion of computational complexity. Furthermore, using the effect characterization method and theX-mechanism reference dynamics, beneficial couplings/disturbances and nonlinearities can be reserved instead of direct cancellation, which leads to significant energy conservation up to 58% compared to other methods. Then, the predefined-time output feedback control is proposed to ensure that settling time can be arbitrarily user-specified using only one parameter, which is independent of initial conditions and control gains. Comparative experiments are performed to present the effectiveness and robustness of the proposed control method.

In nature, fish are excellent swimmers due to their flexible and precise control of tail, which allows them to freely transform between the smooth flapping and the motion of rapid response so that they can move with dexterity. Here, inspired by the versatile motion abilities of fish, a novel robotic fish has been developed, featuring the capability of adaptable bistability. Through tuning the bistability, the robot can acquire two locomotion modes, namely monostable and bistable modes, and it can also swim at different energy barrier that needs to be overcome to realize the bistable motion. The theoretical models are derived to facilitate the control of the robot and the understanding of its nonlinear behavior. The impact of the tunable bistability on the swimming and turning performance is investigated through extensive experiments. The study effectively demonstrates the robotic fish’s capability to swiftly and efficiently navigate through mode switches, enabled by its tunable bistability. This feature is essential for underwater robots to perform tasks in intricate environments.

Correctly understanding surrounding environments is a fundamental capability for autonomous driving. Semantic forecasting of bird-eye-view (BEV) maps can provide semantic perception information in advance, which is important for environment understanding. Currently, the research works on combining semantic forecasting and semantic BEV map generation is limited. Most existing work focuses on individual tasks only. In this work, we attempt to forecast semantic BEV maps in an end-to-end framework for future front-view (FV) images. To this end, we predict depth distributions and context features for FV input images and then forecast depth-context features for the future. The depth-context features are finally converted to the future semantic BEV maps. We conduct ablation studies and create baselines for evaluation and comparison. The results demonstrate that our network achieves superior performance.

Garment folding is a ubiquitous domestic task that is difficult to automate due to the highly deformable nature of fabrics. In this article, we propose a novel method of learning from demonstrations that enables robots to autonomously manipulate an assistive tool to fold garments. In contrast to traditional methods (that rely on low-level pixel features), our proposed solution uses a dense visual descriptor to encode the demonstration into a high-levelhand-object graph(HoG) that allows to efficiently represent the interactions between the manipulated tool and robots. With that, we leverage graph neural network to autonomously learn the forward dynamics model from HoGs, then, given only a single demonstration, the imitation policy is optimized with a model predictive controller to accomplish the folding task. To validate the proposed approach, we conducted a detailed experimental study on a robotic platform instrumented with vision sensors and a custom-made end-effector that interacts with the folding board.

Active vehicle suspension systems exhibit substantial vibration isolation capabilities, however, suffer from external disturbances, high energy consumption, risks of fault signals, limited transient performance, etc. In this paper, a predefined-time fault-tolerant control scheme is proposed for active suspensions to improve ride comfort and reliability, and enhance energy conservation. The reference X-dynamics together with a conditional disturbance cancellation scheme are developed to avoid the cancellation of beneficial nonlinearities and beneficial disturbances, respectively, which can reduce energy consumption without any optimization calculation or hardware alteration. Importantly, the error signals can converge to a predefined bound within the predefined time interval. Both the settling time and the residual bound can be arbitrarily user-defined, which are independent of initial states and control gains. Especially, to avoid singularity and alleviate chattering, a continuous piecewise function and a quadratic fraction inequality are constructed. The utilization of the proposed predefined-time fault-tolerant control facilitates satisfactory ride comfort with low energy cost. Experimental results are presented to validate the superior control performance of the designed control scheme.

The automatic monitoring/tracking of environmental boundaries by multi-agent systems is a fundamental problem that has many practical applications. In this paper, we address this problem with formation control techniques based on parame tric curves that represent the boundary’s feedback shape. For that, we approximate the curve with truncated Fourier series, whose finite coefficients are utilized to characterize the curve’s shape and to automatically distribute the agents along it. These feedback Fourier coefficients are exploited to design a new type of formation controller that drives the agents to form desired curves. A detailed stability analysis is provided for the proposed control methodology, considering both fixed and switching multi-agent topologies. The reported numerical simulation and experimental studies demonstrate the performance and feasibility of our new method to track closed boundaries of different shapes.

This article presents the development of a multisensor user interface to facilitate the instruction of arc welding tasks. Traditional methods to acquire hand-eye coordination skills are typically conducted through one-to-one instruction, where trainees must wear protective helmets and conduct several tests. These approaches are inefficient as the harmful light emitted from the electric arc impedes the close monitoring of the process. Practitioners can only observe a small bright spot. To tackle these problems, recent training approaches have leveraged virtual reality to safely simulate the process and visualize the geometry of the workpieces. However, the synthetic nature of these types of simulation platforms reduces their effectiveness as they fail to comprise actual welding interactions with the environment, which hinders the trainees' learning process. To provide users with a real welding experience, we have developed a new multisensor extended reality platform for arc welding training. Our system is composed of: 1) An HDR camera, monitoring the real welding spot in real time. 2) A depth sensor, capturing the 3-D geometry of the scene; and 3) A head-mounted VR display, visualizing the process safely. Our innovative platform provides users with a “bot trainer,” virtual cues of the seam geometry, automatic spot tracking, and performance scores. To validate the platform's feasibility, we conduct extensive experiments with several welding training tasks. We show that compared with the traditional training practice and recent virtual reality approaches, our automated multisensor method achieves better performances in terms of accuracy, learning curve, and effectiveness.

Developing a good control strategy for biomimetic robots is challenging. Robust control methods require an accurate model of the robot. Nowadays, model-free methods are being extensively explored for the control and navigation of terrestrial robots. In this paper, we consider a novel deep reinforcement learning-based model-free swimming control for our bio-inspired robotic tadpole. To realize this, we utilize dynamic motion primitives, which can represent a large range of motion behaviors, and combine them with a decoupled reinforcement learning framework. The proposed architecture optimizes the motion primitives first to develop a travelling wave undulation pattern in the tail and then to navigate the robot along different predefined paths. Through this framework, effective swimming gait emerges, and the robot is able to navigate well on the surface of water. This framework combines the optimization potential of deep reinforcement learning with stability and generalization properties of dynamic motion primitives. We train and test our method on a simulated model of the robot to demonstrate the effectiveness of the method and also conduct experimental testing on the real robot to verify the results.

While the original goal for developing robots is replacing humans in dangerous and tedious tasks, the final target shall be completely mimicking the human cognitive and motor behavior. Hence, building detailed computational models for the human brain is one of the reasonable ways to attain this. The cerebellum is one of the key players in our neural system to guarantee dexterous manipulation and coordinated movements as concluded from lesions in that region. Studies suggest that it acts as a forward model providing anticipatory corrections for the sensory signals based on observed discrepancies from the reference values. While most studies consider providing the teaching signal as error in joint-space, few studies consider the error in task-space and even fewer consider the spiking nature of the cerebellum on the cellular-level. In this study, a detailed cellular-level forward cerebellar model is developed, including modeling of Golgi and Basket cells which are usually neglected in previous studies. To preserve the biological features of the cerebellum in the developed model, a hyperparameter optimization method tunes the network accordingly. The efficiency and biological plausibility of the proposed cerebellar-based controller is then demonstrated under different robotic manipulation tasks reproducing motor behavior observed in human reaching experiments.

In this article, we introduce a new sensor-based control method that regulates (by means of robot motion) the temperature of objects that are subject to a radiative heat source. This valuable sensorimotor capability is needed in many industrial, dermatology, and field robot applications, and it is an essential component for creating machines with advanced thermomotor intelligence. To this end, we derive a geometric-thermal-motor model, which describes the relation between the robot’s active configuration and the produced dynamic thermal response. We then use the model to guide the design of two new thermal servoing controllers (one model-based and one adaptive), and analyze their stability with Lyapunov theory. To validate our method, we report a detailed experimental study with a robotic manipulator conducting autonomous thermal servoing tasks. We show that the temperature of multiple objects with unknown thermophysical properties attached to the same end-effector can be effectively regulated by controlled robot motion. Although thermal sensing is a mature technology in many industrial thermal engineering applications, its use as a feedback signal for robot control has not been sufficiently studied in the literature. To the best of our knowledge, this is the first time that temperature regulation is formulated as a motion control problem for robots.

The cerebellum plays a distinctive role within our motor control system to achieve fine and coordinated motions. While cerebellar lesions do not lead to a complete loss of motor functions, both action and perception are severally impacted. Hence, it is assumed that the cerebellum uses an internal forward model to provide anticipatory signals by learning from the error in sensory states. In some studies, it was demonstrated that the learning process relies on the jointspace error. However, this may not exist. This work proposes a novel fully spiking neural system that relies on a forward predictive learning by means of a cellular cerebellar model. The forward model is learnt thanks to the sensory feedback in task-space and it acts as a Smith predictor. The latter predicts sensory corrections in input to a differential mapping spiking neural network during a visual servoing task of a robot arm manipulator. In this paper, we promote the developed control system to achieve more accurate target reaching actions and reduce the motion execution time for the robotic reaching tasks thanks to the cerebellar predictive capabilities.

In this paper, we present a robotic system to automatically perform defect inspection tasks over free-form specular surfaces, which the image acquisition sub-system is equipped with a 6-DOF robot manipulator to achieve flexible scanning. Given the mesh model of the workpiece, we implement K-means based region segmentation algorithm on the point cloud after preprocessing. Then, we take the smooth regions as input to plan the scanning path. A projection registration method that robustly localizes the object in the robot’s frame is proposed for real-time workpiece localization. According to the optical features of the high-resolution line scan, we design an image processing pipeline to detect defects from the captured images. We report a detailed experimental study to validate the proposed methodology.

Emerging applications of robotic manipulation of deformable objects have opened up new challenges in robot control. While several control techniques have been developed to manipulate deformable objects, the performance of existing methods is commonly limited by two issues: 1) implicit assumption that the physical contact between the end-effector and the object is always maintained, and 2) requirements of exact parameters of deformation model, which are difficult to obtain. This paper presents a new control scheme for robotic manipulation of deformable objects, which allows the robot to automatically contact then actively deform the deformable object by assessing the status of deformation in real time. Instead of designing multiple controllers and switching among them, the proposed method smoothly and stably integrates two control phases (i.e. region reaching and active deforming) into a single controller. The stability of the closed-loop system is rigorously proved with the consideration of the uncertain deformation model and uncalibrated cameras. Hence, the proposed control scheme enhances the autonomous capability of active deformable object manipulation. Experimental studies are conducted with different initial conditions to demonstrate the performance of the proposed controller.

This paper addresses the design of a vision-based method to automatically deform soft objects into desired two-dimensional shapes with robot manipulators. The method presents an innovative feedback representation of the object's shape (based on a truncated Fourier series) and effectively exploits it to guide the soft object manipulation task. A new model calibration scheme that iteratively approximates a local deformation model from vision and motion sensory feedback is derived; this estimation method allows us to manipulate objects with unknown deformation properties. Pseudocode algorithms are presented to facilitate the implementation of the controller. Numerical simulations and experiments are reported to validate this new approach.

In this paper, we address the 3D position control of RCM-constrained instruments with monocular cameras. To compute the instrument's position from a single 2D image, we develop an innovative gradient descent algorithm which rotates and translates a line segment (over the plane spanned by the imaged instrument and the optical centre) until it best aligns with the manipulated tool. In contrast with other approaches in the literature, our algorithm only requires to simultaneously observe two feature points; the proposed iterative algorithm is not based on the exact solution, therefore it can still work with noisy image measurements. We derive a kinematic controller that uses the proposed position estimator to guide the 3D motion of a robotic instrument with a monocular camera. We evaluate the performance of our approach with numerical simulations and experiments.

In this paper, we address the problem of the image-based 3D pose computation of a semi-circle suturing needle using monocular image feedback for laparoscopy. We propose a constrained two-degree-of-freedom (2-DOF) geometry-based modelling method to parametrise the needle's 6-DOF pose, including depth information. The modelling solely relies on the simultaneous observation of the needle's apparent tip and junction. No external markers are needed for extra constraints. An adaptive controller combining gradient descent and vector-flow method is introduced to iteratively guide the needle's initial guessing pose to its real pose by minimizing image-based position errors. Experiments have been conducted using both numerical simulations and simulated laparoscopic scenarios to evaluate the performance of the algorithm.

In this paper, we present a new feedback method to automatically servo-control the 3-D shape of soft objects with robotic manipulators. The soft object manipulation problem has recently received a great deal of attention from robotics researchers because of its potential applications in, e.g., food industry, home robots, medical robotics, and manufacturing. A major complication to automatically control the shape of an object is the estimation of its deformation properties, which determines how the manipulator's motion actively transforms into deformations. Note that these properties are rarely known beforehand, and its offline parametric identification is difficult and/or impractical to conduct in many applications. To cope with this issue, we developed a new algorithm that computes in real time the unknown deformation parameters of a soft object; this algorithm provides a valuable adaptive behavior to the deformation controller, something we cannot achieve with traditional fixed-model approaches. In contrast with most controllers in the literature, our new method can explicitly servo-control 3-D deformations (and not just 2-D image projections) in an entirely model-free way. To validate the proposed adaptive controller, we present a detailed experimental study with robotic manipulators.

In this paper, we propose a novel design of compliant safe joint, which has flexibility when the work load exceeds a predefined threshold. The compliance is generated by a spring. We design a special transmission mechanism to convert axial motion into circumferential motion such that the linear compliance can be converted into circular one. When the end-effector of a surgical robot actuated by the compliant safe joints collides with patient's body, the compliance of the joints will protect the patient by absorbing part of the collision energy. Because of the system's special mechanical structure, the control methods should be different when it works under different states. We propose a simple algorithm to choose control methods so that the system can work both under rigid and flexible states with different controllers. We have built a prototype to validate the design and the controller.

This paper presents the development of an endoscope manipulator with passive and active structures for functional endoscopic sinus surgery (FESS). The 5-DoF passive structure has three translations and two rotations (T3R2) that allows the surgeon to manually place the endoscope near to the entry point during. The 4-DoF motorized structure (T2R2) actively controls the endoscope's position based on the surgeon's input commands. We analyze the reciprocal screw of the passive and active structures. The motion control system is based on a real-time Linux kernel that processes the commands from the surgeon and controls the manipulator's active joints. A user control interface based on an IMU fastened on the surgeon's foot is developed; this interface measures the foot's posture and through a series of gestures, it provides the desired pan/tilt/zoom motions of the camera. The developed endoscope manipulator allows the surgeon to conduct ‘two-hand’ operations while retaining direct control of the camera. We present an experimental study to validate the performance of the robotic prototype.

In this paper, we address the positioning problem of remote centre of motion (RCM) mechanisms with uncalibrated image feedback from a monocular camera. Nowadays, RCM mechanisms are widely used in minimally invasive robotic surgery due to their ability to distally rotate a tool around a fixed entry port; note that in most surgical applications, the tools are typically controlled by manual/teleoperated motion commands given by a human user. In this paper, we depart from the traditional manual control scheme and derive sensor-based methods to automatically position the manipulated tool using real-time image feedback. To this end, we first characterise the mechanism's 3-DOF configuration with the angle of the image projected tool and scalar distances between feature points. To cope with uncertainty in the camera's calibration parameters, we propose two gradient descent estimators that adaptively compute the unknown Jacobian matrix; the stability of these algorithms is proved with Lyapunov theory. Finally, we derive a kinematic image-based controller and evaluate its performance with several positioning experiments.

In this paper, we present a new robotic uterine positioner for total laparoscopic hysterectomy. The robot is designed to actively position the patient's uterus during surgery, a lengthy and tedious task that is traditionally performed by a human assistant. Safety is simply the most important concern when developing robots for surgical purposes; we address this concern in the design of our robot from a mechanical perspective. To this end, we develop a 3-DOF robotic uterine positioner with an in-body remote center of motion (RCM); this key feature allows to prevent injuries to the patient when large motions occur at the cervix. A linearly-actuated arc-guided RCM mechanism is introduced to guarantee the rigidity and stability of the robot; The system's design allows to manipulate the uterus in a decoupled manner, thus control complexity can be reduced. Passive safety mechanisms are also implemented in all DOF of the robot in order to limit the interaction forces with the patient. Experiments, including an ex-vivo test conducted with cadaver, are conducted to verify the robot's performance.

In this paper, we address the set-point deformation control of elastic objects by fully-constrained grippers. We propose an uncalibrated Lyapunov-based algorithm that iteratively estimates the deformation Jacobian matrix, with no prior knowledge of the deformation and camera models. With this new method we show how, by combining pose information of the grippers with several visual measurements, we can independently control elastic deformations of unknown objects. We report experiments with a 6-DOF robot manipulator to validate this control approach.

In this paper, we address the visual servoing problem of robot manipulators with eye-in-hand cameras. To servo-control the image position of a feature point, traditional image-based controllers require the computation of the point's position vector with respect to the camera's frame. However, when the point's location is uncertain, the stability of traditional visual servoing controllers can not be rigorously guaranteed. To contribute to this problem, in this paper we present two new kinematic image-based controllers that do not require the exact location of static features. The first controller is a depth-free method that uses the camera's calibration matrix and visual feedback to compute a quasi-position vector of the feature point. The second controller uses adaptive control techniques to iteratively estimate the calibration matrix and the point's position vector. We prove the stability of both servo-controllers using Lyapunov theory, and present experimental results to evaluate its performance.

Despite the recent progress in physically interactive and surgical robotics, the active deformation of compliant objects remains an open problem. The main obstacle comes from the difficulty to identify/estimate the object's deformation properties. This paper presents a new visually servoed deformation controller for unknown elastic objects. The control law is designed using the passivity-based framework. The proposed method exploits visual feedback to iteratively estimate the deformation Jacobian matrix, avoiding any identification steps. We prove that even in the presence of inexact estimations, the controller ensures input-to-state stability (i.e. dissipativity) with respect to time-varying disturbances. Finally, an experimental study with several deformation tasks is presented to validate the theory.

In this paper, we propose a new vision-based controller to actively deform an unknown elastic object. Note that most deformation controllers in the literature require a-priori knowledge of the object's deformation properties. In contrast to this trend, we present a new Lyapunov-based method that online estimates the unknown deformation Jacobian matrix, avoiding any model identification or calibration steps. To achieve the desired object's deformation, we derive an innovative dynamic-state feedback velocity control law using the passivity-based framework. We present a detailed experimental study to validate the feasibility of our deformation controller.

Despite the recent progress in physically interactive and surgical robotics, the active deformation of compliant objects remains an open problem. The main obstacle to its implementation comes from the difficulty to identify or estimate the object's deformation model. In this paper, we propose a novel vision-based deformation controller for robot manipulators interacting with unknown elastic objects. We derive a new dynamic-state feedback velocity control law using the passivity-based framework. Our method exploits visual feedback to estimate the deformation Jacobian matrix in real time, avoiding any model identification steps. We prove that even in the presence of inexact estimations, the closed-loop dynamical system ensures input-to-state stability (i.e., full dissipativity) with respect to external disturbances. An experimental study with several deformation tasks is presented to validate the theory.

In this paper, we analyse and present a control approach using the energy shaping formulation to explicitly regulate the applied force of a robot manipulator in contact with a purely elastic environment. The potential energy of the robot-environment system is shaped in a way that its local equilibrium implies the application of the desired force onto the contact surface. This potential shaping is realised based on the available contact force feedback. Also, it is shown how force feedback can be employed to modulate the amount of energy externally injected by a source subsystem, and with this achieve exact force convergence. Experimental results are presented to validate this approach.