The Dynamic Vision Sensor (DVS) is a distinctive visual sensor that exclusively responds to alterations in pixel brightness, enabling the real-time capture of swift and subtle movements with reduced power consumption and data bandwidth requirements. This paper proposes a DVS-aware visual perception method and presents its application for pose estimation of mobile robots. Specifically, a new marker is designed to provide pose reference data that leverages the inherent advantages of DVS more effectively. Moreover, we formulate a pose recognition system incorporating DVS, an algorithm based on Spiking Convolutional Neural Networks (SCNN) and a neuromorphic computing accelerator (Lynxi HS110). Such a formulation can well explore the DVS's advantages, as its event-triggered feature matches the nature of SCNN while the neuromorphic hardware enables efficient, low-power execution, making the system highly suitable for real-time embedded applications. Comparative analysis with traditional ARcode-based pose recognition methods reveals that our innovative approach demonstrates significant advantages in recognition speed and energy efficiency. The whole system is deployed on mobile robots and evaluated in real-world scenarios.

Ultrasound scanning is a critical imaging technique for real-time, non-invasive diagnostics. However, variations in patient anatomy and complex human-in-the-loop interactions pose significant challenges for autonomous robotic scanning. Existing ultrasound scanning robots are commonly limited to relatively low generalization and inefficient data utilization. To overcome these limitations, we present UltraDP, a Diffusion-Policy-based method that receives multi-sensory inputs (ultrasound images, wrist camera images, contact wrench, and probe pose) and generates actions that are fit for multi-modal action distributions in autonomous ultrasound scanning of carotid artery. We propose a specialized guidance module to enable the policy to output actions that center the artery in ultrasound images. To ensure stable contact and safe interaction between the robot and the human subject, a hybrid force-impedance controller is utilized to drive the robot to track such trajectories. Also, we have built a large-scale training dataset for carotid scanning comprising 210 scans with 460k sample pairs from 21 volunteers of both genders. By exploring our guidance module and DP’s strong generalization ability, UltraDP achieves a 95% success rate in transverse scanning on previously unseen subjects, demonstrating its effectiveness.

While magnetic micro-robots have demonstrated significant potential across various applications, including drug delivery and microsurgery, the open issue of precise navigation and control in complex fluid environments is crucial for in vivo implementation. This paper introduces a novel flow-aware navigation and control strategy for magnetic micro-robots that explicitly accounts for the impact of fluid flow on their movement. First, the proposed method employs a Physics-Informed U-Net (PI-UNet) to refine the numerically predicted fluid velocity using local observations. The predicted velocity is then incorporated into a flow-aware A* path planning algorithm, ensuring efficient navigation while mitigating flow-induced disturbances. Finally, a control scheme is developed to compensate for the predicted fluid velocity, thereby optimizing the micro-robot’s performance. A series of simulation studies and real-world experiments are conducted to validate the efficacy of the proposed approach. This method enhances both planning accuracy and control precision, expanding the potential applications of magnetic micro-robots in fluid-affected environments typical of many medical scenarios.

The solution exchange process is crucial in cell cryopreservation, an assistive reproductive technique that enhances reproductive autonomy and helps women overcome infertility challenges. Such a task is time-critical in the sense that the duration of cell exposure to the solutions significantly impacts cell viability. In this paper, a new micromanipulation system has been developed to automate such a task, where the contributions can be summarized as follows. This paper addresses the challenge of tracking cell positions within a microfluidic chip, due to the limitations of the microscope’s field of view (FOV). By utilizing a region control approach with visual feedback, the proposed method ensures that the cell remains centered in the image. Additionally, a real-time tracking method based on correlation filtering is presented, which precisely localizes cells and controls the syringe pump flow rate to mitigate delays and prevent cell loss. Experimental results illustrate the consistent positioning of the micro objects/cells at each step, the satisfactory success rate, and the robustness to the loss of vision feature. Integrated with novel manipulation strategies, our intelligent manipulation system offers a promising solution for in vitro fertilization (IVF), characterized by an embryologist-centered configuration and standardized robotic manipulation. This study aims to standardize clinical cryopreservation by transitioning from manual operation to a more efficient and reliable automated process. Furthermore, this approach can simplify procedures and enhance oocyte viability. Note to Practitioners—The motivation for this study was to propose an automatic method to address the challenges of manual cell cryopreservation, such as high operational complexity, low efficiency, and significant cell damage. Existing automated cell cryopreservation methods have shortcomings as they fail to simultaneously achieve comprehensive cell tracking and avoid damage caused by direct contact between cells and microtools. Additionally, the high complexity of robotic operations makes clinical application difficult. To overcome these limitations, this study introduces a novel robotic micromanipulation method based on microfluidic chip technology. This method enables full-process cell tracking, FOV tracking of the motorized stage, and non-contact manipulation during the exchange process of cells and solutions. Experimental results showed that the proposed method outperformed existing manual methods in terms of oocyte viability. For fertility experts and medical staff, such a robotic micromanipulation system can be combined with other automatic machines to provide a solution to cell surgery, achieving higher accuracy and efficiency.

A new control pipeline has been proposed for the Rope-Base Soft-tip Manipulator (RBSM) to execute the surface contact task to prevent the jamming and slipping problems. The control pipeline enables smooth surface-to-surface contact for the RBSM using only force sensors, eliminating the dependence on additional pose measurement of the window surface plane and soft-tip deformation information. The pipeline consists of three steps: free contact step implemented by an exponential force shape controller to avoid force overshoot to the window surface; orientation refinement step implemented by a force and torque combined controller to make the RBSM cleaning head surface stable adapt to the smooth window surface; and finally, a release normal force step to reduce head jamming and region covering with a pre-defined vibration-less cleaning trajectory for smooth cleaning on the slippery window surface. The proposed pipeline has been validated in a Rope base Cleaning Manipulator prototype to clean a common window surface. The force and velocity curves during the cleaning experiment show that the proposed method achieves smooth scraping and cleaning under unknown initial significant errors in surface orientation.

A typical application of upper-limb exoskeleton robots is deployment in rehabilitation training, helping patients to regain manipulative abilities. However, as the patient is not always capable of following the robot, safety issues may arise during the training. Due to the bias in different patients, an individualized scheme is also important to ensure that the robot suits the specific conditions (e.g., movement habits) of a patient, hence guaranteeing effectiveness. To fulfill this requirement, this paper proposes a new motion planning scheme for upper-limb exoskeleton robots, which drives the robot to provide customized, safe, and individualized assistance using both human demonstration and interactive learning. Specifically, the robot first learns from a group of healthy subjects to generate a reference motion trajectory via probabilistic movement primitives (ProMP). It then learns from the patient during the training process to further shape the trajectory inside a moving safe region. The interactive data is fed back into the ProMP iteratively to enhance the individualized features for as long as the training process continues. The robot tracks the individualized trajectory under a variable impedance model to realize the assistance. Finally, the experimental results are presented in this paper to validate the proposed control scheme.

Magnetic microrobots can be navigated by an external magnetic field to autonomously move within living organisms with complex and unstructured environments. Potential applications include drug delivery, diagnostics, and therapeutic interventions. Existing techniques commonly impart magnetic properties to the target object, or drive the robot to contact and then manipulate the object, both probably inducing physical damage. This paper considers a non-contact formulation, where the robot spins to generate a repulsive field to push the object without physical contact. Under such a formulation, the main challenge is that the motion model between the input of the magnetic field and the output velocity of the target object is commonly unknown and difficult to analyze. To deal with it, this paper proposes a data-driven-based solution. A neural network is constructed to efficiently estimate the motion model. Then, an approximate model-based optimal control scheme is developed to push the object to track a time-varying trajectory, maintaining the non-contact with distance constraints. Furthermore, a straightforward planner is introduced to assess the adaptability of non-contact manipulation in a cluttered unstructured environment. Experimental results are presented to show the tracking and navigation performance of the proposed scheme.

Current orthopedic robotic systems largely focus on navigation, aiding surgeons in positioning a guiding tube but still requiring manual drilling and screw placement. The automation of this task not only demands high precision and safety due to the intricate physical interactions between the surgical tool and bone but also poses significant risks when executed without adequate human oversight. As it involves continuous physical interaction, the robot should collaborate with the surgeon, understand the human intent, and always include the surgeon in the loop. To achieve this, this paper proposes a new cognitive human–robot collaboration framework, including the intuitive AR-haptic human–robot interface, the visual-attention-based surgeon model, and the shared interaction control scheme for the robot. User studies on a robotic platform for orthopedic surgery are presented to illustrate the performance of the proposed method. The results demonstrate that the proposed human– robot collaboration framework outperforms full robot and full human control in terms of safety and ergonomics.

Dexterous in-hand manipulation is an essential skill of production and life. However, the highly stiff and mutable nature of contacts limits real-time contact detection and inference, degrading the performance of model-based methods. Inspired by recent advances in contact-rich locomotion and manipulation, this paper proposes a novel model-based approach to control dexterous in-hand manipulation and overcome the current limitations. The proposed approach has an attractive feature, which allows the robot to robustly perform long-horizon in-hand manipulation without predefined contact sequences or separate planning procedures. Specifically, we design a high-level contact-implicit model predictive controller to generate real-time contact plans executed by the low-level tracking controller. Compared to other model-based methods, such a long-horizon feature enables replanning and robust execution of contact-rich motions to achieve large displacements in-hand manipulation more efficiently; Compared to existing learning-based methods, the proposed approach achieves dexterity and also generalizes to different objects without any pre-training. Detailed simulations and ablation studies demonstrate the efficiency and effectiveness of our method. It runs at 20Hz on the 23-degree-of-freedom, long-horizon, in-hand object rotation task.

The ultrasound scanning robot operates in environments where frequent human-robot interactions occur. Most existing control methods for ultrasound scanning address only one specific interaction situation or implement hard switches between controllers for different situations, which compromises both safety and efficiency. In this paper, we propose a unified interaction control framework for ultrasound scanning robots capable of handling all common interactions, distinguishing both human-intended and unintended types, and adapting with appropriate compliance. Specifically, the robot suspends or modulates its ongoing main task if the interaction is intended, e.g., when the doctor grasps the robot to lead the end effector actively. Furthermore, it can identify unintended interactions and avoid potential collision in the null space beforehand. Even if that collision has happened, it can become compliant with the collision in the null space and try to reduce its impact on the main task (where the scan is ongoing) kinematically and dynamically. The multiple situations are integrated into a unified controller with a smooth transition to deal with the interactions by exhibiting human-intention-aware compliance. Experimental results validate the framework’s ability to cope with all common interactions including intended intervention and unintended collision in a collaborative carotid artery ultrasound scanning task.

Most research on deformable linear object (DLO) manipulation assumes rigid grasping. However, beyond rigid grasping and re-grasping, in-hand following is also an essential skill that humans use to dexterously manipulate DLOs, which requires continuously changing the grasp point by in-hand sliding while holding the DLO to prevent it from falling. Achieving such a skill is very challenging for robots without using specially designed but not versatile end-effectors. Previous works have attempted using generic parallel grippers, but their robustness is unsatisfactory owing to the conflict between following and holding, which is hard to balance with a one-degree-of-freedom gripper. In this work, inspired by how humans use fingers to follow DLOs, we explore the usage of a generic dexterous hand with tactile sensing to imitate human skills and achieve robust in-hand DLO following. To enable the hardware system to function in the real world, we develop a framework that includes Cartesian-space arm-hand control, tactile-based in-hand 3-D DLO pose estimation, and task-specific motion design. Experimental results demonstrate the significant superiority of our method over using parallel grippers, as well as its great robustness, generalizability, and efficiency.

One of the typical purposes of using lower-limb exoskeleton robots is to provide assistance to the wearer by supporting their weight and augmenting their physical capabilities according to a given task and human motion intentions. The generalizability of robots across different wearers in multiple tasks is important to ensure that the robot can provide correct and effective assistance in actual implementation. However, most lower-limb exoskeleton robots exhibit only limited generalizability. Therefore, this article proposes a human-in-the-loop learning and adaptation framework for exoskeleton robots to improve their performance in various tasks and for different wearers. To suit different wearers, an individualized walking trajectory is generated online using dynamic movement primitives and Bayes optimization. To accommodate various tasks, a task translator is constructed using a neural network to generalize a trajectory to more complex scenarios. These generalization techniques are integrated into a unified variable impedance model, which regulates the exoskeleton to provide assistance while ensuring safety. In addition, an anomaly detection network is developed to quantitatively evaluate the wearer's comfort, which is considered in the trajectory learning procedure and contributes to the relaxation of conflicts in impedance control. The proposed framework is easy to implement, because it requires proprioceptive sensors only to perform and deploy data-efficient learning schemes. This makes the exoskeleton practical for deployment in complex scenarios, accommodating different walking patterns, habits, tasks, and conflicts. Experiments and comparative studies on a lower-limb exoskeleton robot are performed to demonstrate the effectiveness of the proposed framework.

Accurately and robustly estimating the state of deformable linear objects (DLOs), such as ropes and wires, is crucial for DLO manipulation and other applications. However, it remains a challenging open issue due to the high dimensionality of the state space, frequent occlusions, and noises. This paper focuses on learning to robustly estimate the states of DLOs from single-frame point clouds in the presence of occlusions using a data-driven method. We propose a novel two-branch network architecture to exploit global and local information of input point cloud respectively and design a fusion module to effectively leverage the advantages of both methods. Simulation and real-world experimental results demonstrate that our method can generate globally smooth and locally precise DLO state estimation results even with heavily occluded point clouds, which can be directly applied to real-world robotic manipulation of DLOs in 3-D space.

Manipulating deformable linear objects (DLOs) to achieve desired shapes in constrained environments with obstacles is a meaningful but challenging task. Global planning is necessary for such a highly-constrained task; however, accurate models of DLOs required by planners are difficult to obtain owing to their deformable nature, and the inevitable modeling errors significantly affect the planning results, probably resulting in task failure if the robot simply executes the planned path in an open-loop manner. In this paper, we propose a coarse-to-fine framework to combine global planning and local control for dual-arm manipulation of DLOs, capable of precisely achieving desired configurations and avoiding potential collisions between the DLO, robot, and obstacles. Specifically, the global planner refers to a simple yet effective DLO energy model and computes a coarse path to find a feasible solution efficiently; then the local controller follows that path as guidance and further shapes it with closed-loop feedback to compensate for the planning errors and improve the task accuracy. Both simulations and real-world experiments demonstrate that our framework can robustly achieve desired DLO configurations in constrained environments with imprecise DLO models, which may not be reliably achieved by only planning or control.

Exoskeleton robots provide assistive forces to suit the human subject via physical human-robot interaction. During the closely-coupled interaction, a mismatch between the wearer and the robot may result in physical conflict, which could affect assistance efficiency or even compromise safety. Therefore, such conflicts should be accurately detected and then properly relaxed by adjusting the robot's action. This paper proposes a new learning scheme to detect physical conflicts between humans and robots. The constructed learning network receives multi-modal information from proprioceptive sensors and then outputs the anomaly score to specify the physical conflict, which score is further used to continuously adjust the robot impedance to ensure a safe and efficient interaction. Such a formulation allows the robot to explore the semantic information during the interaction (e.g., gait phases, imbalance, human fatigue) and hence react properly to the physical conflict. Experimental results and comparative studies on a lower-limb exoskeleton robot are presented to illustrate that the proposed learning scheme can deal with physical conflicts in a faster and more accurate manner.

Non-prehensile manipulation methods usually use a simple end effector, e.g., a single rod, to manipulate the object. Compared to the grasping method, such an end effector is compact and flexible, and hence it can perform tasks in a constrained workspace; As a trade-off, it has relatively few degrees of freedom (DoFs), resulting in an under-actuation problem with complex constraints for planning and control. This paper proposes a new non-prehensile manipulation method for the task of object retrieval in cluttered environments, using a rod-like pusher. Specifically, a candidate trajectory in a cluttered environment is first generated with an improved Rapidly-Exploring Random Tree (RRT) planner; Then, a Model Predictive Control (MPC) scheme is applied to stabilize the slider's poses through necessary contact with obstacles. Different from existing methods, the proposed approach is with the contact-aware feature, which enables the synthesized effect of active removal of obstacles, avoidance behavior, and switching contact face for improved dexterity. Hence both the feasibility and efficiency of the task are greatly promoted. The performance of the proposed method is validated in a planar object retrieval task, where the target object, surrounded by many fixed or movable obstacles, is manipulated and isolated. Both simulation and experimental results are presented.

The advantages of cable-driven exoskeleton robots with series elastic actuators can be summarized in twofold: 1) the inertia of the robot joint is relatively low, which is more friendly for human-robot interaction; 2) the elastic element is tolerant to impacts and hence provides structural safety. As trade-offs, the overall dynamic model of such a system is of high order and subject to both unmodelled disturbances (due to the cable-driven mechanism) and external torques (due to the human-robot interaction), opening up challenges for the controller development. This paper proposes a new trajectory-tracking control scheme for cable-driven upper-limb exoskeleton robots with series elastic actuators. The control objectives are achieved in two stages: Stage I is to approximate then compensate for unmodelled disturbances with iterative learning techniques; Stage II is to employ a suboptimal model predictive controller to drive the robot to track the desired trajectory. While controlling such a robot is not trivial, the proposed control scheme exhibits the advantages of force-sensorlessness, high accuracy, and low complexity compared with other methods in the real-world experiments.

The automatic monitoring/tracking of environmental boundaries by multi-agent systems is a fundamental problem that has many practical applications. In this paper, we address this problem with formation control techniques based on parame tric curves that represent the boundary’s feedback shape. For that, we approximate the curve with truncated Fourier series, whose finite coefficients are utilized to characterize the curve’s shape and to automatically distribute the agents along it. These feedback Fourier coefficients are exploited to design a new type of formation controller that drives the agents to form desired curves. A detailed stability analysis is provided for the proposed control methodology, considering both fixed and switching multi-agent topologies. The reported numerical simulation and experimental studies demonstrate the performance and feasibility of our new method to track closed boundaries of different shapes.

Deep reinforcement learning is confronted with problems of sampling inefficiency and poor task migration capability. Meta-reinforcement learning (meta-RL) enables meta-learners to utilize the task-solving skills trained on similar tasks and quickly adapt to new tasks. However, meta-RL methods lack enough queries toward the relationship between task-agnostic exploitation of data and task-related knowledge introduced by latent context, limiting their effectiveness and generalization ability. In this article, we develop an algorithm for off-policy meta-RL that can provide the meta-learners with self-oriented cognition toward how they adapt to the family of tasks. In our approach, we perform dynamic task-adaptiveness distillation to describe how the meta-learners adjust the exploration strategy in the meta-training process. Our approach also enables the meta-learners to balance the influence of task-agnostic self-oriented adaption and task-related information through latent context reorganization. In our experiments, our method achieves 10%-20% higher asymptotic reward than probabilistic embeddings for actor-critic RL (PEARL).

Grasping has long been considered an important and practical task in robotic manipulation. Yet achieving robust and efficient grasps of diverse objects is challenging, since it involves gripper design, perception, control, and learning, etc. Recent learning-based approaches have shown excellent performance in grasping a variety of novel objects. However, these methods either are typically limited to one single grasping mode or else more end effectors are needed to grasp various objects. In addition, gripper design and learning methods are commonly developed separately, which may not adequately explore the ability of a multimodal gripper. In this article, we present a deep reinforcement learning (DRL) framework to achieve multistage hybrid robotic grasping with a new soft multimodal gripper. A soft gripper with three grasping modes (i.e.,enveloping,sucking, andenveloping_then_sucking) can both deal with objects of different shapes and grasp more than one object simultaneously. We propose a novel hybrid grasping method integrated with the multimodal gripper to optimize the number of grasping actions. We evaluate the DRL framework under different scenarios (i.e., with different ratios of objects of two grasp types). The proposed algorithm is shown to reduce the number of grasping actions (i.e., enlarge the grasping efficiency, with maximum values of 161.0% in simulations, and 153.5% in real-world experiments) compared to single grasping modes.

The robotic manipulation of deformable linear objects (DLOs) has broad application prospects in many fields. However, a key issue is to obtain the exact deformation models (i.e., how robot motion affects DLO deformation), which are hard to theoretically calculate and vary among different DLOs. Thus, the shape control of DLOs is challenging, especially for large deformation control that requires global and more accurate models. In this article, we propose a coupled offline and online data-driven method for efficiently learning a global deformation model, allowing for both accurate modeling through offline learning and further updating for new DLOs via online adaptation. Specifically, the model approximated by a neural network is first trained offline on random data, then seamlessly migrated to the online phase, and further updated online during actual manipulation. Several strategies are introduced to improve the model's efficiency and generalization ability. We propose a convex-optimization-based controller and analyze the system's stability using the Lyapunov method. Detailed simulations and real-world experiments demonstrate that our method can efficiently and precisely estimate the deformation model and achieve the large deformation control of untrained DLOs in 2-D and 3-D dual-arm manipulation tasks better than the existing methods. It accomplishes all 24 tasks with different desired shapes on different DLOs in the real world, using only simulation data for the offline learning.

Human-robot collaboration aims to extend human ability through cooperation with robots. This technology is currently helping people with physical disabilities, has transformed the manufacturing process of companies, improved surgical performance, and will likely revolutionize the daily lives of everyone in the future. Being able to enhance the performance of both sides, such that human-robot collaboration outperforms a single robot/human, remains an open issue. For safer and more effective collaboration, a new control scheme has been proposed for redundant robots in this paper, consisting of an adaptive vision-based control term in task space and an interactive control term in null space. Such a formulation allows the robot to autonomously carry out tasks in an unknown environment without prior calibration while also interacting with humans to deal with unforeseen changes (e.g., potential collision, temporary needs) under the redundant configuration. The decoupling between task space and null space helps to explore the collaboration safely and effectively without affecting the main task of the robot end-effector. The stability of the closed-loop system has been rigorously proved with Lyapunov methods, and both the convergence of the position error in task space and that of the damping model in null space are guaranteed. The experimental results of a robot manipulator guided with the technology of augmented reality (AR) are presented to illustrate the performance of the control scheme.

The shape control of deformable linear objects (DLOs) is challenging, since it is difficult to obtain the deformation models. Previous studies often approximate the models in purely offline or online ways. In this paper, we propose a scheme for the shape control of DLOs, where the unknown model is estimated with both offline and online learning. The model is formulated in a local linear format, and approximated by a neural network (NN). First, the NN is trained offline to provide a good initial estimation of the model, which can directly migrate to the online phase. Then, an adaptive controller is proposed to achieve the shape control tasks, in which the NN is further updated online to compensate for any errors in the offline model caused by insufficient training or changes of DLO properties. The simulation and real-world experiments show that the proposed method can precisely and efficiently accomplish the DLO shape control tasks, and adapt well to new and untrained DLOs.

Laser-driven micro-tools are formulated by treating highly-focused laser beams as actuators, to control the tool's motion to contact then manipulate a micro object, which allows it to manipulate opaque micro objects, or large cells without causing photodamage. However, most existing laser-driven tools are limited to relatively simple tasks, such as moving and caging, and cannot carry out in-hand dexterous tasks. This is mainly because in-hand manipulation involves continuously coordinating multiple laser beams, micro-tools, and the object itself, which has high degrees of freedom (DoF) and poses up challenge for planner and controller design. This paper presents a new hierarchical formulation for the grasping and manipulation of micro objects using multiple laser-driven micro-tools. In hardware, multiple laser-driven tools are assembled to act as a robotic hand to carry out in-hand tasks (e.g., rotating); in software, a hierarchical scheme is developed to shrunken the action space and coordinate the motion of multiple tools, subject to both the parametric uncertainty in the tool and the unknown dynamic model of the object. Such a formulation provides potential for achieving robotic in-hand manipulation at a micro scale. The performance of the proposed system is validated in simulation studies under different scenarios.

Sanding is a common yet important task in the manufacturing of many wooden objects (e.g. furniture, decoration box), where the coated layer attached on objects is removed after the interaction with sanding belts. Existing sanding operation is heavily dependent on manual works, which is highly labor-intensive and with the low consistency of quality, and the issues of safety and health also arise after continuous working in the noisy and dusty environment. To deal with the aforementioned, this paper presents the development of a new autonomous sanding robot. The autonomous capability of the developed robot is reflected in the whole procedure of sanding. In particular, the CAD model of the target object is automatically constructed with the structured-light technology, and the sanding behavior on the target surface is self-regulated under the desired impedance model. Such feature makes the robot capable of working towards uncertain objects with minimum human involvement. The proposed impedance controller has the model-free advantage, by using the adaptive neural networks (NNs) to compensate the uncertain dynamics and the unknown disturbances online. The stability of the closed-loop system is rigorously proved with Lyapunov methods, and experimental results on different objects are presented to validate the performance of the developed robot. The implementation of the developed robot can systematically address the problems associated with manual works. Note to Practitioners—The current working environment of sanding is not healthy or safe to humans, due to the nature of noise, dust, high-speed sanding belt. By controlling the robot to autonomously perform sanding tasks can keep humans away from such environment and hence systematically address the issues of health and safety. This paper presents a new impedance control method for sanding robot. In impedance control, the control goal is specified as a dynamic relationship between the contact position and the interaction force. In this paper, the contact position is determined by using the 3D vision sensor, while the interaction force is regulated by referring to the human experience. In addition, the influence caused by unmodeled factors (e.g. unknown dynamics of the sanded object) is also dealt with by using the techniques of NNs. Such setting can effectively guarantee the sanding quality and also avoid the physical damage to the sanded object. Therefore, it lays the foundation for the autonomous robotic sanding.

Embryos/oocytes vitrification is an essential cryopreservation technique in IVF (in vitro fertilization) clinics. The reliable and effective transferring of embryos/oocytes is crucial to the subsequent steps in the whole procedure of vitrification. After each transferring, the straw needs to be replaced with a new one. Due to the uncertainties in the fabrication and installation, the exact knowledge of the kinematic model of the straw is usually unknown, and the relationship between the microscope and the straw is also unknown without calibration beforehand. In such situation, automatically transferring the oocytes from micropipette to the narrow tip of straw (0.7mm) is very challenging. In this paper, a new vision-guided robotic system is developed to automate the transferring of the oocyte without calibration. To this end, the unknown depth information is estimated then compensated by constructing a deep vision network through microscope image, and an approximate Jacobian control algorithm is also proposed to servo control the end tip of the uncalibrated straw to contact the micropipette with the vision feedback. After that, the oocyte is automatically transferred from the micropipette to the straw to finalize the task. The stability of the closed-loop control system is rigorously proved with Lyapunov methods, and the effectiveness of the developed robot is validated in experiments.

Large demand for robotics and automation has been reflected in the sanding works, as current manual operations are labor-intensive, without consistent quality, and also subject to safety and health issues. While several machines have been developed to automate one or two steps in the sanding works, the autonomous capability of existing solutions is relatively low, and the human assistance or supervision is still heavily required in the calibration of target objects or the planning of robot motion and tasks. This paper presents the development of an autonomous sanding robot, which is able to perform the sanding works on an unknown object automatically, without any prior calibration or human intervention. The developed robot works as follows. First, the target object is scanned then modeled with the structured-light camera. Second, the robot motion is planned to cover all the surfaces of the object with an optimized transition sequence. Third, the robot is controlled to perform the sanding on the object under the desired impedance model. A prototype of the sanding robot is fabricated and its performance is validated in the task of sanding a batch of wooden boxes. With sufficient degrees of freedom (DOFs) and the customization of the end effector, the developed robot is able to provide a general solution to the autonomous sanding on many other different objects.

While the mechanism of Rope-Climbing provides much flexibility, it opens up challenges to the development of the controller for Robotic Manipulator installed on Rope-Climbing robot(RCR), which is called Rope-Climbing Robot Manipulator(RCRM) here. In particular, the deformable nature of the rope results in the vibration to the manipulator and hence affects the positioning of the end effector. In this paper, a new adaptive vision-based controller is proposed for RCRM, which enables the robot to carry out the high-accuracy task under the unknown vibration from the rope. The proposed controller guarantees the performance of the robot in twofold. First, the control problem is directly formulated in the image space such that the exact spatial relationship between the moving base of the manipulator (due to the vibrating rope) and the target (e.g. the wall) is not required. Second, novel adaptation laws are developed to estimate the vibration from the rope online and are cancelled out in the robot control input to stabilize the end effector. The stability of the closed-loop system is rigorously proved with Lyapunov methods, and experimental results are presented to illustrate the performance of the proposed controller.

Wall polishing is a typical and essential procedure in the interior renovation. However, such works are mainly carried out by humans, which have the disadvantages of low efficiency, inconsistent quality, and issues of safety and health. A new vision-based impedance controller is proposed for polishing robots to automate the labor-intensive works. The desired impedance model is specified as the control objective to regulate the dynamic relationship between the interaction force and the motion of the robot end effector, where the motion is measured with the vision feedback. The use of the vision feedback guarantees the performance of the robot from two aspect. First, the vision feedback from the high-resolution camera ensures the accuracy of measurement of the robot end effector and hence guarantees the quality of polishing. Second, the concept of image moment is introduced such that the image Jacobian matrix is non-singular in a global sense, which guarantees the large working range of the robot. The dynamic stability of the closed-loop system is rigorously proved with Lyapunov methods, and experimental results are presented to illustrate the performance of the proposed controller.

The fast expanding 3C (Computer, Communication, and Consumer electronics) manufacturing leads to a high demand on the fabrication of USB cables. While several commercial machines have been developed to automate the process of stripping and soldering of USB cables, the operation of manipulating USB wires according to the color code is heavily dependent on manual works because of the deformation property of wires, probably resulting in the falling-off or the escape of wires during manipulation. In this paper, a new vision-based controller is proposed for robotic grasping and manipulation of USB wires. A novel two-level structure is developed and embedded into the controller, where Level-I is referred to as the grasping and manipulation of wires, and Level-II is referred to as the wire alignment by following the USB color code. The proposed formulation allows the robot to automatically grasp, manipulate, and align the wires in a sequential, simultaneous, and smooth manner, and hence to deal with the deformation of wires. The dynamic stability of the closed-loop system is rigorously proved with Lyapunov methods, and experiments are performed to validate the proposed controller.

The ability of rope-climbing robots in aloft operation is limited by its self-supporting and locomotion ability. In many applications, a given task is also too complex to be achieved by a single rope-climbing robot acting alone. The solution of multiple rope-climbing robots can overcome the limitations. However, existing control methods for rope-climbing robots are limited to single robot, and the open issue of coordination between multiple rope-climbing robots has not been systematically addressed. This paper presents a new synchronization scheme for position control of multiple rope-climbing robots, such that each robot moves to the corresponding desired position while synchronizing the heights between each other. Maintaining the same height is very important to guarantee the stability of the task-oriented manipulator installed among multiple robots, when it is performing the manipulation task. The development of the proposed controller is based on the singular perturbation approach, by treating the fast actuator dynamics as a perturbation of the slow robot dynamics, such that the lowest control complexity is achieved. The exponential stability of the overall system that consists of the fast and slow subsystems is proved by using Tikhonov’ s theorem. Experimental results are presented to illustrate the performance of the proposed controller.

Emerging applications of robotic manipulation of deformable objects have opened up new challenges in robot control. While several control techniques have been developed to manipulate deformable objects, the performance of existing methods is commonly limited by two issues: 1) implicit assumption that the physical contact between the end-effector and the object is always maintained, and 2) requirements of exact parameters of deformation model, which are difficult to obtain. This paper presents a new control scheme for robotic manipulation of deformable objects, which allows the robot to automatically contact then actively deform the deformable object by assessing the status of deformation in real time. Instead of designing multiple controllers and switching among them, the proposed method smoothly and stably integrates two control phases (i.e. region reaching and active deforming) into a single controller. The stability of the closed-loop system is rigorously proved with the consideration of the uncertain deformation model and uncalibrated cameras. Hence, the proposed control scheme enhances the autonomous capability of active deformable object manipulation. Experimental studies are conducted with different initial conditions to demonstrate the performance of the proposed controller.

Emerging applications of robots with direct physical interactions with humans have led to the development of a variety of series elastic actuators (SEAs) which are compliant, force controllable, and back drivable. The performance of current SEAs is mainly dependent on the specific stiffness of the spring. In our previous work, a compliant actuator with two-stage stiffness has been designed to overcome the performance limitations in current SEAs. The key novelty is that a low-stiffness spring and a high-stiffness spring are employed instead of a single spring in current SEAs, which has the advantages of high fidelity, low output impedance, and also large force range and bandwidth. In this paper, a tracking control scheme is proposed for the compliant actuator with two-stage stiffness. Although the overall stiffness is discontinuous, the proposed controller is continuous by integrating different control modes for two springs into a single one. The transition between control modes is smooth and embedded inside the controller, and it is also automatically realized by monitoring the output force of the actuator. The stability and convergence of the closed-loop system are analyzed, and experimental results are presented to demonstrate the effectiveness of the proposed control scheme.Note to Practitioners—An SEA is developed by placing an elastic element into the actuator; this elasticity gives SEAs several unique properties including low mechanical output impedance, tolerance to impact loads, and passive mechanical energy storage, which makes it suitable for human–robot interaction. The performance of existing SEAs is highly dependent on the stiffness of a single spring. To overcome the limitations, a novel SEA with two-stage stiffness was proposed in our previous work. This paper suggests a continuous tracking control method for the proposed compliant actuator. Although the overall stiffness is discontinuous, the transition between different control modes for two springs is smooth and automatically realized. Experimental results show that the output force of the actuator is bounded. In future research, uncertainties in actuator dynamics will be considered, such that system identification or calibration is not required.

The expanding 3C (Computer, Communication, and Consumer electronics) manufacturing industry leads to a high demand on the soldering of flexible PCBs. Current manual soldering has the disadvantages of low output, low speed, and low efficiency, and upgrading soldering operations with robotic technologies is mainly limited by the property of deformation of flexible PCBs. In this paper, a novel robotic manipulation system is developed for automatic soldering of flexible PCBs, consisting of the hardware of a dual-arm configuration and the software of a cooperative control scheme. The proposed system works in a sequential manner, in the sense that a Cartesian-space region reaching controller drives an assistive arm to actively contact the PCB first, and a vision-based tracking controller activates a soldering arm after the deformation is stabilized. The proposed formulation eliminates uncertain deformation of flexible PCBs and thus guarantees the feasibility of robotic soldering.

Series elastic actuators (SEAs) are known to offer a range of advantages over stiff actuators for human–robot interaction, such as high force/torque fidelity, low impedance, and tolerance to shocks. While a variety of SEAs have been developed and implemented in initiatives that involve physical interactions with humans, relatively few control schemes were proposed to deal with the dynamic stability and uncertainties of robotic systems driven by SEAs, and the open issue of safety that resolves the conflicts of motion between the human and the robot has not been systematically addressed. In this paper, a novel continuous adaptive control method is proposed for SEA-driven robots used in human–robot interaction. The proposed method provides a unified formulation for both therobot-in-chargemode, where the robot plays a dominant role to follow a desired trajectory, and thehuman-in-chargemode, in which the human plays a dominant role to guide the movement of robot. Instead of designing multiple controllers and switching between them, both typical modes are integrated into a single controller, and the transition between two modes is smooth and stable. Therefore, the proposed controller is able to detect the human motion intention and guarantee the safe human–robot interaction. The dynamic stability of the closed-loop system is theoretically proven by using the Lyapunov method, with the consideration of uncertainties in both the robot dynamics and the actuator dynamics. Both simulation and experimental results are presented to illustrate the performance of the proposed controller.

Series elastic actuators (SEAs) are known to offer a number of advantages such as high force/torque fidelity, low impedance, and tolerance to shocks, which make it suitable for the applications involving human-robot interaction. In existing SEA-driven robot systems, the control objective is usually specified as a predefined trajectory or an impedance model that describes the relationship between the desired motion of robot and the external force, and controllers are always activated to regulate the desired motion or the desired impedance model. In this paper, a region control scheme is proposed for robots driven by SEAs, where the control objective is specified as a region, instead of trajectory or desired impedance. The region control has the advantage of flexibility, in the sense that the robot is able to move freely inside the desired region and thus compliant with the environment or physical interactions with humans. Though the overall dynamics that includes both actuator and robot dynamics is a fourth-order system, the proposed control method does not require the high-order derivatives or the construction of any observer. Experimental results are presented to demonstrate the effectiveness of the proposed control method.

Several control schemes have been proposed for optical tweezers, but most existing methods assume that the optical trapping is maintained throughout the manipulation, and Brownian motion is ignored for the simplification of stability analysis. However, the optical trapping is not effective when a biological cell is initially outside the optical trap, and even if the cell is initially trapped, it may escape from the optical trap due to random Brownian perturbations and kinetic energy gained during manipulation. This paper presents a new robotic control technique to address the trapping and manipulation problem of biological cell in presence of random Brownian perturbations. By using the proposed method, the cell can be automatically trapped when it is not inside the optical trap and the optical manipulation of cell is enhanced by saturating the position feedback. The stability of the closed-loop system is analysed from stochastic perspectives. Experimental results are presented to illustrate the performance of the proposed control method.

Multi-cellular aggregation is a fundamental phenomenon observed in many biological processes. Investigating cells aggregation helps us to have better understanding of many biological processes. Moreover, it is useful in finding cure for the diseases caused by cells aggregation. In this paper, we present a control methodology to obtain multi-cellular aggregation by using multiple optical trapping. The proposed method is also useful for study of multiple cell fusion which is an important cellular process. Experimental results are presented to show the effectiveness of the proposed method in achieving multi-cellular aggregation.

In this paper, a tracking control scheme is proposed for optical manipulation of biological cell with unknown trapping stiffness. The requirement on the model of the trapping stiffness is eliminated in the proposed formulation and thus system identification and calibration are not needed. The unknown trapping stiffness and the uncertain dynamic parameters are estimated separately, with on-line update laws. By using the proposed control scheme, the laser beam is able to manipulate the trapped cell to track various time-varying trajectories, to suit different applications in cell manipulation. The proposed control scheme is based on the dynamic formulation where the position of laser beam is controlled by closed-loop robotic manipulation techniques. The stability of the overall system is analyzed by using Lyapunov-like method, with consideration of the dynamics of both the cell and the manipulator of laser source. Experimental results are presented to illustrate the performance of the proposed tracking controller with unknown trapping stiffness.

Emerging applications of robot systems that involve close physical interaction with human have opened up new challenges in robot control. For these applications, it is important to consider the stability and coordination of human-robot interaction. While various control techniques have been developed for human-robot interaction, existing methods do not take the advantages of human ability in responding and adapting to unknown environment. In this paper, a human-guided manipulation problem which is able to take advantages of both the human knowledge and the robot's ability, is formulated and solved. The workspace is divided into a human region, where human play a more active role in the manipulation task, and a robot region, where the robot is more dominant in the manipulation. The proposed formulation allows the involvement of human control action to deal with unforeseen changes or uncertainty in the real world. We present a theoretical foundation that allows the stability and coordination of the human-guided manipulation problem to be analyzed. Based on the human region and the robot region, an adaptive tracking controller is developed. Experimental results are presented to illustrate the performance of the proposed control method.

Microscopic optics and cameras are commonly used in micromanipulation or biomanipulation workstations since they provide a large spectrum of visual details and information. The visual feedback information also improves robustness to uncertainty and accuracy of micromanipulation. Among various micromanipulation systems, optical tweezers are one of the most useful instruments that utilize a focused beam of light to manipulate biological cell or nanoparticles without physical contact. However, current optical manipulation techniques fail if the laser beam is not within the field of view (FOV) of the microscope. To solve this problem, we present a robotic control technique for optical manipulation with limited FOV of microscope. The proposed control strategy consists of a vision based control that manipulates the trapped cell to move to a desired position inside the FOV and a Cartesian-space feedback control that drives the laser beam back when it is outside the FOV. Thus, the proposed method allows the laser beam to leave the FOV during the course of manipulation and the transition from one feedback to another is smooth. The stability of the closed-loop system is analysed by using Lyapunov-like methods, with consideration of the dynamic interaction between the cell and the manipulator of the laser source. Experimental results are presented to illustrate the performance of the proposed method.

While several automatic manipulation techniques have recently been developed for optical tweezer systems, the measurement of the velocity of cell is required and the interaction between the cell and the manipulator of laser source is usually ignored in these formulations. Although the position of cell can be measured by using a camera, the velocity of cell is not measurable and usually estimated by differentiating the position of cell, which amplifies noises and may induce chattering of the system. In addition, it is also assumed in existing methods that the image Jacobian matrix from the Cartesian space to image space of the camera is exactly known. In the presence of estimation errors or variations of depth information between the camera and the cell, it is not certain whether the stability of the system could still be ensured. In this paper, vision-based observer techniques are proposed for optical manipulation to estimate the velocity of cell. Using the proposed observer techniques, tracking control strategies are developed to manipulate biological cells with different Reynolds numbers, which do not require camera calibration and measurement of the velocity of cell. The control methods are based on the dynamic formulation where the laser source is controlled by the closed-loop robotic manipulation technique. The stability is analyzed using Lyapunov-like analysis. Simulation and experimental results are presented to illustrate the performance of the proposed cell manipulation methods.

In this paper, an observer based adaptive control method is proposed for optical manipulation of cell. The dynamics of the robotic manipulator of the laser source is introduced in the optical tweezers system, so that a closed-loop control method is formulated and solved, and a backstepping approach is used to derive a control input for the manipulator. The interaction between the cell dynamics and the manipulator dynamics leads to a fourth-order overall dynamics, and hence a nonlinear observer is constructed to avoid the use of high-order derivatives of the positions in the control input. Stability of the closed-loop system is analyzed by using Lyapunov-like analysis. Simulation results are presented to illustrate the performance of the proposed control methods.

Task-space sensory feedback information such as visual feedback is used in many modern robot control systems as it improves robustness to model uncertainty. However, existing sensory feedback control schemes are only valid locally in a finite task space within a limited sensing zone where singularity of the Jacobian matrix is avoided. In this paper, the global stability problem of task-space sensory feedback control system is formulated and solved. The proposed method is based on multiple regional feedback information where each feedback information is employed in a local region. The combination of the local feedback covers the entire workspace and thus guarantees the global movement of the robot. In addition, the switching from one feedback information to another is embedded in the controller without using any hard or discontinuous switching. Experimental results are presented to illustrate the performance of the proposed controller.

Current manipulation techniques of optical tweezers treat the position of the laser beam as the control input and an open-loop kinematic controller is designed to move the laser source. In this paper, a closed-loop robotic control method for optical tweezers is formulated and solved. While robotic manipulation has been a key technology driver in factory automation, robotic manipulation of cells or nanoparticles is less well understood. The proposed formulation shall bridge the gap between traditional robot manipulation techniques and optical manipulation techniques of cells. A dynamic region controller is proposed for cell manipulation using optical tweezers. The desired objective can be specified as a dynamic region rather than a position or trajectory, and the desired region can thus be scaled up and down to allow flexibility in the task specifications. Experimental results are presented to illustrate the performance of the proposed controller.

Singularity issue has been a long standing problem in task-space control of robot. It is commonly assumed in the theoretical analysis of task-space control system that the robot is operating in a finite task space such that singularity problem can be avoided. This limits the potential workspace of the robot when task-space control is employed. In this paper, a singularity-robust task-space controller is proposed for tracking control of robot manipulator. The proposed controller consists of a joint-space position controller that is activated when the robot is near singular configurations, and a task-space tracking controller that is used when the robot leaves the singular region. Therefore, the robot can start from the singular regions and transit smoothly from joint space to task space. It can also enter the singular region during the course of movement. The stability of the closed-loop system is analyzed with consideration of the singularity issue. Experimental results are presented to illustrate the performance of the proposed controller.

This paper presents an adaptive region tracking control for Autonomous Underwater Vehicle (AUV). The AUV is required to track a moving region to accomplish a given task. The desired target is specified as a region rather than a point so that the control effort used to track the region is minimal. In the applications where the accuracy is of utmost importance, the desired region can be chosen to be small so that the precision is not lost. The desired region can be scaled up or scaled down so that the AUV can adjust its position to suit the applications. A Lyapunov-like function is presented for the stability analysis. Simulation results on AUV with 6 degrees of freedom are presented to demonstrate the effectiveness of the proposed controller.

It is interesting to observe from human visually guided tasks that visual feedback is not used for the entire movement, but only at end phases when our hand is near the target. We are able to move our hand from an initial position that is not within our field of view and transit smoothly and easily into visual feedback when the target is near. Inspired by this natural action, this paper presents a new task-space adaptive controller with dual feedback information. The proposed controller consists of a Cartesian-space region reaching controller at the initial stage and a vision based tracking controller that is only activated when the end effector enters an image region. A new potential energy function is proposed such that the image region can be fixed as the field of view of the camera and does not have to vary with the desired trajectory. The proposed task-space controller can transit smoothly from Cartesian-space reaching to vision-space tracking control. The stability of the closed-loop system is analyzed with consideration of the nonlinear dynamics. The proposed adaptive controller is implemented on an industrial robot and experimental results are presented to illustrate the performance of the proposed controller.