The geometric parameters severely impact the performance of dexterous manipulation, but manually adjusting them is time-consuming. In this paper, we employ the dexterity-guided dimensional synthesis to explore the geometric design space of a robotic hand, and propose a multi-objective optimization framework to improve its manipulation dexterity. Specifically, we first establish a screw-based mathematical model to describe the hand-object system’s kinetostatic properties. Three objectives are specified to achieve Pareto optimality: maximizing the system’s feasible position space, feasible orientation space, and manipulation stability. Furthermore, to validate the optimal design parameters in practice, we develop a multi-task object motion controller and apply it to a letter handwriting task. Finally, the optimized results are quantitatively analyzed by simulation, yielding a Pareto front and 112 optimal solutions. The applicability of the optimal parameters is assessed through a letter “O” handwriting experiment. Results show that the optimized hand can write the “O” with a maximum radius of 30.0 mm, which is 27.12% larger than that of the non-optimized hand. This controller is also used to schedule subtasks with varying priorities to avoid constrained regions. In contrast to the neural network-based PID controller, the designed controller prioritizes the task’s critical parts, ensuring the legibility of the letters.

Dexterous hands are the key element for robots to achieve human-like manipulation capabilities. An outstanding challenge is to provide fingertips of dexterous hands with precise tactile deformation sensing capabilities. In this paper, we present the GelStereo Tip, a spherical and easy-to-integrate GelStereo-type visuotactile sensor capable of sensing high-resolution 3D elastomer deformation. Previous calibration method does not take into account the impact of imaging errors caused by the sensor’s compact and high-curvature structural characteristics on the accuracy of tactile sensing. Therefore, we propose a novel self-calibration method based on the Refractive Stereo Ray Tracing model, named GTSC, and demonstrate the accuracy of less than 0.3 mm for deformation sensing. Furthermore, we also propose a Contact Retention Tactile Controller to address the issue of fingertips being unable to overcome obstructive torque during the multi-finger bottle cap screwing. After integrating GelStereo Tip into fingertips of Allegro Hand, the controller adjusts the joint positions of the given trajectory using proportional control based on the difference between the sensor’s actual deformation and the reference state for contact retention. We believe that the GelStereo Tip sensor combined with robotic dexterous hands has great application potential in the field of multi-finger fingertip manipulation. Note to Practitioners—The motivation of this paper is to design a fingertip visuotactile sensor with high-precision 3D tactile deformation sensing capabilities for multi-finger robotic hands and to validate its sensing performance. Additionally, it aims to address the issue of overcoming resistance in multi-finger screwing manipulations. Currently, most sensors do not consider the refraction effect or ignore the impact of planar imaging errors in refractive calibration. This paper proposes a visuotactile sensor along with a corresponding self-calibration method to ensure its sensing accuracy. Experiments show that our sensor possesses high-precision and robust 3D deformation sensing capabilities. On the other hand, multi-finger hands often struggle to complete screwing tasks along the given trajectory due to disturbances from torque resistance. This paper proposes a tactile controller that evaluates the contact state through aforementioned tactile sensing to improve subsequent trajectory and achieve continuous screwing. Comparative experiments highlight the necessity of this controller and the reliability of tactile sensing. We hope that the design of our sensor, the self-calibration method, and the tactile controller applied to multi-finger screwing can provide new insights for other practitioners.

Visuotactile sensors have been shown to provide rich contact information for robots. However, how to build a high-fidelity visuotactile simulator that supports multi-mode tactile imprints and various sensor configurations (such as coating patterns) remains a challenging problem. In this paper, we present TacFlex, an efficient and flexible simulator for visuotactile sensors, which physically simulates the elastomer deformation using Finite Element Methods (FEM), and focuses on linking the deformed elastomer mesh to diverse tactile imprints, including tactile images with arbitrary coating patterns and tactile 3D point clouds. We further propose a ray tracing-based rectification method to deal with multi-medium refraction effects to make the simulated tactile images more realistic. Extensive qualitative and quantitative experiments are conducted to demonstrate the effectiveness of TacFlex on several visuotactile sensors. Furthermore, we explore the Sim2Real performance of different tactile imprints provided by TacFlex in tactile perception and manipulation tasks, such as cylindrical object pose estimation and peg-in-hole. The perception/policy models trained in simulation are successfully deployed in the real world. Finally, we present the outlook on the potential of TacFlex in visuotactile manipulation learning. The TacFlex simulator is open-sourced to the community. See supplementary video, code, and results athttps://sites.google.com/view/tacflex/.

This paper presents a novel vision-based artificial lateral line (ALL) sensor, FlowSight, enhancing the perception capabilities of underwater robots. Through an autonomous vision system, FlowSight allows for simultaneous sensing the speed and direction of local water flow without relying on external auxiliary equipment. Inspired by the lateral line neuromast of fish, a flexible bionic tentacle is designed to sense water flow. Deformation and motion characteristics of the tentacle are modeled and analyzed using bidirectional fluid-structure interaction (FSI) simulation. Upon contact with water flow, the tentacle converts water flow information into elastic deformation information, which is captured and processed into an image sequence by the autonomous vision system. Subsequently, a water flow perception method based on deep neural networks is proposed to estimate the flow speed and direction from the captured image sequence. The perception network is trained and tested using data collected from practical experiments conducted in a controllable swim tunnel. Finally, the FlowSight sensor is integrated into the bionic underwater robot RoboDact, and a closed-loop motion control experiment based on water flow perception is conducted. Experiments conducted in the swim tunnel and water pool demonstrate the feasibility and effectiveness of FlowSight sensor and the water flow perception method.

This article addresses an autonomous manipulation problem for an underwater vehicle-manipulator system (UVMS) operating in a free-floating way while subjecting to unknown continuous disturbance. More specifically, a composite control scheme composed of disturbance observer (DOB), predictor model network (PM-Net), and nonlinear model predictive control (NMPC), is devised to improve the control performance of UVMS (i.e., unicycle-like UVMS actuated only in the surge, heave, and yaw for vehicle body) in the case of disturbance, model mismatch, and input saturation. A RBF-DOB is formulated by combining a DOB and a Radial Basis Function (RBF) neural network to estimate disturbance at the current step. Then, the PM-Network, composed of a disturbance predictor network and state predictor network, is developed based on long short-term memory (LSTM) network that predicts UVMS state sequences considering model mismatch and disturbance. The NMPC is deployed as a feedback control law to endow the input saturation of the UVMS and produce optimal control action. Compared with conventional DOB control methods using feed-forward compensation of disturbance, the primary merit of the proposed approach is that the disturbance estimated by RBF-DOB is utilized in the PM-Net to predict future UVMS state sequences, which are exploited on the NMPC’s receding optimization. Finally, realistic simulation and relevant experiment are conducted to demonstrate the effectiveness of the proposed method. Note to Practitioners—The motivation behind this article is the autonomous manipulation of an underwater vehicle-manipulator system subjected to unknown disturbance. However, it is not always feasible or straightforward to obtain the external disturbance and unmodeled dynamics for designing robust controllers. On the one hand, how to manipulate the disturbance into the designed controller to generate optimal control action rather than by using feed-forward compensation. On the other hand, the control input saturation often occurs in the UVMS control, especially under the disturbance rejection conditions, where it should be considered in the controller design. Currently, the predominant methods for UVMS control lack a control scheme that provides a complete and credible control strategy that takes the aforementioned issues into consideration. Motivated by the above analysis, this study provides a composite control scheme to deal with the dynamic uncertainties, unknown disturbance, and input saturation. The results of realistic simulation and relevant experiments demonstrate the effectiveness of the proposed method. Hopefully, our control method can provide valuable theoretical and technical guidance to practicing marine engineers for controller design.

This paper addresses a learning-based path following control scheme for a biomimetic underwater vehicle (BUV) driven by undulatory fins. A dynamic line-of-sight (DLOS) guidance system is designed, which uses a virtual ball with a dynamic radius to detect the reference path. This DLOS system guides our BUV in the path following control and extracts essential information for the Markov decision process (MDP) of the control task. A deep reinforcement learning (DRL) algorithm, sample-observed soft actor-critic (SOSAC) is proposed. The can train out control policy with greater cumulative reward and higher success rate by using two tricks: sample observation and sample diversification. Based on the DLOS system, the MDP of the control task, and a multilayer perceptron (MLP) trained by the SOSAC, our control scheme is established. Experiments show that our BUV can successfully achieve path following control in an indoor pool environment by using this control scheme.Note to Practitioners—The motivation of this paper is to design a practical end-to-end path following control scheme for the BUV driven by undulatory fins, and verify this scheme in a real-world environment. Unlike common autonomous underwater vehicles (AUVs) using axial propellers, the BUVs apply biomimetic propellers such as the undulatory fin. Multimodel wave patterns can be implemented by the undulatory fin, which generates nonlinear thrust and lateral force simultaneously. This propulsive feature makes the driving force on different directions of the BUV to be strong coupled, and it is complicated to convert the outputs of a common controller into waveform parameters of the undulatory fins to control the BUV. Therefore, in this paper, we proposed an end-to-end learning-based path following controller, which observes environmental information and directly generates waveform parameters to control our BUV. Experiments suggest that our control scheme is practical and valid.

Endowing the robot with tactile perception can effectively improve manipulation dexterity, along with various benefits of human-like touch. Using GelStereo (GS) tactile sensing, which gives high-resolution contact geometry information, including 2-D displacement field, and 3-D point cloud of the contact surface, we present a learning-based slip detection system in this study. The results reveal that the well-trained network achieves 95.79% accuracy on the never-seen testing dataset, which surpasses the current model-based and learning-based methods using visuotactile sensing. We also propose a general framework for slip feedback adaptive control for dexterous robot manipulation tasks. The experimental results show the effectiveness and efficiency of the proposed control framework using GS tactile feedback when deployed on real-world grasping and screwing manipulation tasks on various robot setups.

Recently, visuotactile sensors have shown promising potential in robotics due to their high-resolution sensing ability. Unfortunately, the majority of available visuotactile sensors are limited to flat shapes, which severely limits their application possibilities. In this article, we propose a novel curved visuotactile sensor, the GelStereo Palm, which senses the 3-D contact geometry on a curved surface using a binocular vision system. Meanwhile, to solve the light refraction problem in the binocular stereo vision system under a curved medium, a refractive stereo ray tracing model for GelStereo Palm is presented. Moreover, a 3-D tactile point cloud sensing pipeline is introduced to reconstruct the 3-D contact geometry in real-time. Finally, extensive experiments are conducted to verify the accuracy and robustness of the 3-D contact geometry sensing of our GelStereo Palm sensor.

Zeqian Ju, Peiling Lu, Xu Tan, Rui Wang, Chen Zhang, Songruoyao Wu, Kejun Zhang, Xiang-Yang Li, Tao Qin, Tie-Yan Liu. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022.

Visuotactile sensors have recently attracted much attention in robot communities due to the benefit of high spatial resolution sensing. However, force/torque estimation by visuotactile sensors remains a challenging problem. In this paper, we propose a learning-based six-axis force/torque estimation network using GelStereo visuotactile sensor, which can provide two-dimensional (2D) and three-dimensional (3D) displacements of markers embedded in the sensor surface. The convolutional neural networks are employed to extract multi-modal tactile deformation features; and a novel contact positional encoding method is proposed to eliminate the influence of translation invariance in convolutional operators. The well-trained model achieves the best RMSE of 0.290 N in force and 0.0084 Nm in torque. Furthermore, the proposed force/torque estimation network is integrated with a force-feedback policy for adaptive grasping tasks. The experimental results demonstrate the effectiveness of the proposed method and its potential application in robotic grasping and manipulation tasks.

This article presents the design and implementation of an innovative biomimetic underwater vehicle (BUV) and its locomotion controller. Through mimicking a dactylopteridae, the hybrid propulsion BUV is designed with two symmetrical bio-inspired long-fins and a double-joint fishtail. The mechatronic design of the dactylopteridae-inspired BUV with the pectoral long-fins and a double-joint fishtail is first provided. The two flexible long-fins compose the median and/or paired fin (MPF) propulsion, while the fishtail acts as the body and/or caudal fin (BCF) propulsion. Through the coordination of BCF and MPF propulsion modes, the BUV obtains excellent low-speed locomotion stability and also keeps high maneuverability. Moreover, the locomotion control methods based on central pattern generators (CPGs) model and fuzzy adaptive proportion integral differential (PID) are proposed for this BUV. In the end, the experimental results of the multimode motion and closed-loop motion control demonstrate the feasibility and effectiveness of the mechanism and the locomotion control system.Note to Practitioners—The motivation behind this article is the design of a novel biomimetic underwater vehicle (BUV) that possesses low-speed locomotion stability and fast swimming ability, which is suitable for carrying relevant sensors to complete water quality monitoring, biological observation, underwater equipment inspection, underwater structure detection, and other marine tasks. Currently, BUVs are usually designed as only one propulsion mode by caudal fin or paired fins, which makes it difficult to have the advantages of both modes. In order to further study the problem, we designed a dactylopteridae-inspired BUV with the bilateral pectoral long-fins (providing low-speed locomotion stability) and a double-joint fishtail (providing fast swimming ability). A hybrid-driven motion control framework is presented for the BUV based on a central pattern generators (CPGs) model and fuzzy adaptive proportion integral differential (PID). A series of experiments suggests that the mechanism and the locomotion control system are practical and valid. Hopefully, our mechanism and control framework can provide valuable theoretical and technical support guidance to the practicing marine engineer for the codesign of propulsion mode and control.

In this article, the underwater target tracking control problem of a biomimetic underwater vehicle (BUV) is addressed. Since it is difficult to build an effective mathematic model of a BUV due to the uncertainty of hydrodynamics, target tracking control is converted into the Markov decision process and is further achieved via deep reinforcement learning. The system state and reward function of underwater target tracking control are described. Based on the actor-critic reinforcement learning framework, the deep deterministic policy gradient actor-critic algorithm with supervision controller is proposed. The training tricks, including prioritized experience replay, actor network indirect supervision training, target network updating with different periods, and expansion of exploration space by applying random noise, are presented. Indirect supervision training is designed to address the issues of low stability and slow convergence of reinforcement learning in the continuous state and action space. Comparative simulations are performed to show the effectiveness of the training tricks. Finally, the proposed actor-critic reinforcement learning algorithm with supervision controller is applied to the physical BUV. Swimming pool experiments of underwater object tracking of the BUV are conducted in multiple scenarios to verify the effectiveness and robustness of the proposed method.

Biomimetic underwater robots have attracted considerable research attention globally, owing to their quieter actuations, higher propulsion efficiency, and stronger maneuverability when compared with conventional underwater vehicles equipped with axial propellers. This article provides a comprehensive survey of current research in this field. First, we review the development status of biomimetic underwater robots in both body/caudal fin (BCF), median/paired fin (MPF), and their hybrid propulsion modes. Then, we outline the motion control methods employed in biomimetic underwater robots, including open-loop swimming control and typical closed-loop control strategies. In particular, we detail our latest studies on the RobCutt series underwater robots. On this basis, some critical issues and future directions are summarized. We predict that biomimetic underwater robots will have excellent prospects in underwater environment exploration and resource utilization.

This article addresses a problem of seabed terrain following control (STFC) for an underwater vehicle-manipulator system (UVMS). The motivation is to perform a visual search of marine products closely to seabed in unknown environment. In terms of this issue, we propose a novel and robust STFC framework for our UVMS to maintain an appropriate height to seabed. A nonlinear model predictive control (NMPC) method is formulated to solve the STFC problem. To relieve online computational burden and system noisy influence, Ohtsuka's continuation/generalized minimal residual (C/GMRES) algorithm incorporated with a tracking differentiator (TD) is investigated. In order to improve the following accuracy, the system state prediction part of the NMPC and a long short-term memory (LSTM) network are elaborated to predict future seabed terrain using a depth gauge and an altimeter, respectively. Finally, the three different physical scenarios for STFC problem are established using ROS to demonstrate the robustness and efficiency of the proposed algorithm.

This article presents a coordinated vehicle-manipulator control method for an underwater biomimetic vehicle-manipulator system (UBVMS) to implement floating autonomous manipulation in practice. An algorithm framework composed of adaptive tracking differentiator (ATD), extended state observer (ESO), improved nonsingular terminal sliding-mode control (I-NTSMC), fuzzy-logic controller (FLC), and estimator of manipulator disturbances, is proposed. The ATD is designed to generate desired motion state and alleviate noise. The ESO is developed to estimate the motion state, systematic uncertainties, and external disturbances. The proposed I-NTSMC method assures the finite-time convergence of the system states and alleviate chattering. The estimation of the manipulator disturbances is incorporated into the control strategy to enhance the station keeping of the vehicle. Finally, underwater autonomous free floating manipulation experiments about opening a door and grasping objects are conducted to validate the theoretical results and confirm the feasibility of the proposed control strategy.

This paper aims to investigate optimal formation solutions of multiple biomimetic underwater vehicles (BUVs). The BUV is propelled by undulatory fins on both sides, and can perform various locomotion patterns, especially turning in situ and diving vertically. Firstly, the optimal formation problem is formulated, followed by theoretical analysis of a special case of optimal line formation. Then, a solution is proposed from the perspective of evolutionary computation. In particularly, the coordinates and the slope of the desired line formation, together with the pairings between initial positions and target positions, are obtained based on particle swarm optimization. Furthermore, we demonstrate the validity of this method by comparing the simulation results with the results of theoretical analysis. Finally, simulations results of multiple BUVs verify the feasibility of the proposed optimal formation methods.

Humans can quickly determine the force required to grasp a deformable object to prevent its sliding or excessive deformation through vision and touch, which is still a challenging task for robots. To address this issue, we propose a novel 3D convolution-based visual-tactile fusion deep neural network (C3D-VTFN) to evaluate the grasp state of various deformable objects in this paper. Specifically, we divide the grasp states of deformable objects into three categories of sliding, appropriate and excessive. Also, a dataset for training and testing the proposed network is built by extensive grasping and lifting experiments with different widths and forces on 16 various deformable objects with a robotic arm equipped with a wrist camera and a tactile sensor. As a result, a classification accuracy as high as 99.97% is achieved. Furthermore, some delicate grasp experiments based on the proposed network are implemented in this paper. The experimental results demonstrate that the C3D-VTFN is accurate and efficient enough for grasp state assessment, which can be widely applied to automatic force control, adaptive grasping, and other visual-tactile spatiotemporal sequence learning problems.

This article presents the comprehensive framework for a hybrid-driven underwater vehicle-manipulator system (HD-UVMS) to grasp marine products on the seabed. The purpose of the proposed hybrid-driven propulsion system is to improve the swimming ability of the HD-UVMS by using thrusters and enhance the stability of its pose adjustment mechanism via two unique long fin propulsors. The control mode for the thrusters and long fin propulsors is based on a fuzzy logic control method. Subsequently, a lightweight manipulator is developed to grasp marine products. The open-closed angle and current controls for the gripper help to avoid damaging marine products. A vision system is installed to enable the HD-UVMS to gradually approach marine products with the aid of monocular vision and grasp them with the aid of binocular vision. A detailed method for monocular passive ranging and stereo matching, in accordance with real-time metrics, is elaborated. Finally, relevant experiments are conducted in an indoor pool and under real sea condition to assess the effectiveness of the proposed framework. Note to Practitioners-The motivation behind this article is the design of an underwater vehicle-manipulator system that can grasp marine products on the real seabed and perform other underwater intervention tasks. Currently, the predominant method of fishing for marine products relies on human divers, which has disadvantages for human divers' health due to the long periods of time spent working underwater. In order to further study the problem, this article develops a hybrid-driven underwater vehicle-manipulator system (HD-UVMS) to work in a real seabed environment. A hybrid-driven motion control framework is presented using the thrusters to achieve effective cruising and searching for marine products and long fin propulsors for the fine pose adjustment required to grasp marine products. The proposed lightweight underwater manipulator can grasp marine products on the seabed with the aid of a vision system. A series of experiments suggests that the HD-UVMS is practical and valid.

This paper addresses the problem of path following for biomimetic underwater vehicles (BUVs) propelled by undulatory ribbon-fins. First, the general kinematics and dynamics models of underwater vehicles are presented, followed by a fuzzy logic model for dealing with a nonlinear relationship between the propulsive force/torque and the control parameters of the undulatory fins of the BUV. Then the path following problem of the BUV is formulated. A path following control paradigm integrating the line-of-sight guidance system with backstepping (BP) technique is proposed to maneuver the BUV to follow a predefined parameterized curve without time constraints. The stability of the BP controller is analyzed and guaranteed by Lyapunov stability theory. Finally, simulations and experimental results illustrate the performance of the proposed path following control paradigm.

This paper addresses the problem of path following for a biomimetic underwater vehicle (BUV) propelled by undulatory fins with uncertain model and unknown disturbance. The mechanical structure of the BUV is briefly described. Moreover, the general kinematics and dynamics models of the vehicle are presented and the path following problem is formulated. The controller combining line-of-sight (LOS) guidance system with active disturbance rejection control (ADRC) technique is designed to maneuver the BUV to follow a predefined parameterized curve. Specifically, a guidance system based on LOS principle is implemented to decouple the multi-variable system to steer the surge speed and the course respectively. Furthermore, in order to deal with model uncertainty, ADRC is used in development of the surge speed controller and the course controller. Finally, simulations and experimental results validated the performance of the proposed path following control scheme.