For field tasks, it is quite challenged to operate in a complex environment for the underwater robots, especially for those with multiple control surfaces due to different response and gain characteristics. To this end, this paper develops a highly integrated robotic dolphin followed by a robust motion control system. For better maneuverability and fault-tolerant capabilities, a newly-designed robotic dolphin is presented, owning a wide array of sensors and multiple control surfaces, in which passive flukes are particularly applied. On this basis, a robust motion control system is proposed, including a depth controller based on velocity-related allocation strategies and a heading controller based on clearance compensation. In detail, considering the degradation of motion performance caused by passive flukes, a sliding mode controller for gain uncertainty and an allocation-related parameter tuning strategy for inputs response characteristics are designed. Extensive simulations and aquatic experiments are conducted, and the obtained results demonstrate the satisfied maneuverability of the designed prototype and the effectiveness of the proposed methods. This study can lay a foundation for further development of robotic dolphins with a robust motion system to execute complex tasks in the field.Note to Practitioners—This paper is inspired by the issue of robust motion control system for a newly-designed practical robotic dolphin that possesses a passive tail and redundant control surfaces. The traditional methods are usually susceptible to uncertainties in the passive tail gain, exhibiting degraded control performance. Moreover, control oscillations and slow convergence speed often occur caused by neglecting the characteristics of different control surfaces, including response patterns and clearance. This paper suggests a robust depth controller based on velocity-related allocation strategies and a robust heading controller based on clearance compensation. Specifically, an allocation-related parameter tuning strategy is given by considering inputs response characteristics, including response speed, saturations, and hydrodynamic force variation patterns. To guarantee fine regulations of heading control, a nonlinear disturbance observer (NDOB)-based clearance compensation is proposed. Extensive aquatic experiments on the newly-designed robotic dolphin verified the effectiveness of the proposed methods. It is envisioned that all these presented results can provide valuable engineering practice insights for industry practitioners.

The development of bionic underwater robots has brought new vitality to ocean exploration. Motion control is crucial for the stability of underwater robots due to significant differences in flow field characteristics at various swimming speeds. This study focuses on vertical-plane motion and proposes a model predictive control method to achieve integrated control of depth position and pitch attitude for bionic robotic fish. First, based on a robotic tuna system, high-maneuverability vertical-plane motion configuration elements are analyzed and summarized, laying the foundation for motion stability and controllability. Second, through hydrodynamic sampling in aquatic environments, a system model covering the range of swimming speeds is established. Regarding the control method, the proposed motion planning approach converts the desired motion sequence into an equivalent “pitch-depth” trajectory curve. A nonlinear model predictive controller (NMPC) is then designed to track the trajectory curve, ultimately achieving the desired vertical-plane motion. Experimental results validate that the proposed method not only ensures control accuracy under both low and high-speed conditions, but also enables the execution of complex motion sequence control. This study provides a fresh perspective on the motion instability analysis of robotic fish at high swimming speed and a novel control framework for regulating continuous posture sequences in the vertical plane.Note to Practitioners—The motivation of this paper is to address the challenges associated with stable motion and control of robotic fish in the vertical plane, given the variability of flow field characteristics at different swimming speeds. Existing methods for controlling pitch attitude and depth in bionic underwater robots are typically designed for stable flow conditions encountered during low-speed swimming. However, the instability and agility of high-speed robotic fish movements have not been adequately considered. Additionally, the coupling between pitch attitude and depth poses challenges for joint control of their combined states. This paper proposes a configuration analysis and control methodology to achieve desired vertical-plane locomotion for robotic fish. Specifically, using a robotic tuna as the research subject, a configuration analysis method for high-maneuverability motion in the vertical plane is presented, providing a foundation for ensuring motion stability and controllability. To accurately evaluate the motion of robotic fish under varying flow speeds, a system model for vertical plane motion is constructed based on hydrodynamic data collected from aquatic environments. A motion planning approach is proposed to convert desired vertical plane motion sequences into controllable “pitch-depth” trajectory curves, and a nonlinear model predictive controller is designed to track these trajectories. Configuration simulations and control experiments validate the effectiveness of the proposed method. Hopefully, our proposed methods can provide valuable insights and support for high-maneuverability motion control and continuous posture sequences tracking of bionic underwater robots in the vertical plane.

Directly learning the swimming behaviors of real fish can significantly enhance the swimming performance of bionic robotic fish. This paper presents a novel transfer learning method based on a dynamic trajectory control approach for the robotic fish to learn swimming skills from real fish. First, we develop a fish motion capture system and a crucial motion extraction approach to realize precise decomposition of fish motions and collect abundant meaningful features from a snakehead fish as pre-training data. Next, we construct a two-stage transfer learning method based on Deep Deterministic Policy Gradient (DDPG), including an offline and an online stage. Specifically, in the offline stage, the obtained pre-training data is processed for experience learning within a DDPG-based network, whereas in the online stage, a dynamic trajectory tracking method is utilized to refine the robotic fish’s motions in real time based on the learned strategies. Experimental results on a self-developed four-joint robotic fish show that the proposed method effectively extracts and transfers biological motion features into the motion control of the robotic fish. Compared to the conventional CPG method, the proposed approach exhibits stronger acceleration capabilities and more efficient swimming, resulting in enhanced maneuverability of the robotic fish. Overall, this approach provides a technical foundation for bionic robotics to learn from nature.

In this article, we propose a tightly coupled visual-inertial-acoustic sensor fusion method to improve the autonomous localization accuracy of underwater vehicles. To address the performance degradation encountered by existing visual or visual-inertial simultaneous localization and mapping systems when applied in underwater environments, we integrate the Doppler velocity log (DVL), an acoustic velocity sensor, to provide additional motion information. To fully leverage the complementary characteristics among visual, inertial, and acoustic sensors, we perform multimodal information fusion in both frontend tracking and backend mapping processes. Specifically, in the frontend tracking process, we first predict the vehicle's pose using the angular velocity measurements from the gyroscope and linear velocity measurements from the DVL. Thereafter, measurements performed by the three sensors between adjacent camera frames are utilized to construct visual reprojection error, inertial error, and DVL displacement error, which are jointly minimized to obtain a more accurate pose estimation at the current frame. In the backend mapping process, we utilize gyroscope and DVL measurements to construct relative pose change residuals between keyframes, which are minimized together with visual and inertial residuals to further refine the poses of the keyframes within the local map. Experimental results on both simulated and real-world underwater datasets demonstrate that the proposed fusion method improves the localization accuracy by more than 30% compared to the current state-of-the-art ORB-SLAM3 stereo-inertial method, validating the potential of the proposed method in practical underwater applications.

In nature, fish have evolved sophisticated muscular systems that enable them to dynamically regulate their body movements for efficient and agile swimming, which has inspired the development of compact and fast flexibility regulation mechanisms in robotic fish. While existing robotic fish have primarily relied on passive flexible mechanisms and tunable stiffness mechanisms, these approaches often lack the dynamic adjustment capabilities that are characteristic of living fish. This article proposes a novel biomimetic flexible fishtail capable of dynamically controlling its deformation through artificial muscles made from macrofiber composite. In detail, the fishtail is equipped with a servo motor as the sole driving joint, while the artificial muscles regulate the deformation to indirectly adjust stiffness. A dynamic model considering both flexibility and hydrodynamics is established, and a partial differential equation observer is particularly developed to estimate the tail's full states. Subsequently, a deformation control framework incorporating a deep reinforcement learning strategy is constructed and successfully deployed on an embedded platform via lightweight design. Simulation and experimental results validate the accuracy and effectiveness of the dynamic model, observer, and control strategy. Especially, the proposed fishtail demonstrates the ability to enhance propulsion in fishlike swimming modes across various frequencies, ranging from 15% to 203%. When assembled into an untethered robotic prototype, deformation control allows the prototype's swimming speed to vary, achieving up to 42% slower or 37% faster speeds compared to passive compliance. Its rapid adjustability and adaptability to different frequencies represent significant advancements not widely reported in previous studies. The obtained results will offer some significant insights for flexible robotic systems to enhance their agility and interactivity.

Artificial lateral line (ALL) sensors hold the potential to enhance the perception abilities of robotic fish by capturing surface pressure gradients and identifying near-field object, such as dipole source. However, the robotic fish’s free-swimming motion introduces periodic low-frequency noise into the ALL data, while dipole sources with time-varying positions generate pressure signals with complex time-frequency characteristics. This paper proposes a complete solution to these challenges that would enable freely swimming robotic fish to locate dipole source. Firstly, an ALL system consisting of pressure sensors is integrated into the robotic fish, further constructing a real-time data acquisition and processing system. Secondly, to effectively estimate and remove the swimming-induced noise from the ALL data, a noise estimation model is developed based on the bionic motion mode and unsteady Bernoulli equation. Subsequently, short-time Fourier transform is applied to the high-quality data after noise elimination, followed by developing a convolution regression neural network for feature extraction and dipole source localization. Finally, extensive simulations and experiments are conducted to validate the effectiveness of the proposed methods and perform the positive impact of the noise estimation model. Remarkably, within the range of perception, the average accuracy of dipole source location can reach 13.6 mm, providing a promising reference for improving the perception abilities of underwater robots.Note to Practitioners—This paper is motivated by the problem of blind zones in near-field perception of underwater robots. The existing perception methods as visual sensing are limited by the dark and cloudy underwater environment, and are powerless in near-field localization. In addition, the artificial lateral line, as a potential near-field sensor, is challenging to be applied in self-propelled robots due to the swimming noise. This paper proposes an integrated near-field sensory system that includes an ALL sensor, a swimming noise elimination method and a dipole source localization method. Specifically, a fish-inspired ALL sensor is designed by high-accuracy pressure sensors and integrated into the robotic fish. To enhance localization performance, a swimming noise elimination model is constructed based on unsteady Bernoulli equation. Furthermore, a convolution regression network is developed for accurate localization of near-field objects. A series of simulations and experiments demonstrate the effectiveness and superiority of the proposed near-field sensory system. Hopefully, our proposed methods can provide valuable guidance and support for near-field object localization to improve the intelligent operation ability of underwater bionic robots, such as cooperative control, underwater navigation, environment exploration, and so forth.

The application potential of robotic fish embedded with intelligent visual navigation algorithms in underwater autonomous operation is on full display recently. However, the existing visual navigation methods are limited by the underwater visual conditions and the motion characteristics of robotic fish. To this end, this paper proposes a novel autonomous navigation framework integrated with visual stabilization control. In practice, a stereo vision-based navigation network is proposed to generate the guidance law. On this basis, a biomimetic robotic hammerhead shark with a controllable cephalofoil is developed, and a nonlinear model predictive controller for cephalofoil stabilization relying on the dynamic model is elaborately designed. Extensive simulations and underwater experiments are conducted to validate the effectiveness and superiority of the proposed methods, which significantly enhance exploration efficiency and reduce image jitter by 26.02% compared to the traditional methods. The obtained results provide a new idea for underwater robots to autonomously explore the ocean.Note to Practitioners—This paper is motivated by the problem of vision-based underwater autonomous navigation for a biomimetic robotic fish that possesses underwater visual stability and good maneuverability. The existing visual navigation networks usually generate unexpected navigation instructions when dealing with complex or ambiguous underwater scenes. Additionally, image jitter caused by the rhythmic motion of robotic fish can lead to navigation failure. This paper suggests an integrated navigation framework that includes a biomimetic platform design, a visual stabilization controller, and an intelligent underwater navigation network. Specifically, a novel sphyrnidae-inspired robotic shark is designed as a new platform with superior motion performance. To enhance underwater visual stability, a nonlinear model predictive control-based visual stabilization controller is proposed. Furthermore, a deep stereo attention navigation network based on a parallax attention mechanism is proposed to improve the generalization of vision-based autonomous navigation. A series of underwater search experiments on the robotic shark demonstrate the effectiveness and superiority of the proposed navigation framework. Hopefully, our proposed methods can provide valuable guidance and support for universal underwater robot navigation to accomplish practical marine tasks, such as underwater rescue, resource exploitation, biological observation, and so on.

The broad and powerful pectoral fins of manta rays are crucial to their efficient and maneuverable swimming. However, very little is currently known about the pectoral-fin-driven 3-D locomotion of manta-inspired robots. This study is focused on the development and 3-D path-following control of an agile robotic manta. First, a novel robotic manta with 3-D mobility is constructed, of which the distinctive pectoral fins provide the only propulsion. Specifically, the unique pitching mechanism is detailed in which the time-coupled coordination movement of the pectoral fins is applied. Second, based on a 6-axis force measuring platform, the propulsion characteristics of the flexible pectoral fins are analyzed. Then, the force-data-driven 3-D dynamic model is further established. Third, a control scheme combined with a line-of-sight (LOS) guidance system and a sliding-mode fuzzy controller is conceived, addressing the 3-D path-following task. Finally, various simulated and aquatic experiments are conducted, demonstrating the superior performance of our prototype and the effectiveness of the proposed path-following scheme. This study will hopefully generate fresh insights into the updated design and control of agile bioinspired robots performing underwater tasks in dynamic environments.

For few-shot object detection, this work proposes a binary similarity detector (BSDet), which realizes a novel similarity-based multiple binary classification and enhances the feature margin between positive and hard negative samples. First, we revisit the classification paradigm, concluding that multiple binary classification paradigm is more suitable than multi-class classification paradigm for the few-shot task. Hence, we propose a binary similarity head (BSH) by posing the classification task as multiple binary similarity measurements rather than a multi-class prediction. Second, focusing on the hard negative samples, we propose a feature enhancement module (FEM). During training phase, the FEM can push the features of positive and hard negative samples far away from each other, and thus effectively suppresses false positives. Abundant experiments and visualizations indicate that our method achieves state-of-the-art performances on few-shot object detection tasks.

As a key technology for autonomous underwater operations, precise tracking control in tight space environments is a great challenge. With the aid of high maneuverability of the underwater bionic robot, this article proposes an integrated tracking control framework for a robotic dolphin to move through narrow areas, including top-level planning, middle-level tracking, and bottom-level control allocation. First, a nonlinear model predictive control-based planning method is presented with full consideration of tracking accuracy and obstacle avoidance safety. Second, in order to improve the anti-interference ability, we derive a nonlinear path tracking control law by combining the backstepping technique with a nonlinear disturbance observer. More importantly, through hydrodynamic analysis of the bionic multimodal motions under flippers and flukes, a fuzzy-based nonlinear control allocation system is particularly adopted to convert calculated control forces into bionic motion parameters. Finally, extensive simulations and aquatic experiments are conducted, and the obtained results validate the effectiveness of proposed methods, providing a new idea to further ocean exploration.

This paper proposes a tightly-coupled visual-Doppler-Velocity-Log (visual-DVL) fusion method for underwater robot localization through integrating the velocity measurements from a DVL into a visual odometry (VO). Considering that employing the DVL measurements in dead-reckoning systems easily leads to error accumulation and suboptimal results in previous works, we directly integrate them into the visual tracking process. Specifically, the velocity measurements are utilized to improve the initial estimation of camera pose during visual tracking, aiming to provide a better initial value for pose optimization. Thereafter, these velocity measurements are also directly employed to constrain the position change of the camera between two adjacent frames by constructing a novel DVL error term, which is optimized jointly with the visual constrains to obtain a more accurate camera pose. Various experiments are carried out in the datasets collected from several scenarios of the underwater simulation environment HoloOcean, and the results illustrate that the proposed fusion method can effectively improve the localization accuracy for underwater robots by about 20% compared to pure visual odometry. The proposed method provides valuable guidance for the accurate localization of underwater robots.

This article proposes a novel barrier-based adaptive line-of-sight (ALOS) three-dimensional (3-D) path-following system for an underactuated multijoint robotic fish. The framework of the developed path-following system is established based on a detailed dynamic model, including a barrier-based ALOS guidance strategy, three integrated inner-loop controllers, and a nonlinear disturbance observer (NDOB)-based sideslip angle compensation, which is employed to preserve a reliable tracking under a frequently varying sideslip angle of the robotic fish. First, a barrier-based convergence strategy is proposed to deal with probable along-track error disruption and suppress the error within a manageable range. Meanwhile, an improved adaptive guidance scheme is adopted with an appropriate look-ahead distance. Afterward, a novel NDOB-based sideslip angle compensation is put forward to identify the varying sideslip angle independent of speed estimation. Subsequently, inner-loop controllers are intended for regulation about the controlled references, including a super-twisting sliding-mode control (STSMC)-based speed controller, a global fast terminal sliding-mode control (GFTSMC)-based heading controller, and a GFTSMC-based depth controller. Finally, simulations and experiments with quantitative comparison in 3-D linear and helical path following are presented to verify the effectiveness and robustness of the proposed system. This path-following system provides a solid foundation for future marine autonomous cruising of the underwater multijoint robot.

Within the last few years, great efforts have been made to study few-shot learning. Although general object detection is advancing at a rapid pace, few-shot detection remains a very challenging problem. In this work, we propose a novel decoupled metric network (DMNet) for single-stage few-shot object detection. We design a decoupled representation transformation (DRT) and an image-level distance metric learning (IDML) to solve the few-shot detection problem. The DRT can eliminate the adverse effect of handcrafted prior knowledge by predicting objectness and anchor shape. Meanwhile, to alleviate the problem of representation disagreement between classification and location (i.e., translational invariance versus translational variance), the DRT adopts a decoupled manner to generate adaptive representations so that the model is easier to learn from only a few training data. As for a few-shot classification in the detection task, we design an IDML tailored to enhance the generalization ability. This module can perform metric learning for the whole visual feature, so it can be more efficient than traditional DML due to the merit of parallel inference for multiobjects. Based on the DRT and IDML, our DMNet efficiently realizes a novel paradigm for few-shot detection, called single-stage metric detection. Experiments are conducted on the PASCAL VOC dataset and the MS COCO dataset. As a result, our method achieves state-of-the-art performance in few-shot object detection. The codes are available at https://github.com/yrqs/DMNet.

Robust vision restoration of underwater images remains a challenge. Owing to the lack of well-matched underwater and in-air images, unsupervised methods based on the cyclic generative adversarial framework have been widely investigated in recent years. However, when using an end-to-end unsupervised approach with only unpaired image data, mode collapse could occur, and the color correction of the restored images is usually poor. In this paper, we propose a data- and physics-driven unsupervised architecture to perform underwater image restoration from unpaired underwater and in-air images. For effective color correction and quality enhancement, an underwater image degeneration model must be explicitly constructed based on the optically unambiguous physics law. Thus, we employ the Jaffe-McGlamery degeneration theory to design a generator and use neural networks to model the process of underwater visual degeneration. Furthermore, we impose physical constraints on the scene depth and degeneration factors for backscattering estimation to avoid the vanishing gradient problem during the training of the hybrid physical-neural model. Experimental results show that the proposed method can be used to perform high-quality restoration of unconstrained underwater images without supervision. On multiple benchmarks, the proposed method outperforms several state-of-the-art supervised and unsupervised approaches. We demonstrate that our method yields encouraging results in real-world applications.

Underwater robotic operation usually requires visual perception (e.g., object detection and tracking), but underwater scenes have poor visual quality and represent a special domain which can affect the accuracy of visual perception. In addition, detection continuity and stability are important for robotic perception, but the commonly used static accuracy based evaluation (i.e., average precision) is insufficient to reflect detector performance across time. In response to these two problems, we present a design for a novel robotic visual perception framework. First, we generally investigate the relationship between a quality-diverse data domain and visual restoration in detection performance. As a result, although domain quality has an ignorable effect on within-domain detection accuracy, visual restoration is beneficial to detection in real sea scenarios by reducing the domain shift. Moreover, non-reference assessments are proposed for detection continuity and stability based on object tracklets. Further, online tracklet refinement is developed to improve the temporal performance of detectors. Finally, combined with visual restoration, an accurate and stable underwater robotic visual perception framework is established. Small-overlap suppression is proposed to extend video object detection (VID) methods to a single-object tracking task, leading to the flexibility to switch between detection and tracking. Extensive experiments were conducted on the ImageNet VID dataset and real-world robotic tasks to verify the correctness of our analysis and the superiority of our proposed approaches. The codes are available at https://github.com/yrqs/VisPerception .

A heuristic fast marching (FM*)-based comprehensive path planning system involving task allocation, initial planning, and replanning is presented for the robotic floating garbage cleaning mission. There are three primary contributions in this paper. First, to tackle the invalidation of the Euclidean distance metric in the obstacle environment, the task allocation is modeled as a travelling salesman problem (TSP) employing the FM*-based distance metric in order to obtain an optimal travel sequence. Second, to meet the maneuverability constraint from the surface robot and avoid the collision, a Gaussian filter is employed to adjust the curvature radius of the generated path. Third, for an efficient replanning, a neural network-based replanning point generator with the input of garbage movement vector is provided to strike a compromise for the distance cost and the computational burden. Moreover, a case study and a virtual obstacle experiment in the laboratory water tank demonstrate the feasibility of the proposed comprehensive path planning system. This work lays a firm foundation for the development of intelligent equipment for aquatic environment protection.

This article proposes the mechanical design and dynamic model of an innovative manta-inspired robot system for both efficient fast swimming and high spatial maneuverability. Inspired by some biological studies, a pair of unique pectoral fins with six separate degrees of freedoms (DOFs) are developed. The novel design is characterized by an improved crank-rocker mechanism and a distinctive horizontal DOF. The former not only endows the robot with high swimming speed, but also guarantees efficient flapping patterns which are close to manta rays. The latter is employed to coordinate with the flapping movement, allowing remarkable pitch adjustment. Further, the basic motion strategy is presented by detailed analyses to the pectoral fins. Besides, based on the Morrison equation and infinitesimal method, a complete dynamic model for robotic manta with flexible pectoral fins is established, whose parameters are determined through experimental data. Moreover, the linear swimming and pitching experiments are conducted, demonstrating the prominent movement performance of the presented design and the effectiveness of the dynamic model. The obtained results shed light on updated design and control of next-generation agile underwater vehicles and robots capable of multimodal motions in dynamic and complex aquatic environments.

This paper proposes a multi-objective visionbased navigation network for biomimetic underwater robots to cope with scientific observation, target selection, and obstacle avoidance in marine missions. Structurally, a stereo block attention module is first constructed to serially extract the channel and spatial attention portion of the real-time visual feedback. Next, the parallax attention mechanism is introduced to enable the network to excavate implicit parallax information in stereo pairs, effectively eliminating the oscillation of the network output in the presence of ambiguous visual input. Further, with the assistance of other low-cost sensors, the proposed navigation network can be expanded in some largescale application scenarios, such as sparse coral observation. Finally, underwater simulations reveal that the proposed method obtains significantly improved control effect and real-time ability, compared with other related works. In particular, based on a self-developed biomimetic robotic dolphin, collision-free simulations with a cumulative distance beyond 1000 m were carried out and validated the effectiveness and the superiority of the navigation network, where both dense and sparse targets were fully tested. The robotic dolphin can not only successfully conduct accurate coral observation without collision, but also quest the observation targets as much as possible in the area where the observation targets are concentrated. The proposed network provides an intelligent and efficient navigation scheme for autonomous underwater operation of small-size underwater robots.

Underwater visual localization is an essential technique for the autonomous operation of underwater robots. However, the unique underwater image characteristics, including refraction, sparse features, and severe noise, pose an enormous challenge to it. For addressing these issues, this paper proposes an open-source fiducial-based underwater stereo visual-inertial localization method under the extended Kalman filter (EKF) framework, which is called FBUS-EKF. First, the refraction is corrected by the refractive camera model and akin triangulation. Second, the fiducial marker and a novel marker pose estimation method are applied to alleviate the adverse effect of sparse features. Third, the EKF is utilized to fuse the inertial and visual information so as to reject the serious noise. Finally, extensive experiments on a test bench demonstrate the effectiveness of the FBUS-EKF method, where the typical localization error is less than 3%, namely, the average error is lower than 3 cm within one meter. The obtained results reveal that the FBUS-EKF method has the prospect to be applied in the precise short-range operation and the localization for underwater robots, which offers a valuable insight for further autonomous underwater task.

Object detection has been vigorously investigated for years but fast accurate detection for real-world scenes remains a very challenging problem. Overcoming drawbacks of single-stage detectors, we take aim at precisely detecting objects for static and temporal scenes in real time. Firstly, as a dual refinement mechanism, a novel anchor-offset detection is designed, which includes an anchor refinement, a feature location refinement, and a deformable detection head. This new detection mode is able to simultaneously perform two-step regression and capture accurate object features. Based on the anchor-offset detection, a dual refinement network (DRNet) is developed for high-performance static detection, where a multi-deformable head is further designed to leverage contextual information for describing objects. As for temporal detection in videos, temporal refinement networks (TRNet) and temporal dual refinement networks (TDRNet) are developed by propagating the refinement information across time. We also propose a soft refinement strategy to temporally match object motion with the previous refinement. Our proposed methods are evaluated on PASCAL VOC, COCO, and ImageNet VID datasets. Extensive comparisons on static and temporal detection verify the superiority of DRNet, TRNet, and TDRNet. Consequently, our developed approaches run in a fairly fast speed, and in the meantime achieve a significantly enhanced detection accuracy, i.e., 84.4% mAP on VOC 2007, 83.6% mAP on VOC 2012, 69.4% mAP on VID 2017, and 42.4% AP on COCO. Ultimately, producing encouraging results, our methods are applied to online underwater object detection and grasping with an autonomous system. Codes are publicly available at https://github.com/SeanChenxy/TDRN.

In this article, an extended state observer (ESO)-based controller with a model predictive governor is designed for 3-D trajectory tracking of underactuated underwater vehicles. The proposed control scheme takes three primary challenges including underactuated property, velocity constraint, and lumped disturbance into consideration. With respect to the model predictive governor, an underactuated kinematic tracking error model is utilized to produce reference velocities. Meanwhile, a heading angle compensation mechanism is utilized to avoid the steady tracking errors resulting from dynamics coupling of the vehicle. Besides, an ESO is designed to estimate the lumped disturbances and unmeasured velocity states. Based on the ESO, a kinetic controller is offered to accomplish the precise velocity tracking only in virtue of the position and orientation information. Note that this article details both the design process of the control scheme and rigorous theoretical analysis. Eventually, simulation and experimental results demonstrate the feasibility and superiority of the proposed method. Notably, this work lays the foundation for the underactuated trajectory tracking control in complicated and turbulent underwater environments.

This article presents the mechanical design and locomotion control of a novel tuna-inspired robot system for both fast swimming and high maneuverability. Mechanically, the developed robotic fish named CasiTuna comprises three important parts, i.e., an innovative two-motor-actuated propulsive mechanism, a buoyancy adjustment structure, and a pair of pectoral fins. Unlike most robotic fishes' multiple concatenated links-based propulsive mechanism, CasiTuna's two-motor-actuated one places both motors in the anterior body and utilizes a transmission system to achieve tuna-like lateral undulations. Meanwhile, the buoyancy adjustment mechanism in conjunction with pectoral fins endows the robot with the capability of three-dimensional maneuverability. Kinematic and dynamic analyses are further conducted to reveal the interactive hydrodynamic forces. Regarding the locomotion control method, a bio-inspired central pattern generator-based controller is adopted to achieve multimodal swimming. In particular, two kinds of turning maneuvers are implemented and discussed. Aquatic experiments, including straight swimming, circular turning, and nearly static pitching validate the effectiveness of proposed mechatronic design and locomotion control methods. Remarkably, CasiTuna achieved a peak forward speed of 0.8 m/s (corresponding to 1.52 body lengths per second) and a minimum turning radius of less than 0.3 body lengths.

In this article, a robot system for intelligent water surface cleaner named IWSCR is developed to collect floating plastic garbage. It is able to accomplish three major tasks autonomously, i.e., cruise and detection, tracking and steering, and grasping and collection. The challenges behind these tasks involve how to realize the accurate and real-time garbage detection, how to resist the disturbances while IWSCR conducts vision-based steering, and how to grasp the floating garbage reliably despite the turbulent conditions on the surface of the water. To overcome these difficulties, three key techniques are proposed for IWSCR. First, the YOLOv3 network, which is widely applied in the high speed and accuracy object detection field, is trained on the proposed floating garbage dataset to realize accurate and real-time garbage detection. Next, to improve the ability of resisting disturbances, a control law based on the sliding-mode controller is proposed for vision-based steering. Furthermore, inspired by the stability of floating bottles in fluid, a feasible grasping strategy is utilized for IWSCR. Finally, the experimental results demonstrate that IWSCR is competent to carry out the task of water surface cleaning.

This article proposes a modified rapidly exploring random tree (RRT)-based path planner and behavior-based cooperative tracking strategy for a dual robotic dolphin system to fulfill a cooperative target-tracking task. Specifically, with full consideration of both task requirements and mechatronic configuration, a robotic dolphin with a waist-caudal propulsive mechanism for thrust forces and differential bilateral flippers for maneuverability is developed. To satisfy the demand of fast path planning, a variant RRT algorithm is employed to generate feasible paths for the dual robotic dolphins with fewer waypoints, faster convergence speed, and better stability. Furthermore, a behavior-based approach in conjunction with centralized architecture is implemented to achieve high-level decision-making. Finally, simulations, analysis, as well as field experiments are carried out to verify the effectiveness of the proposed control scheme. The success of the experiments further offers insight into the mechanisms of cooperative multirobot target tracking in aquatic environments.

This paper presents a three-dimensional (3-D) path planning method that combines the gliding with dolphin-like motions for the gliding robotic dolphin. A specific task that the robot uses the gliding motion for long-distance cruise and the dolphin-like motion for maneuverable obstacle avoidance is employed. The results of simulations and aquatic experiments validate the full-state dynamic model and the specific task, further offer some theoretical supports for 3-D path planning. Further, the 3-D path planning method is composed of three main components: 1) gliding path generation; 2) improved Astar (A*) algorithm; and 3) segmented Bezier curve smoothing. First, the gliding path is generated autonomously with the kinematic constraints that are obtained via the simulations of dynamic model. Furthermore, when the obstacles are detected by the sonar, an improved A* algorithm is employed to avoid the obstacles. Afterward, considering the path planned by A* is unsmoothed, a segmented Bezier curve method is presented. Simulation results demonstrate the effectiveness of the method, offering valuable insight into the utilization of hybrid underwater robots in the context of real-time task execution.

This article proposes a model predictive control (MPC)-based depth control system for the gliding motion of a gliding robotic dolphin. An injector-based buoyancy-driven mechanism is employed to achieve more precise control of net buoyancy. In the system, a novel framework of depth control is proposed on the basis of a simplified model, including a depth controller with improved MPC, a heading controller with velocity-based proportional-integral-derivative, and a sliding mode observer. Extensive simulation and experimental results demonstrate the effectiveness of the proposed control methods. In particular, a variety of slider-based experiments are also conducted to explore the performance of a movable slider in the depth control so as to better govern the gliding angle. The results obtained reveal that it is feasible to realize regular gliding angles via regulating the slider, which offers promising prospects for bio-inspired gliding robots playing a key role in ocean exploration.

Implementing underwater target tracking remains difficult for a free-swimming robotic fish owing to the intrinsically reciprocating motion in fishlike propulsion. In this article, we present a novel robotic fish platform with a camera stabilizing system and achieve real-time two-dimensional target tracking assisted by reinforcement learning (RL) in continuous environments. More specifically, we first develop an active visual tracking system based on cascade control structure to obtain the relative orientation between the robotic fish and the underwater target. Then, we propose a target tracking controller dealing with continuous state and action spaces based on deep RL (DRL). The controller takes the position of the target object as input and yields the motion parameters of the bioinspired central pattern generator governed robotic fish. The robustness and adaptability of the proposed controller as well as the influence of time-delays on the control system are explored via simulated experiments under different scenarios. Finally, both static and dynamic tracking experiments on the actual robotic fish demonstrate the effectiveness of the proposed mechatronic design and control methods, providing insights to executing aquatic vision-based tracking tasks.

Temporal object detection has attracted significant attention, but most popular detection methods cannot leverage rich temporal information in videos. Very recently, many algorithms have been developed for video detection task, yet very few approaches can achieve real-time online object detection in videos. In this paper, based on the attention mechanism and convolutional long short-term memory (ConvLSTM), we propose a temporal single-shot detector (TSSD) for real-world detection. Distinct from the previous methods, we take aim at temporally integrating pyramidal feature hierarchy using ConvLSTM, and design a novel structure, including a low-level temporal unit as well as a high-level one for multiscale feature maps. Moreover, we develop a creative temporal analysis unit, namely, attentional ConvLSTM, in which a temporal attention mechanism is specially tailored for background suppression and scale suppression, while a ConvLSTM integrates attention-aware features across time. An association loss and a multistep training are designed for temporal coherence. Besides, an online tubelet analysis (OTA) is exploited for identification. Our framework is evaluated on ImageNet VID dataset and 2DMOT15 dataset. Extensive comparisons on the detection and tracking capability validate the superiority of the proposed approach. Consequently, the developed TSSD-OTA achieves a fast speed and an overall competitive performance in terms of detection and tracking. Finally, a real-world maneuver is conducted for underwater object grasping.

Most existing robotic fish have a large body size driven by servo motor system, while conventional small-sized actuators hardly generate a high swimming performance. This paper reports a miniature untethered robotic fish, whose body length is 69 mm. In particular, a newly designed magnetic actuator system (MAS) is equipped, which guarantees both small-sized dimension and flexibility of the robot. More specifically, the magnetic field generated by a permanent magnet is first investigated based on Biot-Savart law. Then, a novel tail-beating rhythm called magnetically actuated pulse width modulation (MAPWM) is modeled for the new actuator system. Further, an MAPWM-based control method is presented, in which the duty ratio of MAPWAM is innovatively utilized to realize the turning maneuvers for the first time. In addition, Lagrangian method is employed to establish the dynamic model to assess the MAPWM-based control method and the turning performance of the robotic fish. To further improve the maneuverability, the effect of a shape-variable caudal fin is analyzed based on computational fluid dynamics and the built dynamic model. Finally, combined with the MAS, the MAPWM-based control method, and the optimally selected caudal fin, extensive aquatic experiments are conducted on the robotic prototype. The results indicate that the developed miniature robotic fish achieves a considerably higher level of maneuverability in terms of turning radius when compared to swimming robots with equivalent dimensions.

Object detection methods fall into two categories, i.e., two-stage and single-stage detectors. The former is characterized by high detection accuracy while the latter usually has a considerable inference speed. Hence, it is imperative to fuse their merits for a better accuracy vs. speed trade-off. To this end, we propose a dual refinement network (DRN) to boost the performance of the single-stage detector. Inheriting from the advantages of two-stage approaches (i.e., two-step regression and accurate features for detection), anchor refinement and feature offset refinement are conducted in a novel anchor-offset detection, where the detection head is comprised of deformable convolutions. Moreover, to leverage contextual information for describing objects, we design a multi-deformable head, in which multiple detection paths with different receptive field sizes devote themselves to detecting objects. Extensive experiments on PASCAL VOC and ImageNet VID datasets are conducted, and we achieve a state-of-the-art detection performance in terms of both accuracy and inference speed.

Neural networks always require extensive training samples. However, in some special applications, i.e., insulators in high power grid, it is very hard and costly to collect variety-rich samples. In this study, a synthetic method is proposed to generate segmentation training samples for the insulators. Instead of relying on full-fledged Computer Graphic, this study focuses on the training features of neural networks. Based on this synthetic approach, many kinds of insulators samples including positive, empty and fake ones can be constructed, and their quantities are particularly balanced by an equalization strategy. In order to validate these produced samples, three end-to-end segmentation networks are employed to adapt to the generators in an adversarial training framework. Meanwhile, an improved training strategy is utilized to speed up the convergence. Finally, extensive experiments are executed to further analyze the proposed synthetic method and demonstrate its effectiveness for insulators segmentation.

Temporal object detection has attracted significant attention, but most popular methods can not leverage the rich temporal information in video or robotic vision. Although many different algorithms have been developed for video detection task, real-time online approaches are frequently deficient. In this paper, based on attention mechanism and convolutional long short-term memory (ConvLSTM), we propose a temporal single-shot detector (TSSD)for robotic vision. Distinct from previous methods, we aim to temporally integrate pyramidal feature hierarchy using ConvLSTM, and design a novel structure including a high-level ConvLSTM unit as well as a low-level one (HL-LSTM)for multi-scale feature maps. Moreover, we develop a creative temporal analysis unit, namely, ConvLSTM-based attention and attention-based ConvLSTM (A&CL), in which the ConvLSTM-based attention is specially tailored for background suppression and scale suppression while the attention-based ConvLSTM temporally integrates attention-aware features. Finally, our method is evaluated on ImageNet VID dataset. Extensive comparisons on detection performance confirm the superiority of the proposed approach, and the developed TSSD achieves a considerably enhanced accuracy vs. speed trade-off, i.e., 64.8% mAP vs. 27 FPS.

In this brief, we investigate the parameter optimization issue of a central pattern generator (CPG) network governed forward and backward swimming for a fully untethered, multijoint biomimetic robotic fish. Considering that the CPG parameters are tightly linked to the propulsive performance of the robotic fish, we propose a method for determination of relatively optimized control parameters. Within the framework of evolutionary computation, we use a combination of dynamic model and particle swarm optimization (PSO) algorithm to seek the CPG characteristic parameters for an enhanced performance. The PSO-based optimization scheme is validated with extensive experiments conducted on the actual robotic fish. Noticeably, the optimized results are shown to be superior to previously reported forward and backward swimming speeds.