Sim-to-Real refers to the process of transferring policies learned in simulation to the real world, which is crucial for achieving practical robotics applications. However, recent Sim2real methods either rely on a large amount of augmented data or large learning models, which is inefficient for specific tasks. In recent years, with the emergence of radiance field reconstruction methods, especially 3D Gaussian splatting, it has become possible to construct realistic real-world scenes. To this end, we propose RL-GSBridge, a novel real-to-sim-to-real framework which incorporates 3D Gaussian Splatting into the conventional RL simulation pipeline, enabling zero-shot sim-to-real transfer for vision-based deep reinforcement learning. We introduce a mesh-based 3D GS method with soft binding constraints, enhancing the rendering quality of mesh models. Then utilizing a GS editing approach to synchronize the rendering with the physics simulator, RL-GSBridge could reflect the visual interactions of the physical robot accurately. Through a series of sim-to-real experiments, including grasping and pick-and-place tasks, we demonstrate that RL-GSBridge maintains a satisfactory success rate in real-world task completion during sim-to-real transfer. Furthermore, a series of rendering metrics and visualization results indicate that our proposed mesh-based 3D GS reduces artifacts in unstructured objects, demonstrating more realistic rendering performance.

Soft robots face significant control challenges due to the coupling effects of strain and external forces, particularly with external interactions that change their original model. This paper proposes a vision-based controller integrated with an adaptive algorithm to estimate and compensate for external contact disturbances in such environments. Specifically, the adaptive law can online estimate unknown camera parameters, eliminating the need for costly and environment-specific calibration procedures. In parallel, a contact disturbance estimation strategy is introduced to model and compensate for real-time interaction effects without the foreknowledge of the constitutive equation of the soft mechanism. The adaptive algorithm is seamlessly integrated into the adaptive image-based visual servo (IBVS) controller, allowing simultaneous calibration and contact compensation during task execution. We validated the algorithm on a tendon-driven, octoarticular soft robotic manipulator prototype. The experimental results demonstrated the algorithm’s ability to position the end-effector even under external interactions, with the adaptive parameters converging, thereby validating the effectiveness of the online estimation in assessing the impacts of interactions.

This paper presents a novel pneumatic continuum module featuring high contraction-ratio, bidirectional actuation, and active stiffness regulation. The module comprises four linear pneumatic actuators integrating rigid polygon origami frame into soft bellow. This integration not only helps to regulate motion but also enhances structural strength, facilitating contraction and bending performance. The contraction resulting from vacuum pressure serves as a limiting layer for one opposing pair of actuators, while the other pair operates in a virtual antagonistic configuration, allowing for active regulation of joint stiffness through pressure control. The paper provides a detailed workflow covering the design, fabrication, and mathematical modeling of the pneumatic module. Furthermore, the paper presents verifications of the module’s actuation capacity and active variable stiffness. The findings of this study can serve as valuable references for the design of manipulators.

In this article, to solve trajectory tracing problem and vibration suppression for a planar two-link rigid-flexible manipulator subject to joint-velocity measurement noise, a novel uncalibrated visual servoing control is proposed. To begin with, the manipulator’s dynamic model is established by the assumed mode method (AMM). On this basis, based on the singular perturbation theory, two subsystem controllers are designed, one is slow subsystem controller, and the other one is fast subsystem controller. In the slow subsystem, to cope with the complication of the camera calibration, an adaptive algorithm is formulated to evaluate the parameters of a fixed camera online. Aiming to overcome the challenge that exact joint-velocity measurement may be disturbed by external noise, a nonlinear sliding observer is developed to estimate the state of joint velocity accurately. The asymptotic convergence of image tracking error is proved by means of Lyapunov analysis. Additionally, for the purpose of restraining the flexible beam’s elastic vibration, a linear quadratic regulator (LQR) approach is adopted in the fast subsystem control design. The realistic comparing simulation experiments are presented to demonstrate the performance of the proposed controller.

Soft robots have been applied widely to various constrained scenarios due to the advantages over traditional rigid manipulators such as softness, deformability and adaptability to constrained surroundings. To make full use of this merit, this paper proposes a method that integrates collision detection, localization and force estimation for a cable-driven soft manipulator without any prior geometrical knowledge of its surroundings. First of all, a collision detection algorithm is presented based upon Cosserat-rod statics by a threshold method through using the cable tension and the shape information, which are obtained by the load cells and the Vicon system, respectively. Secondly, a collision localization and force estimation method is proposed through optimizing the discrepancy between the actual and the theoretical shapes. Finally, experiments are carried out to validate these algorithms. The experimental results demonstrate that the site, the magnitude as well as the direction can be estimated.

Place recognition and loop-closure detection are main challenges in the localization, mapping and navigation tasks of self-driving vehicles. In this paper, we solve the loop-closure detection problem by incorporating the deep-learning based point cloud description method and the coarse-to-fine sequence matching strategy. More specifically, we propose a deep neural network to extract a global descriptor from the original large-scale 3D point cloud, then based on which, a typical place analysis approach is presented to investigate the feature space distribution of the global descriptors and select several super keyframes. Finally, a coarse-to-fine strategy, which includes a super keyframe based coarse matching stage and a local sequence matching stage, is presented to ensure the loop-closure detection accuracy and real-time performance simultaneously. Thanks to the sequence matching operation, the proposed approach obtains an improvement against the existing deep-learning based methods. Experiment results on a self-driving vehicle validate the effectiveness of the proposed loop-closure detection algorithm.

In this paper, we propose a novel pose optimizer which can be inserted into either supervised or unsupervised end-to-end visual odometry for the purpose of local pose optimization. The pose optimizer is an analogue of the pose graph optimization used in traditional VSLAM algorithms. Local pose optimization is performed by an attention-based neural network which iteratively refines the predicted pose estimates of an image snippet. Instead of complicated graph convolutional network, the attention mechanism based on geometric consistency of trajectory constraint is utilized because pose features whose spatial distribution is not important can be flattened to vectors and then processed. The pose optimizer is aimed at improving pose estimation accuracy by redistributing errors of pose estimates. Quantitative and qualitative evaluation of the proposed approach on the KITTI Odometry dataset [1] is presented to demonstrate its effectiveness in improving pose estimation accuracy and minimizing pose drift.