In this paper, we propose an adaptive eye-to-hand vision-based control methodology, which enables a closed-loop grow-to-shape capability for variable length continuum manipulators in 2D. Our method utilizes shape features of the continuum robot, i.e. module curvature and length, which are obtained from the image. Our adaptive control algorithm servos the robot to converge and track the desired values of these features in the image space without the need of a robot model. As a result the robot starts from a minimum length configuration and grows into a given desired shape, always staying on the course of the desired shape. We believe that this approach unlocks capabilities for variable length continuum robots by leveraging their actuation redundancy and avoiding obstacles while carrying out object manipulation or inspection tasks in cluttered and constrained environments. We perform experiments in simulations and on a real robot to assess the performance of our visual servoing algorithm. Our experimental results demonstrate the controllers ability to accurately converge the current features to their references, for a variety of desired shapes in the image, while ensuring a smooth tracking response. We also present some proof of concept results demonstrating the effectiveness of this technique for controlling the robot in constrained environments. Markedly, this is the first successful demonstration for automatic grow-to-shape control using visual feedback for variable length continuum manipulators.

This paper presents a visual servoing method for controlling a robot in the configuration space by purely using its natural features. We first created a data collection pipeline that uses camera intrinsics, extrinsics, and forward kinematics to generate 2D projections of a robot's joint locations (keypoints) in image space. Using this pipeline, we are able to collect large sets of real-robot data, which we use to train realtime keypoint detectors. The inferred keypoints from the trained model are used as control features in an adaptive visual servoing scheme that estimates, in runtime, the Jacobian relating the changes of the keypoints and joint velocities. We compared the 2D configuration control performance of this method to the skeleton-based visual servoing method (the only other algorithm for purely vision-based configuration space visual servoing), and demonstrated that the keypoints provide more robust and less noisy features, which result in better transient response. We also demonstrate the first vision-based 3D configuration space control results in the literature, and discuss its limitations. Our data collection pipeline is available at https://github.com/JaniC-WPI/KPDataGenerator.git which can be utilized to collect image datasets and train realtime keypoint detectors for various robots and environments.

In this paper, we present a novel clothoid-based visual servoing method for controlling the shape of a variable length continuum manipulator. Clothoids are curves with linearly changing curvature. They allow us to obtain a smooth representation of a continuum manipulator's shape in a compact form with few parameters. Using this curve model, we generate image features that are used in an adaptive visual servoing method to drive the robot to a desired shape. The adaptive algorithm estimates and updates a local interaction matrix that maps the rate of change in clothoid features to actuator velocities of the continuum manipulator. As such, the method does not require any robot model or even actuator encoder measurements and only uses the visual clothoid features to control the robot shape. A unique advantage of using our clothoid representation is being able to generate reference shape curves without the need for taking images of the robot at the desired shapes. Experiments demonstrate successful shape and end effector pose convergence for a diverse set of references. Our repeatability tests demonstrate that the system performance is consistent. Notably, we also present the first results in the literature for the vision-based shape control of a variable length continuum robot, extending and contracting to achieve the desired shape.

This paper presents a novel visual servoing method that controls a robotic manipulator in the configuration space as opposed to the classical vision-based control methods solely focusing on the end effector pose. We first extract the robot's shape from depth images using a skeletonization algorithm and represent it using parametric curves. We then adopt an adaptive visual servoing scheme that estimates the Jacobian online relating the changes of the curve parameters and the joint velocities. The proposed scheme does not only enable controlling a manipulator in the configuration space, but also demonstrates a better transient response while converging to the goal configuration compared to the classical adaptive visual servoing methods. We present simulations and real robot experiments that demonstrate the capabilities of the proposed method and analyze its performance, robustness, and repeatability compared to the classical algorithms.