This paper addresses the robotic manipulation of deformable linear objects (DLOs) in 3D space, which is a complex problem with relevant applications in, e.g., industrial, agricultural, or medical domains. We propose a simple geometric model of elastic deformation tailored to this problem, and exploit this model to derive a quasi-static deformation Jacobian mapping robot motions to changes in the DLO’s shape. We then propose a control law, based on this Jacobian, to drive multiple points on the DLO toward target positions. The described approach extends prior work restricted to 2D scenarios, and is capable of controlling the DLO’s shape in 3D and with six-degrees-of-freedom gripper motions. The main advantage of this approach is its simplicity, as it does not require training, simulation, or full perception of the object’s shape. We present validation results from both simulations and real-world experiments.

Shape control involves deforming objects to achieve a desired shape. One of the main challenges of this task is to define a suitable control reference, especially when 3D objects that lack distinctive visual texture and geometric features are involved. This paper addresses the problem of generating a suitable shape control reference using surface maps for 3D texture-less objects. The proposed surface mapping method is based on functional maps and ensures time consistency with robustness to non-isometries. Our time consistent method is validated within a shape control strategy, where local exponential stability analysis is provided. The effectiveness of the framework is illustrated through simulations and experiments. Note to Practitioners—We present a method for shape comparison to analyse the geometric similarities between the shape of an object and a desired target shape. The goal is to generate a point map between both surfaces so that, with the use of robots that grasp the object, a control strategy can deform the object towards the desired shape. The point map we compute, while adapting to the changing shape of the deforming object, remains stable enough to enable the shape control task. This can be of use for the automation of processes that involve shape control (e.g., object packaging, moulding, etc.). Our proposed shape control framework combines computer graphics, computer vision, and automation techniques to create a system that is well-suited for industrial setups equipped with commonly used range sensors like RGB-D cameras. Regarding the experiments presented in this paper, controlled lighting conditions are recommendable in order to perform the colour-based object segmentation (e.g. avoidance of abrupt light variations, uniform illumination). Automating the grasping process is not within the scope of this paper, therefore, in each experiment we manually defined the object grasping points according to the shape control task involved. The proposed framework has the potential to increase productivity, reduce costs and, improve safety in hazardous tasks by automating the manipulation of 3D objects that lack distinctive visual texture and geometric features.

A well–known drawback in distributed average consensus of multi–agent systems is that the exchanged information is usually delayed due to the time elapsed during the data transmission process. Using classical dynamic average consensus, delays may lead to poor performance or even instability. In this paper, we propose a novel dynamic consensus method that counteracts the negative effects of delays by means of delay compensation techniques. The interest of our dynamic consensus method with delay compensation is that it converges under mild conditions on graph connectivity and bounded reference signals, no matter how large the delays are, as long as delays are fixed and known. We also provide a formal characterization of the convergence speed of our method. Additionally, our results apply to fixed directed strongly connected, and undirected topologies.

Shape control involves bringing a deformable object to a desired shape. In the shape control literature, the positioning of the grippers on the object is usually predefined (user-defined) and therefore considered as input information. In this paper we address the gripper positioning problem for shape control. We propose a deformation process within a simulated fully-actuated scenario and introduce multi-scale centroid paths as geometry describing points for which we prove individual control feasibility. Analysis on the evolution of multi-scale centroid paths through the fully-actuated deformation process allows us to define an importance metric for gripper candidates. Final gripper positions, based on the importance metric, are obtained through optimisation. We present simulation results for global and local shape control problems.

In this paper, we propose a formation control system for deforming and transporting simultaneously a de-formable object with a team of robots, modeled with double-integrator dynamics. The goal is to reach a target configuration, defined as a combination of shape, scale, orientation and position of the formation. We augment this controller with a set of control barrier functions (CBFs). The CBFs allow us to satisfy fundamental constraints for the success of the task: avoidance of agent-to-agent, agent-to-obstacle and object-to-obstacle collisions, and of excessive stretching. We test the performance of our proposal in different simulation scenarios.

We propose a new approach to control the shape of deformable objects with robots. Specifically, we consider a fixed-length elastic linear object lying on a 2D workspace. Our main idea is to encode the object's deformation behavior in an offline constant Jacobian matrix. To derive this Jacobian, we use geometric deformation modeling and combine recent work from the fields of deformable object control and multirobot systems. Based on this Jacobian, we then propose a robotic control law that is capable of driving a set of shape features on the object toward prescribed values. Our contribution relative to existing approaches is that at run-time we do not need to measure the full shape of the object or to estimate/simulate a deformation model. This simplification is achieved thanks to having abstracted the deformation behavior as an offline model. We illustrate the proposed approach in simulation and in experiments with real deformable linear objects.

Shape control has become a prominent research field as it enables the automation of tasks in many applications. Overall, deforming an object to a desired target shape by using few grippers is a major challenge. The limited information about the object dynamics, the need to combine small and large deformations in order to achieve certain target shapes and the non-linear nature of most deformable objects are factors that significantly hamper shape control performance. In this paper, we propose a shape control method for multi-robot manipulation of large-strain deformable objects. Our approach is based on multi-scale Laplacian descriptors that feed an FMM (Fast Marching Method) for elastic shape contour matching. The FMM's resulting path and the Laplacian operator are used to define a control strategy for the robot grippers. Simulation experiments carried out with an ARAP (As Rigid As Possible) deformation model provide satisfactory results.

Addressing the problem of enclosing and tracking a target requires multiple agents with adequate motion strategies. We consider a team of unicycle robots with a standard camera on board. The robots must maintain the desired enclosing formation while dealing with their nonholonomic motion constraints. The reference formation trajectories must also guarantee permanent visibility of the target by overcoming the limited field of view of the cameras. In this article, we present a novel approach to characterize the conditions on the robots' trajectories taking into account the motion and visual constraints. We also propose online and offline motion planning strategies to address the constraints involved in the task of enclosing and tracking the target. These strategies are based on maintaining the formation shape with variable size or, alternatively, on maintaining the size of the formation with flexible shape.

Autonomous manipulation of deformable objects is a research topic of increasing interest due to the variety of current processes and applications that include this type of tasks. It is a complex problem that involves aspects such as modeling, control, perception, planning, grasping, estimation, etc. A single robot may be unable to perform the manipulation when the deformable object is too big, too heavy or difficult to grasp. Then, using multiple robots working together naturally arises as a solution to perform coordinately the manipulation task. In this paper, we contribute a survey of relevant state-of-the-art approaches concerning manipulation of deformable objects by multiple robots, which includes a specific classification with different criteria and a subsequent analysis of the leading methods, the main challenges and the future research directions.

Building the 3D model of an object is a complex problem that involves aspects such as modeling, control, perception or planning. Performing this task requires a set of different views to cover the entire surface of the object. Since a single camera takes too long to travel through all these positions, we consider a multi-camera scenario. Due to the camera constraints such as the limited field of view or self occlusions, it is essential to use an effective configuration strategy to select the appropriate views that provide more information of the model. In this paper, we develop a multi-camera architecture built on the Robot Operating System. The advantages of the proposed architecture are illustrated with a formation-based algorithm to compute the view that satisfies these constraints for each robot of the formation to obtain the volumetric reconstruction of the target object.

Line-images in non-central cameras contain much richer information of the original 3D line than line projections in central cameras. The projection surface of a 3D line in most catadioptric non-central cameras is a ruled surface, encapsulating the complete information of the 3D line. The resulting line-image is a curve which contains the 4 degrees of freedom of the 3D line. That means a qualitative advantage with respect to the central case, although extracting this curve is quite difficult. In this paper, we focus on the analytical description of the line-images in non-central catadioptric systems with symmetry of revolution. As a direct application we present a method for automatic line-image extraction for conical and spherical calibrated catadioptric cameras. For designing this method we have analytically solved the metric distance from point to line-image for non-central catadioptric systems. We also propose a distance we call effective baseline measuring the quality of the reconstruction of a 3D line from the minimum number of rays. This measure is used to evaluate the different random attempts of a robust scheme allowing to reduce the number of trials in the process. The proposal is tested and evaluated in simulations and with both synthetic and real images.

An emerging application of multirobot systems is the monitoring of a dynamic event. Here, the goal is to enclose and track a moving target by attaining a desired geometric formation around it. By considering a circular pattern configuration for the target enclosing, the multirobot system is able to perform full perception of the target along its motion. In the proposed system, the robots rely only on their onboard vision sensor without external input to complete the task. The key problem resides in overcoming the motion and visual constraints of the agents. In particular, differential-drive robots with limited sensing, that must maintain visibility of the moving target as it navigates in the environment, are considered. A novel approach to characterize the motion of the robots in the formation that allows to enclose and track the target while overcoming their limited field of view (FOV) is presented. The proposed approach is illustrated through simulations.

We introduce a novel hybrid camera configuration composed by a fisheye camera attached to an RGB-D system. Current RGB-D sensors provide the 3D information and scale of the scene, but they are limited by a small field of view. In contrast, wide field of view cameras capture a larger portion of the scene, but providing highly distorted images that require specific algorithms. By coupling a fisheye camera to an RGB-D system we take advantage of both types of cameras overcoming their drawbacks. The system provides a portion of the fisheye image with depth data and we use this seed information to perform scaled operations in the complete image. We also present a calibration procedure of the system to map depth information to the wide angle image. With this purpose, we propose a depth-fisheye calibration algorithm nurturing from state of the art camera models and methods. Several experiments test the accuracy of the system with real images.

This paper shows how to push an unknown object in the plane from an initial pose to a target pose with two cooperating mobile robots. On the object motion, we deliberately impose non-holonomic velocity constraint with pushing mobile robots. This yields smooth and efficient trajectories. Pushing manipulation is performed, for the first time, with a new uncalibrated image-based control scheme. This is achieved by transforming the image information to a scaled Euclidean space without using any metric information or calibration. Stability of the control law is also demonstrated.

This paper describes a new vision-based control method to drive a set of robots moving on the ground plane to a desired formation. As the main contribution, we propose to use multiple camera-equipped unmanned aerial vehicles (UAVs) as control units. Each camera views, and is used to control, a subset of the ground team. Thus, the method is partially distributed, combining the simplicity of centralized schemes with the scalability and robustness of distributed strategies. Relying on a homography computed for each UAV-mounted camera, our approach is purely image-based and has low computational cost. In the control strategy we propose, if a robot is seen by multiple cameras, it computes its motion by combining the commands it receives. Then, if the intersections between the sets of robots viewed by the different cameras satisfy certain conditions, we formally guarantee the stabilization of the formation, considering unicycle robots. We also propose a distributed algorithm to control the camera motions that preserves these required overlaps, using communications. The effectiveness of the presented control scheme is illustrated via simulations and experiments with real robots.

The projection surface of a 3D line in a non-central camera is a ruled surface, containing the complete information of the 3D line. The resulting line-image is a curve which contains the 4 degrees of freedom of the 3D line. In this paper we investigate the properties of the line-image in conical catadioptric systems. This curve is a particular quartic that can be described by only six homogeneous parameters. We present the relation between the line-image description and the geometry of the mirror. This result reveals the coupling between the depth of the line and the distance from the camera to the mirror. If this distance is unknown the 3D information of a projected line can be recovered up to scale. Knowing this distance allows obtaining the 3D metric reconstruction. The proposed parametrization also allows to simultaneously reconstruct the 3D line and computing the aperture angle of the mirror from five projected points on the line-image. We analytically solve the metric distance from a point to a line-image and we evaluate the proposal with real images.

This paper presents a novel method that enables a team of aerial robots to enclose a target in 3D space by attaining a desired geometric formation around it. We propose an approach in which each robot obtains its motion commands using measurements of the relative position of the other agents and of the target, without the need for a central coordinator. As contribution, our method permits any desired 3D target enclosing configuration to be defined, in contrast with the planar circular patterns commonly encountered in the literature. The proposed control strategy relies on the minimization of a cost function that captures the collective motion objective. In our method, the robots do not need to use a common reference frame. This coordinate independence is achieved through the introduction in the cost function of a rotation matrix computed locally by each robot. We prove that our motion controller is exponentially stable, and illustrate its performance through simulations.

In omnidirectional cameras, straight lines in the scene are projected onto curves called line-images.The shape of these curves is strongly dependent of the particular camera configuration.The great diversity of omnidirectional camera systems makes harder the line-image extraction in a general way.Therefore, it is difficult to design uncalibrated general approaches, and existing methods to extract lines in omnidirectional images require the camera calibration.In this paper, we present a novel method to extract lineimages in uncalibrated images which is valid for radially symmetric central systems.In our proposal, the distortion function is analytically solved for different types of camera systems, dioptric or catadioptric.We present the unified line-image constraints to extract the projection plane of each line and main calibration parameter of the camera from a single line-image.The use of gradient-based information allows computing both from a minimum of two image points.This scheme is used in a line-image extraction algorithm to obtain lines from uncalibrated omnidirectional images without any assumption about the scene.The algorithm is evaluated with synthetic and real images showing good performance.

We present a method for visual control of a set of robots moving on the ground plane. The goal of the control task is for the team to reach a desired geometric configuration. Each robot carries an omnidirectional camera and can communicate with a number of the other robots. The approach relies on the computation of the planar motion between two views, by means of 1D homographies. This knowledge, obtained by each robot from its own images and the visual information received from neighboring robots, allows it to define a desired position on the plane. Then, we propose a novel control scheme based on computing a particular 2D transformation to drive each robot towards its goal position. A contribution of this work is the use of 1D homography in a multirobot control framework. This tool allows to deal with purely angular visual information, which is precise and requires no calibration. The approach we present is completely distributed. Each robot uses only information from its formation neighbors and the global centroid to obtain its motion commands. These individual behaviors naturally result in the complete team of robots reaching the desired global configuration.

This paper addresses the estimation of planar camera motion using 1D homographies. As contributions, we show analytically that, contrary to what occurs with the 2D homography, there is a family of infinite solutions to the 1D homography decomposition, and therefore infinite possible motion reconstructions. In addition, we propose a new method to compute the planar motion between two images from the information provided by two different 1D homographies, employing their associated homology transformations. Therefore, our approach computes a general planar camera motion from only two 1D views, when previous works needed three 1D views for this task. The use of 1D information makes the method particularly suitable for omnidirectional cameras, due to the wide field of view and precise angular information provided by this kind of sensors. The performance of our proposal is illustrated through simulations and experiments on real images.

This paper is focused on dynamic coverage control with a team of robots. In this framework, decentralized control algorithms have been investigated to deal with the efficient coordination of the resources. The main contribution is a novel global strategy based on a hierarchical grid decomposition of the domain. This decomposition allows an ordered coverage of the domain that combined with a gradient based control law of the local error, achieves a better performance than previous approaches of dynamic coverage. The total coverage of the domain is proven, and the good performance of the approach is supported with simulations.

This paper addresses the problem of visual control of a set of mobile robots. In our framework, the perception system consists of an uncalibrated flying camera performing an unknown general motion. The robots are assumed to undergo planar motion considering nonholonomic constraints. The goal of the control task is to drive the multirobot system to a desired rendezvous configuration relying solely on visual information given by the flying camera. The desired multirobot configuration is defined with an image of the set of robots in that configuration without any additional information. We propose a homography-based framework relying on the homography induced by the multirobot system that gives a desired homography to be used to define the reference target, and a new image-based control law that drives the robots to the desired configuration by imposing a rigidity constraint. This paper extends our previous work, and the main contributions are that the motion constraints on the flying camera are removed, the control law is improved by reducing the number of required steps, the stability of the new control law is proved, and real experiments are provided to validate the proposal.

This paper addresses the problem of visual control of a set of mobile robots. In our framework, the perception system consists of a calibrated flying camera looking downward to the mobile robots. The robots are assumed to undergo planar motion considering nonholonomic constraints. The goal of the task is to control the multi-robot system to a desired configuration relying solely on visual information given by the flying camera. The desired multi-robot configuration is defined with an image of the set of robots in that configuration. Then, any arbitrary configuration can be easily defined by this image without any additional information. As contribution, a new image-based control scheme is presented relying on the homography induced by the multi-robot system to lead the robots to the desired configuration. The stability of the control law is analyzed and simulations are provided to illustrate the proposal.

Driving mobile robots to precise locations is of recognized interest, and using vision sensors in this context supplies many advantages. We propose a novel control law based on sliding-mode theory in order to drive mobile robots to a target location, which is specified by a previously acquired reference image. The control scheme exploits the piecewise epipolar geometry of three views on the basis of image-based visual servoing, in such a way that no 3-D scene information is required. The contribution of the paper is a new control law that achieves convergence to the target with no auxiliary images and without changing to any approach other than epipolar-based control. Additionally, the use of sliding-mode control deals with singularities, thus allowing the robot to move directly toward the target as well as avoiding the need of a precise camera calibration. The effectiveness of our approach is tested with simulations and real-world experiments.

Catadioptric systems consist of the combination of lenses and mirrors. From them, central panoramic systems stand out because they provide a unique effective viewpoint, leading to the well-known unifying theory for central catadioptric systems. This paper considers catadioptric systems consisting of a conic mirror and a projective camera. Although a system with conic mirror does not possess a single projection point, it has some advantages as the cone is a very simple shape to produce, it has higher resolution in the peripheral, and adds less optical distortion to the images. The contributions of this work are the model of this non-central system by means of projective mappings from a torus to a plane, and the definition of the conic fundamental matrix with a role similar to that of perspective cameras. Additionally, a procedure to compute the relative motion between two views from the conic fundamental matrix is presented.

This paper presents a new method for visual homing to be used on a robot moving on the ground plane. A relevant issue in vision-based navigation is the field-of-view constraints of conventional cameras. We overcome this problem by means of omnidirectional vision and we propose a vision-based homing control scheme that relies on the 1D trifocal tensor. The technique employs a reference set of images of the environment previously acquired at different locations and the images taken by the robot during its motion. In order to take advantage of the qualities of omnidirectional vision, we define a purely angle-based approach, without requiring any distance information. This approach, taking the planar motion constraint into account, motivates the use of the 1D trifocal tensor. In particular, the additional geometric constraints enforced by the tensor improve the robustness of the method in the presence of mismatches. The interest of our proposal is that the designed control scheme computes the robot velocities only from angular information, being this very precise information; in addition, we present a procedure that computes the angular relations between all the views even if they are not directly related by feature matches. The feasibility of the proposed approach is supported by the stability analysis and the results from simulations and experiments with real images.

In this paper, we present a visual servo controller that effects optimal paths for a nonholonomic differential drive robot with field-of-view constraints imposed by the vision system. The control scheme relies on the computation of homographies between current and goal images, but unlike previous homography-based methods, it does not use the homography to compute estimates of pose parameters. Instead, the control laws are directly expressed in terms of individual entries in the homography matrix. In particular, we develop individual control laws for the three path classes that define the language of optimal paths: rotations, straight-line segments, and logarithmic spirals. These control laws, as well as the switching conditions that define how to sequence path segments, are defined in terms of the entries of homography matrices. The selection of the corresponding control law requires the homography decomposition before starting the navigation. We provide a controllability and stability analysis for our system and give experimental results.

This paper addresses the problem of visual control of a mobile robot. The system consists of a calibrated camera fixed onboard a robot with nonholonomic motion constraints. The parking task is defined by a reference image taken at the target location. The proposed control law is based on the essential matrix, but unlike traditional methods, it is not used to compute pose parameters. Instead, the control law is defined directly in terms of individual entries of the essential matrix by means of the input-output linearization of the system. Here we solve the problem of degeneracies due to short baseline by taking advantage of the planar motion constraint of the robot. Thus, a virtual target is defined providing a stable estimation of the essential matrix without degeneracies despite short baseline.

This paper addresses the problem of vision-based closed-loop control for undulatory robots. We present an image-based visual servoing scheme, which drives the robot to a desired location specified by a target image, without explicitly estimating its pose. Instead, the control relies on the computation of the epipolar geometry between the current and target images. We analyze controllability and stability of the proposed control scheme, which is validated by simulation studies using the SIMUUN computational tools. Preliminary experiments, involving the Nereisbot undulatory robotic prototype, are also presented.

This paper presents a switched homography-based visual control for differential drive vehicles. The goal is defined by an image taken at the desired position, which is the only previous information needed from the scene. The control takes into account the field-of-view constraints of the vision system through the specific design of the paths with optimality criteria. The optimal paths consist of straight lines and curves that saturate the sensor viewing angle. We present the controls that move the robot along these paths based on the convergence of the elements of the homography matrix. Our contribution is the design of the switched homography-based control, following optimal paths guaranteeing the visibility of the target.

This paper presents a new visual control approach based on homography. The method is intended for nonholonomic vehicles with a fixed monocular system on board. The idea of visual control used here is the usual approach where the desired position of the robot is given by a target image taken at that position. This target image is the only previous information needed by the control law to perform the navigation from the initial position to the target. The control law is designed by the input-output linearization of the system using elements of the homography as output. The contribution is a controller that deals with the nonholonomic constraints of the mobile platform needing neither decomposition of the homography nor depth estimation to the target.

A significant amount of work has been reported in the area of visual servoing during the last decade. However, most of the contributions are applied in cases of holonomic robots. More recently, the use of visual feedback for control of nonholonomic vehicles has been reported. Some of the examples are docking and parallel parking maneuvers of cars or vision-based stabilization of a mobile manipulator to a desired pose with respect to a target of interest. Still, many of the approaches are mostly interested in the control part of visual servoing loop considering very simple vision algorithms based on artificial markers. In this paper, we present an approach for nonholonomic visual servoing based on epipolar geometry. The method facilitates a classical teach-by-showing approach where a reference image is used to define the desired pose (position and orientation) of the robot. The major contribution of the paper is the design of the control law that considers nonholonomic constraints of the robot as well as the robust feature detection and matching process based on scale and rotation invariant image features. An extensive experimental evaluation has been performed in a realistic indoor setting and the results are summarized in the paper