Learning complex navigation behaviors in simulation and successfully transferring them to the real world remains a significant challenge due to the sim-to-real gap. To address this issue, we propose a hybrid reconstruction approach that combines Light Detection and Ranging (LiDAR) data and monocular images to create visually and structurally realistic simulated environments. Specifically, we extract an accurate mesh from LiDAR measurements and integrate it with geometry-consistent Three-Dimensional Gaussian Splatting (3DGS) to render photorealistic images. Additionally, we present an end-to-end neural network designed to generate control commands for a mobile robot, and it is used to assess the effectiveness of the hybrid reconstructions in narrowing the sim-to-real gap. In the proposed model, we introduce a novel auxiliary branch and define loss functions to improve the network’s ability to learn safe and robust actions. The method, which is based on imitation learning, is evaluated by deploying it on a real robot to navigate over a total distance of 3.5 kilometers, and experimental results show that models trained purely in simulation achieve performance comparable to those trained on real-world data.

This paper presents a touch-based control method for tracking 3D surfaces in robotic finishing tasks, specifically for the footwear industry. Our method combines a normal force controller and an orientation controller, both based on the forces feedback from a tactile sensor with nine contact points. The controller regulates the contact force and the orientation of a tool mounted on a robotic end-effector by generating velocity commands in Cartesian space, which allows the robot to adapt the tool pose according to the interaction with the surface. The system does not require a predefined position trajectory, instead it generates velocity commands. The results show the controller’s capability to maintain a constant contact force with an error of 0.229 ± 0.169 [N] when testing with a real shoe, and it adapts to unknown surfaces with different curvatures and slopes.

The tracking problem remains difficult because trackers may lose the target due to occlusions or changes in the target's appearance and start tracking another one, making it difficult to detect when they fail. To deal with these problems, we design a novel method to track objects robustly by fusing different trackers, using an Extended Kalman Filter, which helps to detect these situations and correct them. We use our tracker-fusion in a mobile robot to follow a pedestrian while also avoiding dynamic obstacles, an aspect not usually analyzed in pedestrian following. The perception of the environment is done with a 3D LiDAR and the tracking is performed on front-view images constructed from the point cloud. We show that by fusing two different trackers, the robot can follow the target during long experimental sessions where there are occlusions of the target by other people. The method is more precise and robust than using the trackers without the fusion.

Single Object Tracking methods are yet not robust enough because they may lose the target due to occlusions or changes in the target’s appearance, and it is difficult to detect automatically when they fail. To deal with these problems, we design a novel method to improve object tracking by fusing complementary types of trackers, taking advantage of each other’s strengths, with an Extended Kalman Filter to combine them in a probabilistic way. The environment perception is performed with a 3D LiDAR sensor, so we can track the object in the point cloud and also in the front-view image constructed from the point cloud. We use our tracker-fusion method in a mobile robot to follow pedestrians, also considering the dynamic obstacles in the environment to avoid them. We show that our method allows the robot to follow the target accurately during long experimental sessions where the trackers independently fail, demonstrating the robustness of our tracker-fusion strategy.

OpenStreetMaps (OSM) is currently studied as the environment representation for autonomous navigation. It provides advantages such as global consistency, a heavy-less map construction process, and a wide variety of road information publicly available. However, the location of this information is usually not very accurate locally. In this paper, we present a complete autonomous navigation pipeline using OSM information as environment representation for global planning. To avoid the flaw of local low-accuracy, we offer the novel LiDAR-based Naive-Valley-Path (NVP) method that exploits the concept of “valley” areas to infer the local path always furthest from obstacles. This behavior allows navigation always through the center of trafficable areas following the road’s shape independently of OSM error. Furthermore, NVP is a naive method that is highly sample-time-efficient. This time efficiency also enables obstacle avoidance, even for dynamic objects. We demonstrate the system’s robustness in our research platform BLUE, driving autonomously across the University of Alicante Scientific Park for more than 20 km with 0.24 meters of average error against the road’s center with a 19.8 ms of average sample time. Our vehicle avoids static obstacles in the road and even dynamic ones, such as vehicles and pedestrians.

The goal of this paper is to present a robotic workcell to automate several tasks of the cementing process in footwear manufacturing. Our cell's main applications are sole digitization of a wide variety of footwear, glue dispensing and sole grasping from conveyor belts. This cell is made up of a manipulator arm endowed with a gripper, a conveyor belt and a 3D scanner. We have integrated all the elements into a ROS simulation environment facilitating control and communication among them, also providing flexibility to support future extensions. We propose a novel method to grasp soles of different shape, size and material, exploiting the particular characteristics of these objects. Our method relies on object contour extraction using concave hulls. We evaluate it on point clouds of 16 digitized real soles in three different scenarios: concave hull, k-NNs extension and PCA correction. While we have tested this workcell in a simulated environment, the presented system's performance is scheduled to be tested on a real setup at INESCOP facilities in the upcoming months.

Nowadays, robotics education programs are used to promote the skills of students in STEM content at different levels of education for a wide variety of studies, not just engineering. This paper describes an educational program of robotics, applied precisely to the laboratory sessions of a robotics initiation course taught in the first year of a degree in Robotic Engineering, as well as the results of the application of that program in different academic years. The educational program contemplates that the students, grouped in work teams, have to design, assemble, and program a small mobile robot, destined to a final competition in which all the teams participate.

In the past, methods for hand sign recognition have been successfully tested in Human Robot Interaction (HRI) using traditional methodologies based on static image features and machine learning. However, the recognition of gestures in video sequences is a problem still open, because current detection methods achieve low scores when the background is undefined or in unstructured scenarios. Deep learning techniques are being applied to approach a solution for this problem in recent years. In this paper, we present a study in which we analyse the performance of a 3DCNN architecture for hand gesture recognition in an unstructured scenario. The system yields a score of 73% in both accuracy and F1. The aim of the work is the implementation of a system for commanding robots with gestures recorded by video in real scenarios.

This paper presents a description and comparison of two segmentation methods for the oil spill detection in the sea surface. SLAR sensors acquire video sequences from which snapshots are extracted for the detection of oil spills. Both approaches are segmentation based on graph techniques and J-image respectively. Finally, the aim of applying both approaches to SLAR snapshots, as shown, is to detect the largest part of the oil slick and minimize the false detection of the spill.

Nowadays, Robotics is increasing its importance for the marine environment in our society. It allows to perform surveillance and data sampling tasks reducing the current cost of these tasks. This paper presents the design of a prototype of an Autonomous Surface Vessel (ASV) based on catamaran shape. It uses the screw theory for the propulsion system, which provides high manoeuvrability to the vessel. Furthermore, the parts of the ASV are designed to be printed by a RepRap 3D printer. This gives flexibility to check the performance of the vessel. Also the software control scheme of the vessel is presented.

This paper describes a whole new framework to quickly develop dynamic visual servoing systems embedded in an FPGA. Parallel design of the algorithms increases the precision of this kind of controllers, whereas minimizes the response time. Embedding dynamic visual servoing algorithms converts these systems into real-time systems. Additionally, an optimal control framework to dynamically visual control robot arms is proposed. The architecture re-configurability is tested with the implementation of two dynamic visual servoing controllers. One of the implemented controllers is derived from the proposed control framework and the other is the well-known image transpose Jacobian controller. From these experiments, FPGA resources utilization and timeline of the pipeline show the optimization of the proposed framework.

Grasping and manipulating deformable objects with a robot hand has several interesting challenges. Deformable objects, due to its texture and deformability, present different features from rigid ones and that issue can cause uncontrolled movements during grasping and/or manipulation. The paper presents a control strategy for grasping deformable objects, focused on elastic foams, based on tactile information. An adaptation at different elastic properties of the object is achieved, because the deformability degree is estimated during the grasping process. Several experiments are shown in order to demonstrate the reliability of the tactile-based strategy.

This paper presents a method to analyse 3D planar surfaces and to measure variations on it. The method is oriented to the detection of deformations on the elastic object surfaces formed by flat faces. These deformations are usually caused when two bodies, a solid and another elastic object, come in contact and there are contact pressures among their faces. Our method describes a strategy to model the shape of deformation using a mathematical approach based on two concepts: Histogram and Map of curvature. In particular, we describe the algorithm for deformations in order to use it in visual control and inspection tasks for manipulation processes with robot hands. Several experiments and their results are shown to evaluate the validity and robustness of the method to detect and measure deformations in grasping tasks. To do it, some virtual scenarios were created to simulate contacts with fingers of a hand robot.

This paper describes a study and analysis of surface normal-base descriptors for 3D object recognition. Specifically, we evaluate the behaviour of descriptors in the recognition process using virtual models of objects created from CAD software. Later, we test them in real scenes using synthetic objects created with a 3D printer from the virtual models. In both cases, the same virtual models are used on the matching process to find similarity. The difference between both experiments is in the type of views used in the tests. Our analysis evaluates three subjects: the effectiveness of 3D descriptors depending on the viewpoint of camera, the geometry complexity of the model and the runtime used to do the recognition process and the success rate to recognize a view of object among the models saved in the database.

This paper presents a novel control approach to dynamic visual servo control multi-fingered robotic hands for the manipulation performance of an object. The control approach allows the path tracking of the object motion by means of visual information and takes into account both the dynamics model of the robot hand and the grasping force of the fingertips. In addition, an optimal approach is employed to obtain the desired behaviour in the joint space based on an indicated cost function which determines how the control effort is distributed over the robot fingers' joints. Finally, authors show experimental verifications on a robotic manipulation system for some of the visual servo controllers derived from the control approach.

Image Based Visual Servoing (IBVS) is a robotic control scheme based on vision. This scheme uses only the visual information obtained from a camera to guide a robot from any robot pose to a desired one. However, IBVS requires the estimation of different parameters that cannot be obtained directly from the image. These parameters range from the intrinsic camera parameters (which can be obtained from a previous camera calibration), to the measured distance on the optical axis between the camera and visual features, it is the depth. This paper presents a comparative study of the performance of D-IBVS estimating the depth from three different ways using a low cost RGB-D sensor like Kinect. The visual servoing system has been developed over ROS (Robot Operating System), which is a meta-operating system for robots. The experiments prove that the computation of the depth value for each visual feature improves the system performance.

Traditional visual servoing systems have been widely studied in the last years. These systems control the position of the camera attached to the robot end-effector guiding it from any position to the desired one. These controllers can be improved by using the event-based control paradigm. The system proposed in this paper is based on the idea of activating the visual controller only when something significant has occurred in the system (e.g. when any visual feature can be loosen because it is going outside the frame). Different event triggers have been defined in the image space in order to activate or deactivate the visual controller. The tests implemented to validate the proposal have proved that this new scheme avoids visual features to go out of the image whereas the system complexity is reduced considerably. Events can be used in the future to change different parameters of the visual servoing systems.

This paper presents a novel planner for dexterous manipulation with a multi-fingered robotic hand. This planner receives as input an initial grasp of the object and a desired trajectory of the object in task space (position and orientation). The planner computes the movements of the fingers which are required to move the object along this trajectory without breaking contacts. It uses a triangle mesh representation of the surfaces of the fingers and the object and a new contact evolution graph in order to compute all the possible transitions between the contact primitives of their surfaces. This planner have been implemented as a program which communicates with a five-fingered hand in order to test them in real manipulation tasks.

This paper develops the kinematic, dynamic and contact models of a three-fingered robotic hand (BarrettHand) in order to obtain a complete description of the system which is required for manipulation tasks. These models do not only take into account the mechanical coupling and the breakaway mechanism of the under-actuated robotic hand but they also obtain the force transmission from the hand to objects, which are represented as triangle meshes. The developed models have been implemented on a software simulator based on the Easy Java Simulations platform. Several experiments have been performed in order to verify the accuracy of the proposed models with regard to the real physic system.

This paper describes the modelling, simulation and visual servo control of a robot-arm designed for supervision of complex task environments. This robot (mini-robot) has 2 degrees of freedom (hereafter d.o.f.) and it is attached at the end of another anthropomorphic robot with 7 d.o.f. (main robot). In order to achieve a suitable supervision, the mini-robot has to perform a given trajectory around the workspace where the task is developed. With the purpose of obtaining an accurate and fast supervision, this paper presents a novel tracking method carried out in the image space using direct visual servo control.

The depth perception in the objects of a scene can be useful for tracking or applying visual servoing in mobile systems. 3D time-of-flight (ToF) cameras provide range images which give measurements in real time to improve these types of tasks. However, the distance computed from these range images is very changing with the integration time parameter. This paper presents an analysis for the online adaptation of integration time of ToF cameras. This online adaptation is necessary in order to capture the images in the best condition irrespective of the changes of distance (between camera and objects) caused by its movement when the camera is mounted on a robotic arm.

The precise localization of human operators in robotic workplaces is an important requirement to be satisfied in order to develop human-robot interaction tasks. Human tracking provides not only safety for human operators, but also context information for intelligent human-robot collaboration. This paper evaluates an inertial motion capture system which registers full-body movements of an user in a robotic manipulator workplace. However, the presence of errors in the global translational measurements returned by this system has led to the need of using another localization system, based on Ultra-WideBand (UWB) technology. A Kalman filter fusion algorithm which combines the measurements of these systems is developed. This algorithm unifies the advantages of both technologies: high data rates from the motion capture system and global translational precision from the UWB localization system. The developed hybrid system not only tracks the movements of all limbs of the user as previous motion capture systems, but is also able to position precisely the user in the environment.

In this paper, a new method to track image trajectories by visual servoing is proposed. This method solves the problem of the previous proposed time-independent tracking systems based on visual servoing. With the proposed method, the robot can track a previously generated trajectory affording a correct tracking not only in the image but also in the 3D space. This new method presents several improvements over the previous ones such as the possibility of specifying the desired tracking velocity, a less oscillating behavior or a correct tracking in the 3D space when high velocities are used. In order to demonstrate the correct behavior of the visual servoing system, an eye-in-hand camera system is used.

Calibration techniques allow the estimation of the intrinsic parameters of a camera. This paper describes an adaptive visual servoing scheme which employs the visual data measured during the task to determine the camera intrinsic parameters. This approach is based on the virtual visual servoing approach. However, in order to increase the robustness of the calibration several aspects have been introduced in this approach with respect to the previous developed virtual visual servoing systems. Furthermore, the system is able to determine the value of the intrinsic parameters when they vary during the task. This approach has been tested using an eye-in-hand robotic system.

This paper describes a method to detect changes in given surfaces tracked by the robot end-effector. To do so, first an approach to combine visual and force information is described. By filtering the interaction forces, a parameter which provides the probability of a change in the tracking surface is obtained. In order to obtain a more accurate determination of the change in the surface, information from a laser located at the robot end-effector is also employed. Therefore, a robust method for detecting surface discontinuities which employs force and visual information jointly with structured light is presented

This paper proposes a new method for the detection and elimination of specular reflectance in color images of real scenes. We use a two-dimensional histogram that allows us to relate the signals of intensity and saturation of a color image, and to identify the specularities in an area of the histogram. This is known as the Intensity-Saturation (MS) diagram, and it is constructed from the Intensity-Saturation-Hue (MSH) generalized color space. An experimental and detailed study of the presence of specularities in the MS diagram for different types of materials in real scenes is carried out. To eliminate the specularities detected, we use a new connected vectorial filter based on the extension of mathematical morphology to color images, employing a lexicographical order. This new filter operates only in the bright areas previously detected, avoiding the high cost of processing the connected filters and the related oversimplification. The proposed method achieves results similar to current methods, but without the need for costly multiple-view systems or stereo images

In this paper, the so-called 2-D movement flow-based visual servoing system is proposed, which allows the tracking of trajectories planned in the image space. This method is time-independent, which is useful when problems such as obstructions occur during the tracking. This method allows the tracking of image trajectories assuring the correct behaviour in the 3-D space and avoiding the limitations of the time-dependent systems used up to now for tracking trajectories. This aspect has allowed its application to manipulation tasks in which the trajectory can be obstructed during the tracking.

A new approach for fusing visual and force information is shown. First, a new method for tracking trajectories, called movement flow-based visual servoing system, which presents the correct behavior in the image and in the three-dimensional space, is described. The information obtained from this system is fused with that obtained from a force control system in unstructured environments. To do so, a new method of recognizing the contact surface and a system for fusing visual and force information are described. The latter method employs variable weights for each sensor system, depending on a criteria based on the detection of changes in the interaction forces processed by a Kalman filter.

This paper shows the state of art teleoperation and simulation systems and proposes some possible network architectures devoted to the development of such systems. A method for the optimization of robot tasks is also suggested. The application proposed can teleoperate a robot arm with five degrees of freedom through Internet by using a previous simulation system and visual feedback. This simulation can be accessed simultaneously by different workers. As a result, the number of robots required can be reduced on new applications or dangerous environments.

Automatic quality control examination of banknotes constitutes an important research challenge for the main printing organizations in the world. It is worth to note that presently there is no commercial product with an acceptable performance, in spite of the efforts spent during last ten years. The product complexity lies in the combination of several printings with the purpose of achieving hard to counterfeit banknotes. The printing system that is necessary for banknotes manufacturing, causes significant differences between several printed sheets in similar conditions. However, in spite of these differences, while respecting certain tolerances, the banknotes are accepted. The required quality of the product demands an exhaustive inspection of the bill sheets. Nowadays the quality control is carried out by manual turn over of the bill sheet, reaching a speed of 400 sheets/hour. It has been developed a system that works out an automatic inspection at a rate of 3600 sheets/hour.