Mastering rough terrain locomotion is a tough challenge for robots due to its dynamic, unpredictable nature and frequent physical contact. Traditionally, robots rely on carefully planned foot placements to maintain grip and stability. Recent advancements in quadruped robot feet offer diverse shapes and high grip for various terrains. However, control systems and planners often struggle to leverage these varied capabilities, relying instead on simplified foot models e.g., ball-like, flat. The simplified feet models committed to the single shape of the foot can not be used on robots equipped with diverse feet or modern adaptive feet. This work proposes a novel foothold optimization method that efficiently searches for optimal contact points for different foot shapes using a polynomial approximation. The system leverages a Convolutional Neural Network (CNN) trained on simulated data to predict a cost for each candidate foothold. We show that a single neural network can work with different and new foot mechanical designs without retraining the system. We experimentally validate our system on the ANYmal robot using both ball feet and adaptive soft feet, in indoor and outdoor environments, finding that our system improves stability, in terms of pitch and roll angles of the base, with respect to a state-of-the-art method.

In this paper, we deal with the problem of full-body path planning for walking robots. The state of walking robots is defined in multi-dimensional space. Path planning requires defining the path of the feet and the robot's body. Moreover, the planner should check multiple constraints like static stability, self-collisions, collisions with the terrain, and the legs' workspace. As a result, checking the feasibility of the potential path is time-consuming and influences the performance of a planning method. In this paper, we verify the feasibility of sampling-based planners in the path planning task of walking robots. We identify the strengths and weaknesses of the existing planners. Finally, we propose a new planning method that improves the performance of path planning of legged robots.

In this paper, we focus on the detection of tiny objects. The goal is to improve the performance of tiny object detection while preserving the average precision and recall metrics compared to a brute-force, sliding-window approach. We extend the object categories in COCO detection metrics from small, medium, and large by defining tiny, very tiny, and micro-objects. We propose an evaluation protocol for all six object sizes. To detect tiny objects, we offer a novel ROI proposal method based on a two-level nested U-structure architecture U2-Net. For this purpose, we experiment with multiple dilation techniques as well as Region of Interest (ROI) aggregation methods. We evaluate our method using the Mapillary Traffic Sign Dataset. The obtained detection strategy outperforms the single-step prediction approach and is comparable to the quality obtained with the use of the sliding-window, being nearly 7 times faster.

In this paper, we investigate the problem of state estimation of rotational articulated objects during robotic interaction. We estimate the position of a joint axis and the current rotation of an object from a pair of RGB-D images registered by the depth camera mounted on the robot. However, the camera mounted on the robot has a limited view due to occlusions of the robot's arm. Moreover, some configurations of objects are difficult to register by typical RGB-D sensors. Thus, the model-based methods fail in these cases. To deal with this problem, we propose a CNN-based architecture that gradually estimates the parameters and the state of the rotational joint. To meet real-time requirements on the real robot, we propose a fast inference on 2D images without directly operating on the 3D model of the object. The proposed method is trained and verified on the RBO dataset that contains RGB-D sequences of manipulated articulated objects.

In this research, we deal with the problem of estimating 3D object positions based on 2D localization data and 2D object bounding boxes determined on the RGB images by a CNN object detector. We use a mobile-manipulating robot equipped with an RGB-D camera and a 2D laser scanner. In contrast to other methods, which are based either on end-to-end neural networks or machine learning solutions, we propose an approach that allows estimating 3D object positions from a sequence of images and 2D robot poses obtained from an onboard 2D localization system. We determine a set of lines from the robot localization data and the object detections, which define the observation directions. The closest point to a set of lines determines an approximate object location. We define the 3D object position estimation as an optimization problem and solve using efficient GPU implementation. Finally, we present results based on data collected on the real robot in an unstructured environment.

Robots have limited perception capabilities when observing a new scene. When the objects on the scene are registered from a single perspective, only partial information about the shape of the objects is registered. Incomplete models of objects influence the performance of grasping methods. In this case, the robot should scan the scene from other perspectives to collect information about the objects or use methods that fill in unknown regions of the scene. The CNN-based method for objects reconstruction from a single view utilize 3D structures like point clouds or 3D grids. In this research, we revisit the problem of scene reconstruction and show that scene reconstruction can be formulated in the 2D image space. We propose a new representation of the scene reconstruction problem for a robot equipped with an RGB-D camera. Then, we present a method that generates a depth image of the object from the pose of the camera that is on the other side of the scene. We show how to train a neural network to obtain accurate depth images of the objects and reconstruct a 3D model of the scene observed from a single viewpoint. Moreover, we show that the obtained model can be applied to improve the success rate of the grasping method.

In this paper, we propose a system which detects and estimates the kinematic structures of objects in the indoor environment. We are interested in specific types of objects like doors, sliding doors, and drawers which are common in the human environment and very important taking into account the full autonomy of mobile robots. We assume that the mobile robot is equipped with an RGB-D camera. We utilize a Convolutional Neural Network-based (CNN-based) object detector to locate the articulated objects on the input image created from a pair of RGB-D images. Taking into account strong prior knowledge about the articulated object, we detect the segments on the image which belong to the articulated object. Then, the optimization-based procedure finds the 3D pose and configuration of the joint detected on the scene. We train and verify the method on the images from the Kinect sensor. The performance of the proposed method shows that we can estimate articulated objects in the indoor environment using typical sensors available on the mobile robot.

In this article, we propose a new keyframe-based mapping system. The proposed method updates local Normal Distribution Transform maps (NDT) using data from an RGB-D sensor. The cells of the NDT are stored in 2D view-dependent structures to better utilize the properties and uncertainty model of RGB-D cameras. This method naturally represents an object closer to the camera origin with higher precision. The local maps are stored in the pose graph which allows correcting global map after loop closure detection. We also propose a procedure that allows merging and filtering local maps to obtain a global map of the environment. Finally, we compare our method with Octomap and NDT-OM and provide example applications of the proposed mapping method.

In this paper, we consider a problem of foothold selection for the quadrupedal robots equipped with compliant adaptive feet. Starting from a model of the foot we compute the quality of the potential footholds considering also kinematic constraints and collisions during evaluation. Since terrain assessment and constraints checking are computationally expensive we applied a Convolutional Neural Network (CNN) to evaluate the potential footholds on the elevation map. We propose an efficient strategy for data clustering and segmentation with CNN. The data for training the neural network is collected off-line but the inference works on-line when the robot walks on rough terrains and allows for efficient adaptation to the terrain and exploitation of the properties of the soft adaptive feet.

In this research, we focus on the 6D pose estimation of known objects from the RGB image. In contrast to state of the art methods, which are based on the end-to-end neural network training, we proposed a hybrid approach. We use separate deep neural networks to: detect the object on the image, estimate the center of the object, and estimate the translation and ”in-place” rotation of the object. Then, we use geometrical relations on the image and the camera model to recover the full 6D object pose. As a result, we avoid the direct estimation of the object orientation defined in SO3 using a neural network. We propose the 4D-NET neural network to estimate translation and ”in-place” rotation of the object. Finally, we show results on the images generated from the Pascal VOC and ShapeNet datasets.

In this paper, we propose a method for selecting the optimal footholds for legged systems. The goal of the proposed method is to find the best foothold for the swing leg on a local elevation map. First, we evaluate the geometrical characteristics of each cell on the elevation map, checks kinematic constraints and collisions. Then, we apply the Convolutional Neural Network to learn the relationship between the local elevation map and the quality of potential footholds. During execution time, the controller obtains the qualitative measurement of each potential foothold from the neural model. This method evaluates hundreds of potential footholds and checks multiple constraints in a single step which takes 10 ms on a standard computer without GPU. The experiments were carried out on a quadruped robot walking over rough terrain in both simulation and real robotic platforms.

In order to operate autonomously, a robot should explore the environment and build a model of each of the surrounding objects. A common approach is to carefully scan the whole workspace. This is time-consuming. It is also often impossible to reach all the viewpoints required to acquire full knowledge about the environment. Humans can perform shape completion of occluded objects by relying on past experience. Therefore, we propose a method that generates images of an object from various viewpoints using a single input RGB image. A deep neural network is trained to imagine the object appearance from many viewpoints. We present the whole pipeline, which takes a single RGB image as input and returns a sequence of RGB and depth images of the object. The method utilizes a CNN-based object detector to extract the object from the natural scene. Then, the proposed network generates a set of RGB and depth images. We show the results both on a synthetic dataset and on real images.

In this paper, we propose a new mapping method based on Normal Distribution Transform Occupancy Maps (NDT-OM) for environment exploration. Our goal is to propose a new architecture which can be used by an industrial mobile robot in a priori unknown environment. The mobile robot introduced in a new environment has to explore the workspace, localize itself and build a map. Current state of the art methods require storing all data collected during this stage and finally build a dense model of the environment. We propose a method which allows building local dense maps of the environment which are organized in a graph-like structure. The change in the registered trajectory of the robot, which may occur after loop closure detection, can be easily utilized by our architecture. Finally, we build a global map which can be later used for collision checking and motion planning.

This paper deals with the problem of modeling spatial uncertainty of point features in feature-based RGB-D SLAM. Although the feature-based approach to SLAM is very popular, in the case of systems using RGB-D data the problem of explicit uncertainty modeling is largely neglected in the implementations. Therefore, we investigate the influence of the uncertainty models of point features on the accuracy of the estimated trajectory and map. We focus on the recent SLAM formulation employing factor graph optimization. Unlike some visual SLAM systems employing factor graph optimization that minimize the reprojection errors of features, we explicitly use depth measurements and minimize the errors in the 3-D space. The paper analyzes the impact of the information matrices used in factor graph optimization on the achieved accuracy. We introduce three different models of point feature spatial uncertainty. Then, applying the most simple model, we demonstrate in simulations how important is the influence of the spatial uncertainty model on the graph optimization results in an idealized SLAM system with perfect feature matching. A novel software tool allows us to visualize the statistical behavior of the features over time in a real SLAM system. This enables the analysis of the distribution of feature measurements employing synthetic RGB-D data processed in an actual SLAM pipeline. Finally, we show on publicly available real RGB-D datasets how an uncertainty model, which reflects the properties of the RGB-D sensor and the image processing pipeline, improves the accuracy of sensor trajectory estimation.

Many recent solutions to the RGB-D SLAM problem use the pose-graph optimization approach, which marginalizes out the actual depth measurements. In this paper we employ the same type of factor graph optimization, but we investigate the gains coming from maintaining a map of RGBD point features and modeling the spatial uncertainty of these features. We demonstrate that RGB-D SLAM accuracy can be increased by employing uncertainty models reflecting the actual errors introduced by measurements and image processing. The new approach is validated in simulations and in experiments involving publicly available data sets to ensure that our results are verifiable.

Predicting the motions of rigid objects under contacts is a necessary precursor to planning of robot manipulation of objects. On the one hand physics based rigid body simulations are used, and on the other learning approaches are being developed. The advantage of physics simulations is that because they explicitly perform collision checking they respect kinematic constraints, producing physically plausible predictions. The advantage of learning approaches is that they can capture the effects on motion of unobservable parameters such as mass distribution, and frictional coefficients, thus producing more accurate predicted trajectories. This paper shows how to bring together the advantages of both approaches to achieve learned simulators of specific objects that outperform previous learning approaches. Our approach employs a fast simplified collision checker and a learning method. The learner predicts trajectories for the object. These are optimised post prediction to minimise interpenetrations according to the collision checker. In addition we show that cleaning the training data prior to learning can also improve performance. Combining both approaches results in consistently strong prediction performance. The new simulator outperforms previous learning based approaches on a single contact push manipulation prediction task. We also present results showing that the method works for multi-contact manipulation, for which rigid body simulators are notoriously unstable.

This paper presents a posture optimization algorithm for a six-legged walking robot. During walking on rough terrain and planning its motion the robot has to determine the horizontal position, distance to the ground, and inclination of the platform. The proposed posture optimization algorithm is based on the Particle Swarm Optimization method. The algorithm increases the stability margin and maximizes the possible motion range of the robot (by maximizing the kinematic margin of each leg). The computation of the kinematic margin is performed by using an analytical function obtained with the Gaussian approximation. The Gaussian-based approximation significantly decreases the time consumed by the algorithm and allows to implement the posture optimization procedure on the real robot. The posture optimization is used as a part of the RRT-based motion planer to find a full-body path while climbing the obstacles.

This paper presents an adaptive foothold planning method for a hexapod walking robot. A local terrain map acquired with an inexpensive structured light sensor is exploited as the information source for the planning algorithm, which uses a polynomial-based approximation method to create a decision surface. The robot learns from simulations, therefore no a priori knowledge is required. The results show that the method is general enough to work on various types of terrain. The planned footholds enable the robot to walk more stable, avoiding slippages and fall-downs.

The objective of this paper is to develop feasible gait patterns that could be used to control a real hexapod walking robot. These gaits should enable the fastest movement that is possible with the given robot mechanics and drives on a flat terrain. We show in a series of evolutionary simulations how a gradual reduction of the permissible state space of the movements of the robot legs leads to the proper leg trajectories for a hexapod robot. This strategy enables the learning system to discover feasible gaits, using only simple dependencies between the control signals of the legs and a simple fitness function. Finally, a stable and fast tripod gait evolved in simulation is shown in an experiment on the real walking robot Ragno.