Motion planning is a mature area of research in robotics with many well-established methods based on optimization or sampling the state space, suitable for solving kinematic motion planning. However, when dynamic motions under constraints are needed and computation time is limited, fast kinodynamic planning on the constraint manifold is indispensable. In recent years, learning-based solutions have become alternatives to classical approaches, but they still lack comprehensive handling of complex constraints, such as planning on a lower-dimensional manifold of the task space while considering the robot's dynamics. This paper introduces a novel learning-to-plan framework that exploits the concept of constraint manifold, including dynamics, and neural planning methods. Our approach generates plans satisfying an arbitrary set of constraints and computes them in a short constant time, namely the inference time of a neural network. This allows the robot to plan and replan reactively, making our approach suitable for dynamic environments. We validate our approach on two simulated tasks and in a demanding real-world scenario, where we use a Kuka LBR Iiwa 14 robotic arm to perform the hitting movement in robotic Air Hockey.

Although haptic sensing has recently been used for legged robot localization in extreme environments where a camera or LiDAR might fail, the problem of efficiently representing the haptic signatures in a learned prior map is still open. This paper introduces an approach to terrain representation for haptic localization inspired by recent trends in machine learning. It combines this approach with the proven Monte Carlo algorithm to obtain an accurate, computation-efficient, and practical method for localizing legged robots under adversarial environmental conditions. We apply the triplet loss concept to learn highly descriptive embeddings in a transformer-based neural network. As the training haptic data are not labeled, the positive and negative examples are discriminated by their geometric locations discovered while training. We demonstrate experimentally that the proposed approach outperforms by a large margin the previous solutions to haptic localization of legged robots concerning the accuracy, inference time, and the amount of data stored in the map. As far as we know, this is the first approach that completely removes the need to use a dense terrain map for accurate haptic localization, thus paving the way to practical applications.

In this paper, we consider a challenging scenario of localising a camera with respect to a charging station for electric buses.In this application, we face a number of problems, including a substantial scale change as the bus approaches the station, and the need to detect keypoints on a weakly textured object in a wide range of lighting and weather conditions.Therefore, we use a deep convolutional neural network to detect the features, while retaining a conventional procedure for pose estimation with 2D-to-3D associations.We leverage here the backbone of HRNet, a state-of-the-art network used for detection of feature points in human pose recognition, and we further improve the solution adding constraints that stem from the known scene geometry.We incorporate the reprojection-based geometric priors in a novel loss function for HRNet training and use the object geometry to construct sanity checks in postprocessing.Moreover, we demonstrate that our Geometry-Aware Keypoint Network yields feasible estimates of the geometric uncertainty of point features.The proposed architecture and solutions are tested on a large dataset of images and trajectories collected with a real city bus and charging station under varying environmental conditions.

Although accurate and reliable localization is a prerequisite for autonomous driving, in urban environments neither the Global Navigation Satellite System (GNSS) nor the Simultaneous Localization and Mapping (SLAM) ensure satisfying results in terms of both local accuracy and global consistency. Hence, we contribute in this paper a method to augment the existing LiDAR-based SLAM systems with GNSS measurements, applying the factor graph formulation of the problem. We contribute a tightly coupled GNSS/LiDAR SLAM considering constraints from LiDAR and GNSS measurements, and propose a filtering procedure to cope with GNSS measurements that introduce non-Gaussian noise. We evaluate our approach on the challenging UrbanNav dataset, considering different LiDAR SLAM algorithms and different GNSS receivers, and showing that our solution outperforms previous approaches to GNSS/LiDAR integration.

This paper demonstrates how an efficient repre-sentation of the planned path using B-splines, and a construction procedure that takes advantage of the neural network's inductive bias, speed up both the inference and training of a DNN-based motion planner. We build upon our recent work on learning local car maneuvers from past experience using a DNN architecture, introducing a novel B-spline path construction method, making it possible to generate local maneuvers in almost constant time of about 11 ms, respecting a number of constraints imposed by the environment map and the kinematics of a car-like vehicle. We evaluate thoroughly the new planner employing the recent Bench-MR framework to obtain quantitative results showing that our method outperforms state-of-the-art planners by a large margin in the considered task.

We investigate a new method to design$G$-invariant neural networks that approximate functions invariant to the action of a given permutation subgroup$G$of the symmetric group on input data. The key element of the new network architecture is a$G$-invariant transformation module, which produces a$G$-invariant latent representation of the input data. This latent representation is then processed with a multi-layer perceptron in the network. We prove the universality of the new architecture, discuss its properties and highlight its computational and memory efficiency. Theoretical considerations are supported by numerical experiments involving different network configurations, which demonstrate the efficiency and strong generalization properties of the new approach to design symmetry invariant neural networks, in comparison to other$G$-invariant neural architectures.

We propose an extension to the segment-based global localization method for LiDAR SLAM using descriptors learned considering the visual context of the segments. A new architecture of the deep neural network is presented that learns the visual context acquired from synthetic LiDAR intensity images. This approach allows a single multi-beam LiDAR to produce rich and highly descriptive location signatures. The method is tested on two public datasets, demonstrating an improved descriptiveness of the new descriptors, and more reliable loop closure detection in SLAM. Attention analysis of the network is used to show the importance of focusing on the broader context rather than only on the 3-D segment.

We describe and experimentally demonstrate a practical method for indoor localization using measurements obtained from resource-constrained devices with limited sensing capabilities. We focus on handheld/mobile devices but the method can be useful for a variety of wearable devices. Our system works with sparse WiFi or image-based measurements, avoiding laborious site surveying for dense signal maps and runs in real-time. It uses Conditional Random Fields to infer the most probable sequence of agent positions from a known floor plan, dead reckoning and sparse absolute position estimates. Our solution leverages known topology of the environment by pre-computing allowed motion sequences of an agent, which are then used to constraint the motion inferred from the sensory data. The system is evaluated in a typical office building, demonstrating good accuracy and robustness to sparse, low-quality measurements.

Reliable perception and detection of objects are one of the fundamental aspects of vehicle autonomy. Although model-based approaches perform well in the area of planning and control, they often fail when applied to perception due to the open-world nature of problems for autonomous vehicles. Therefore, data-driven approaches to object detection and location are likely to be used in both self-driving cars and advanced driver assistance systems. In particular, the deep neural networks proved to be excellent in detection and classification of objects from images, often achieving super-human performance. However, neural networks applied in intelligent vehicles need to be explainable, providing rationales for their decisions. In this paper, we demonstrate how such an interpretation can be provided for a deep learning system that detects specific objects (charging posts) for driver assistance in an electric bus. The interpretation, achieved by visualization of attention heat maps, has twofold use: it allows us to augment the dataset used for training, improving the results, but it also may be used as a tool when fielding the system with the given bus operator. Explaining which parts of the images triggered the decision helps to eliminate misdetections.

This paper deals with the problem of modeling spatial uncertainty of point features in feature-based RGB-D SLAM. Although the feature-based approach to SLAM is very popular, in the case of systems using RGB-D data the problem of explicit uncertainty modeling is largely neglected in the implementations. Therefore, we investigate the influence of the uncertainty models of point features on the accuracy of the estimated trajectory and map. We focus on the recent SLAM formulation employing factor graph optimization. Unlike some visual SLAM systems employing factor graph optimization that minimize the reprojection errors of features, we explicitly use depth measurements and minimize the errors in the 3-D space. The paper analyzes the impact of the information matrices used in factor graph optimization on the achieved accuracy. We introduce three different models of point feature spatial uncertainty. Then, applying the most simple model, we demonstrate in simulations how important is the influence of the spatial uncertainty model on the graph optimization results in an idealized SLAM system with perfect feature matching. A novel software tool allows us to visualize the statistical behavior of the features over time in a real SLAM system. This enables the analysis of the distribution of feature measurements employing synthetic RGB-D data processed in an actual SLAM pipeline. Finally, we show on publicly available real RGB-D datasets how an uncertainty model, which reflects the properties of the RGB-D sensor and the image processing pipeline, improves the accuracy of sensor trajectory estimation.

Although the introduction of commercial RGB-D sensors has enabled significant progress in the visual navigation methods for mobile robots, the structured-light-based sensors, like Microsoft Kinect and Asus Xtion Pro Live, have some important limitations with respect to their range, field of view, and depth measurements accuracy. The recent introduction of the second- generation Kinect, which is based on the time-of-flight measurement principle, brought to the robotics and computer vision researchers a sensor that overcomes some of these limitations. However, as the new Kinect is, just like the older one, intended for computer games and human motion capture rather than for navigation, it is unclear how much the navigation methods, such as visual odometry and SLAM, can benefit from the improved parameters. While there are many publicly available RGB-D data sets, only few of them provide ground truth information necessary for evaluating navigation methods, and to the best of our knowledge, none of them contains sequences registered with the new version of Kinect. Therefore, this paper describes a new RGB-D data set, which is a first attempt to systematically evaluate the indoor navigation algorithms on data from two different sensors in the same environment and along the same trajectories. This data set contains synchronized RGB-D frames from both sensors and the appropriate ground truth from an external motion capture system based on distributed cameras. We describe in details the data registration procedure and then evaluate our RGB-D visual odometry algorithm on the obtained sequences, investigating how the specific properties and limitations of both sensors influence the performance of this navigation method.

Many recent solutions to the RGB-D SLAM problem use the pose-graph optimization approach, which marginalizes out the actual depth measurements. In this paper we employ the same type of factor graph optimization, but we investigate the gains coming from maintaining a map of RGBD point features and modeling the spatial uncertainty of these features. We demonstrate that RGB-D SLAM accuracy can be increased by employing uncertainty models reflecting the actual errors introduced by measurements and image processing. The new approach is validated in simulations and in experiments involving publicly available data sets to ensure that our results are verifiable.

The paper presents a thorough evaluation of two representative visual place recognition algorithms that can be applied to the problem of indoor localization of a person equipped with a modern smartphone. The evaluation focuses on comparing two different state-of-the-art approaches: single image-based place recognition, represented by the FAB-MAP algorithm, and recognition based on a sequence of images, represented by the ABLE-M algorithm. The evaluation focuses on real-life localization examples in buildings of different structure and the influence of the presence of people in the environment on the recognition results. Moreover, the paper demonstrates feasibility and real-time performance of the visual place recognition methods implemented on an Android smartphone.

Consumer electronics mobile devices, such like smartphones and tablets, are quickly growing in computing power and become equipped with advanced sensors. This makes a modern mobile device a viable platform for many computation-intensive, real-time applications. In this paper we present a study on the performance and robustness of point features detection and description in images acquired by a mobile device in the context of visual navigation. This is an important step towards infrastructure-less indoor self-localization and user guidance using only a smartphone or tablet. We rigorously evaluate the performance of several interest point detector and descriptor pairs on three different Android devices, using image sequences from publicly available robotics-related data sets, as well as our own data set obtained using a smartphone.

This paper presents a posture optimization algorithm for a six-legged walking robot. During walking on rough terrain and planning its motion the robot has to determine the horizontal position, distance to the ground, and inclination of the platform. The proposed posture optimization algorithm is based on the Particle Swarm Optimization method. The algorithm increases the stability margin and maximizes the possible motion range of the robot (by maximizing the kinematic margin of each leg). The computation of the kinematic margin is performed by using an analytical function obtained with the Gaussian approximation. The Gaussian-based approximation significantly decreases the time consumed by the algorithm and allows to implement the posture optimization procedure on the real robot. The posture optimization is used as a part of the RRT-based motion planer to find a full-body path while climbing the obstacles.

This paper presents an adaptive foothold planning method for a hexapod walking robot. A local terrain map acquired with an inexpensive structured light sensor is exploited as the information source for the planning algorithm, which uses a polynomial-based approximation method to create a decision surface. The robot learns from simulations, therefore no a priori knowledge is required. The results show that the method is general enough to work on various types of terrain. The planned footholds enable the robot to walk more stable, avoiding slippages and fall-downs.

The objective of this paper is to develop feasible gait patterns that could be used to control a real hexapod walking robot. These gaits should enable the fastest movement that is possible with the given robot mechanics and drives on a flat terrain. We show in a series of evolutionary simulations how a gradual reduction of the permissible state space of the movements of the robot legs leads to the proper leg trajectories for a hexapod robot. This strategy enables the learning system to discover feasible gaits, using only simple dependencies between the control signals of the legs and a simple fitness function. Finally, a stable and fast tripod gait evolved in simulation is shown in an experiment on the real walking robot Ragno.

This article presents results concerning recognition and classification of the qualitative-type range measurement errors in a time-of-flight principle based 2D laser scanner used on mobile robots for navigation. The main source of the qualitative uncertainty are the mixed measurements. This effect has been investigated experimentally and explained by analyzing the physical phenomena underlying the sensor operation. A local grid map has been used to remove the erroneous range measurements. A novel fuzzy-set-based algorithm has been employed to update evidence in the grid. The results of tests show that this algorithm is superior to the common Bayesian approach, when qualitative errors in range measurements are present.

In this paper we discuss methods to reduce spatial uncertainty in the simultaneous localization and mapping (SLAM) procedure for a mobile robot equipped with a 2D laser scanner and operating in a structured, but non-static environment. We augment the classic EKF-based SLAM procedure with two new modules. The first one reliably extracts line segments from the laser scans, employing a novel fuzzy-set-based grid map. The second one corrects the robot odometry by using scan matching. Both modules rely on a laser scanner measurement model, which covers both the quantitative and qualitative types of uncertainty.

In the paper, the architecture of a mobile robot cooperating with other robots and some stationary devices in a task of collective perception and world modelling is considered. We analyze data-driven processing of information performed by an individual robot treated as an agent and we propose to organize it as a set of experts (also treated as agents) exchanging data by means of a blackboard. We analyze functions performed by the blackboard agents and present results of preliminary experiments with real robots.

The use of intelligent robotic systems operated through computer networks allows the users to interact with remote environments. We present the INTERLABNET system that allows us to tele-operate a semiautonomous mobile robot over the Internet/intranet network. The software of this system has been written entirely in Java to achieve high platform-independence. The user interface runs as an applet within a Web browser. We present the configuration of the system, main software modules and some results of experiments.