Text is one of the richest sources of information in an urban environment. Although textual information is heavily relied on by humans for a majority of the daily tasks, its usage has not been completely exploited in the field of robotics. In this work, we propose a localization approach utilizing textual features in urban environments. Starting at an unknown location, equipped with an RGB-camera and a compass, our approach uses off-the-shelf text extraction methods to identify text labels in the vicinity. We then apply a probabilistic localization approach with specific sensor models to integrate multiple observations. An extensive evaluation with real-world data gathered in different cities reveals an improvement over GPS-based localization when using our method.

As service robots become more and more capable of performing useful tasks for us, there is a growing need to teach robots how we expect them to carry out these tasks. However, learning our preferences is a nontrivial problem, as many of them stem from a variety of factors including personal taste, cultural background, or common sense. Obviously, such factors are hard to formulate or model a priori. In this paper, we present a solution for tidying up objects in containers, e.g., shelves or boxes, by following user preferences. We learn the user preferences using collaborative filtering based on crowdsourced and mined data. First, we predict pairwise object preferences of the user. Then, we subdivide the objects in containers by modeling a spectral clustering problem. Our solution is easy to update, does not require complex modeling, and improves with the amount of user data. We evaluate our approach using crowdsoucing data from over 1,200 users and demonstrate its effectiveness for two tidy-up scenarios. Additionally, we show that a real robot can reliably predict user preferences using our approach.

Quickly finding a free parking spot that is close to a desired target location can be a difficult task. This holds for human drivers and autonomous cars alike. In this paper, we investigate the problem of predicting the occupancy of parking spaces and exploiting this information during route planning. We propose an MDP-based planner that considers route information as well as the occupancy probabilities of parking spaces to compute the path that minimizes the expected total time for finding an unoccupied parking space and for walking from the parking location to the target destination. We evaluated our system on real world data gathered over several days in a real parking lot. We furthermore compare our approach to three parking strategies and show that our method outperforms the alternative behaviors.

The ability to localize a robot is an important capability and matching of observations under substantial changes is a prerequisite for robust long-term operation. This paper investigates the problem of efficiently coping with seasonal changes in image data. We present an extension of a recent approach [15] to visual image matching using sequence information. Our extension allows for exploiting GPS priors in the matching process to overcome the main computational bottleneck of the previous method and to handle loops within the image sequences. We present an experimental evaluation using real world data containing substantial seasonal changes and show that our approach outperforms the previous method in case a noisy GPS pose prior is available.

Accurate metrical localization is one of the central challenges in mobile robotics. Many existing methods aim at localizing after building a map with the robot. In this paper, we present a novel approach that instead uses geo-tagged panoramas from the Google Street View as a source of global positioning. We model the problem of localization as a non-linear least squares estimation in two phases. The first estimates the 3D position of tracked feature points from short monocular camera sequences. The second computes the rigid body transformation between the Street View panoramas and the estimated points. The only input of this approach is a stream of monocular camera images and odometry estimates. We quantified the accuracy of the method by running the approach on a robotic platform in a parking lot by using visual fiducials as ground truth. Additionally, we applied the approach in the context of personal localization in a real urban scenario by using data from a Google Tango tablet.

Robust object recognition is a crucial ingredient of many, if not all, real-world robotics applications. This paper leverages recent progress on Convolutional Neural Networks (CNNs) and proposes a novel RGB-D architecture for object recognition. Our architecture is composed of two separate CNN processing streams - one for each modality - which are consecutively combined with a late fusion network. We focus on learning with imperfect sensor data, a typical problem in real-world robotics tasks. For accurate learning, we introduce a multi-stage training methodology and two crucial ingredients for handling depth data with CNNs. The first, an effective encoding of depth information for CNNs that enables learning without the need for large depth datasets. The second, a data augmentation scheme for robust learning with depth images by corrupting them with realistic noise patterns. We present state-of-the-art results on the RGB-D object dataset [15] and show recognition in challenging RGB-D real-world noisy settings.

In this paper, we present an appearance-based visual SLAM approach that focuses on detecting loop closures across seasons. Given two image sequences, our method first extracts one descriptor per image for both sequences using a deep convolutional neural network. Then, we compute a similarity matrix by comparing each image of a query sequence with a database. Finally, based on the similarity matrix, we formulate a flow network problem and compute matching hypotheses between sequences. In this way, our approach can handle partially matching routes, loops in the trajectory and different speeds of the robot. With a matching hypothesis as loop closure information and the odometry information of the robot, we formulate a graph based SLAM problem and compute a joint maximum likelihood trajectory.

Accurate localization in indoor environments is widely regarded as a key opener for various location-based services. Despite tremendous advancements in the development of innovative sensor concepts, the most effective and accurate solutions to this problem make use of a map computed from sensory data. In this paper, we present an efficient approach to localize an RGB-D smartphone or tablet that only makes use of a two-dimensional outline of the environment as a map as it is typically available from architectural drawings. Our technique employs a particle filter to estimate the 6DoF pose. We propose a sensor model that robustly estimates the likelihood of measurements and accommodates the disagreements between floor plans and real world data. In extensive experiments, we demonstrate that our approach is able to globally localize a user in a given 2D floor plan using a Google Tango device and to accurately track the user in such an environment.

Image-based localization is an important problem in robotics and an integral part of visual mapping and navigation systems. An approach to robustly match images to previously recorded ones must be able to cope with seasonal changes especially when it is supposed to work reliably over long periods of time. In this paper, we present a novel approach to visual localization of mobile robots in outdoor environments, which is able to deal with substantial seasonal changes. We formulate image matching as a minimum cost flow problem in a data association graph to effectively exploit sequence information. This allows us to deal with non-matching image sequences that result from temporal occlusions or from visiting new places. We present extensive experimental evaluations under substantial seasonal changes. Our approach achieves accurate matching across seasons and outperforms existing state-of-the-art methods such as FABMAP2 and SeqSLAM.

Learning from demonstrations is an intuitive way for instructing robots by non-experts. One challenge in learning from demonstrations is to infer what to imitate, especially when the robot only observes the teacher and does not have further knowledge about the demonstrated actions. In this paper, we present a novel approach to the problem of inferring what to imitate to successfully reproduce a manipulation action based on a small number of demonstrations. Our method employs techniques from recommender systems to include expert knowledge. It models the demonstrated actions probabilistically and formulates the problem of inferring what to imitate via model selection. We select an appropriate model for the action each time the robot has to reproduce it given a new starting condition. We evaluate our approach using data acquired with a PR2 robot and demonstrate that our method achieves high success rates in different scenarios.

Non-linear error minimization methods became widespread approaches for solving the simultaneous localization and mapping problem. If the initial guess is far away from the global minimum, converging to the correct solution and not to a local one can be challenging and sometimes even impossible. This paper presents an experimental analysis of dynamic covariance scaling, a recently proposed method for robust optimization of SLAM graphs, in the context of a poor initialization. Our evaluation shows that dynamic covariance scaling is able to mitigate the effects of poor initializations. In contrast to other methods that first aim at finding a good initial guess to seed the optimization, our method is more elegant because it does not require an additional method for initialization. Furthermore, it can robustly handle data association outliers. Experiments performed with real world and simulated datasets show that dynamic covariance scaling outperforms existing methods, both in the presence and absence of data association outliers.

Map consistency is an important requirement for applications in which mobile robots need to effectively perform autonomous navigation tasks. While recent SLAM techniques provide an increased robustness even in the context of bad initializations or data association outliers, the question of how to determine whether or not the resulting map is consistent is still an open problem. In this paper, we introduce a novel measure for map consistency. We compute this measure by taking into account the discrepancies in the sensor data and leverage it to address two important problems in SLAM. First, we derive a statistical test for assessing whether a map is consistent or not. Second, we employ it to automatically set the free parameter of dynamic covariance scaling, a robust SLAM back-end. We present an evaluation of our approach on over 50 maps sourced from 16 publicly available datasets and illustrate its capability for the inconsistency detection and the tuning of the parameter of the back-end.

In the past, highly effective solutions to the SLAM problem based on solving nonlinear optimization problems have been developed. However, most approaches put their major focus on runtime and accuracy rather than on memory consumption, which becomes especially relevant when large-scale SLAM problems have to be solved. In this paper, we consider the SLAM problem from the point of view of memory consumption and present a novel approximate approach to SLAM with low memory consumption. Our approach achieves this based on a hierarchical decomposition consisting of small submaps with limited size. We perform extensive experiments on synthetic and publicly available datasets. The results demonstrate that in situations in which the representation of the complete map requires more than the available main memory, our approach, in comparison to state-of-the-art exact solvers, reduces the memory consumption and the runtime up to a factor of 2 while still providing highly accurate maps.

Most robots need the ability to communicate with a base station or with an operator during their mission. Teleoperated and semi-autonomous robots typically communicate continuously through a network connection with an operator. Transmitting raw sensor data over a low bandwidth network such as wireless or HSDPA, however, is problematic as the stream of sensor data is often large. In this paper, we present a method that exploits H.264 compression to reduce the size of range data streams from sensors such as the Kinect camera or the Velodyne 3D laser scanner. We developed a practical and effective solution that exploits the state of the art in video compression to produce high-quality results. Our method is easy to implement and can have practical impact for researchers building robots for the real world. We implemented and thoroughly tested our approach using a large number of range data streams. Furthermore, we analyzed the impact of data compression on the accuracy and size of the transmitted data. We show that even a highly compressed stream of depth images can be used with dense mapping techniques such as KinFu for building environment models.

To efficiently plan complex manipulation tasks, robots need to reason on a high level. Symbolic planning, however, requires knowledge about the preconditions and effects of the individual actions. In this work, we present a practical approach to learn manipulation skills, including preconditions and effects, based on teacher demonstrations. We believe that requiring only a small number of demonstrations is essential for robots operating in the real world. Therefore, our main focus and contribution is the ability to infer the preconditions and effects of actions based on a small number of demonstrations. Our system furthermore expresses the acquired manipulation actions as planning operators and is therefore able to use symbolic planners to solve new tasks. We implemented our approach on a PR2 robot and present real world manipulation experiments that illustrate that our system allows non-experts to transfer knowledge to robots.

Developing the perfect SLAM front-end that produces graphs which are free of outliers is generally impossible due to perceptual aliasing. Therefore, optimization back-ends need to be able to deal with outliers resulting from an imperfect front-end. In this paper, we introduce dynamic covariance scaling, a novel approach for effective optimization of constraint networks under the presence of outliers. The key idea is to use a robust function that generalizes classical gating and dynamically rejects outliers without compromising convergence speed. We implemented and thoroughly evaluated our method on publicly available datasets. Compared to recently published state-of-the-art methods, we obtain a substantial speed up without increasing the number of variables in the optimization process. Our method can be easily integrated in almost any SLAM back-end.

Place recognition, i.e., the problem of recognizing if the robot is navigating in an already visited place, is a fundamental problem in mobile robot navigation. Efficient solutions to this problem are relevant for effectively localizing robots and for creating maps in real time. Relatively few methods have been proposed to efficiently solve this problem in very large environments using 2D range data. In this paper, we introduce geometrical FLIRT phrases (GFPs) as a novel retrieval method for very efficient and precise place recognition. GFPs perform approximate 2D range data matching, have low computational cost, can handle complicated partial matching patterns and are robust to noise. Experiments carried out with publicly available datasets demonstrate that GFPs largely outperform state-of-the-art approaches in 2D range-based place recognition in terms of efficiency and recall. We obtain retrieval performances with more than 85% recall at 99% precision in less than a second, even on data sets obtained from several kilometer long runs.

Platforms with holonomic drives are particularly interesting due to their maneuvering capabilities. Robots used for transportation tasks usually have a non-circular footprint. In this work, we present a navigation strategy for a holonomic mobile robot with anyshape footprint. Our technique introduces an efficient navigation method based on a strategy that makes use of discrete and continuous techniques. We introduce compact discrete intervals to represent the free space for computing fast-to-update plans. Based on these, we provide a continuous motion generation approach to generate smooth motions that are fast to compute. We evaluated our approach by running simulated experiments and by using a real holonomic L-shaped robot. Our experiments demonstrate that our technique can be carried out online and is able to smoothly drive the robot to its goal locations even in dynamic environments.

Vision and range sensing belong to the richest sensory modalities for perception in robotics and related fields. This paper addresses the problem of how to best combine image and range data for the task of object detection. In particular, we propose a novel adaptive fusion approach, hierarchical Gaussian Process mixtures of experts, able to account for missing information and cross-cue data consistency. The hierarchy is a two-tier architecture that for each modality, each frame and each detection computes a weight function using Gaussian Processes that reflects the confidence of the respective information. We further propose a method called cross-cue domain adaptation that makes use of large image data sets to improve the depth-based object detector for which only few training samples exist. In the experiments that include a comparison with alternative sensor fusion schemes, we demonstrate the viability of the proposed methods and achieve significant improvements in classification accuracy.

The ability to act in a socially-aware way is a key skill for robots that share a space with humans. In this paper we address the problem of socially-aware navigation among people that meets objective criteria such as travel time or path length as well as subjective criteria such as social comfort. Opposed to model-based approaches typically taken in related work, we pose the problem as an unsupervised learning problem. We learn a set of dynamic motion prototypes from observations of relative motion behavior of humans found in publicly available surveillance data sets. The learned motion prototypes are then used to compute dynamic cost maps for path planning using an any-angle A* algorithm. In the evaluation we demonstrate that the learned behaviors are better in reproducing human relative motion in both criteria than a Proxemics-based baseline method.

Human activity recognition is a key component for socially enabled robots to effectively and naturally interact with humans. In this paper we exploit the fact that many human activities produce characteristic sounds from which a robot can infer the corresponding actions. We propose a novel recognition approach called Non-Markovian Ensemble Voting (NEV) able to classify multiple human activities in an online fashion without the need for silence detection or audio stream segmentation. Moreover, the method can deal with activities that are extended over undefined periods in time. In a series of experiments in real reverberant environments, we are able to robustly recognize 22 different sounds that correspond to a number of human activities in a bathroom and kitchen context. Our method outperforms several established classification techniques.

This paper presents a novel self-supervised online learning method to discover driving behaviors from data acquired with an inertial measurement unit (IMU) and a camera. Both sensors where mounted in a car that was driven by a human through a typical city environment with intersections, pedestrian crossings and traffic lights. The presented system extracts motion segments from the IMU data and relates them to visual cues obtained from camera data. It employs a Bayesian on-line estimation method to discover the motion segments based on change-point detection and uses a Dirichlet Compound Multinomial (DCM) model to represent the visual features extracted from the camera images. By incorporating these visual cues into the on-line estimation process, labels are computed that are equal for similar motion segments. As a result, typical traffic situations such as braking maneuvers in front of a red light can be identified automatically. Furthermore, appropriate actions in form of observed motion changes are associated to the discovered traffic situations. The approach is evaluated on a real data set acquired in the center of Zurich.

People detection and tracking is a key component for robots and autonomous vehicles in human environments. While prior work mainly employed image or 2D range data for this task, in this paper, we address the problem using 3D range data. In our approach, a top-down classifier selects hypotheses from a bottom-up detector, both based on sets of boosted features. The bottom-up detector learns a layered person model from a bank of specialized classifiers for different height levels of people that collectively vote into a continuous space. Modes in this space represent detection candidates that each postulate a segmentation hypothesis of the data. In the top-down step, the candidates are classified using features that are computed in voxels of a boosted volume tessellation. We learn the optimal volume tessellation as it enables the method to stably deal with sparsely sampled and articulated objects. We then combine the detector with tracking in 3D for which we take a multi-target multi-hypothesis tracking approach. The method neither needs a ground plane assumption nor relies on background learning. The results from experiments in populated urban environments demonstrate 3D tracking and highly robust people detection up to 20 m with equal error rates of at least 93%.

People tracking is a key component for robots that are deployed in populated environments. Previous works have used cameras and 2D and 3D range finders for this task. In this paper, we present a 3D people detection and tracking approach using RGB-D data. We combine a novel multi-cue person detector for RGB-D data with an on-line detector that learns individual target models. The two detectors are integrated into a decisional framework with a multi-hypothesis tracker that controls on-line learning through a track interpretation feedback. For on-line learning, we take a boosting approach using three types of RGB-D features and a confidence maximization search in 3D space. The approach is general in that it neither relies on background learning nor a ground plane assumption. For the evaluation, we collect data in a populated indoor environment using a setup of three Microsoft Kinect sensors with a joint field of view. The results demonstrate reliable 3D tracking of people in RGB-D data and show how the framework is able to avoid drift of the on-line detector and increase the overall tracking performance.

People detection is a key issue for robots and intelligent systems sharing a space with people. Previous works have used cameras and 2D or 3D range finders for this task. In this paper, we present a novel people detection approach for RGB-D data. We take inspiration from the Histogram of Oriented Gradients (HOG) detector to design a robust method to detect people in dense depth data, called Histogram of Oriented Depths (HOD). HOD locally encodes the direction of depth changes and relies on an depth-informed scale-space search that leads to a 3-fold acceleration of the detection process. We then propose Combo-HOD, a RGB-D detector that probabilistically combines HOD and HOG. The experiments include a comprehensive comparison with several alternative detection approaches including visual HOG, several variants of HOD, a geometric person detector for 3D point clouds, and an Haar-based AdaBoost detector. With an equal error rate of 85% in a range up to 8m, the results demonstrate the robustness of HOD and Combo-HOD on a real-world data set collected with a Kinect sensor in a populated indoor environment.

People tracking is a key technology for autonomous systems, intelligent cars and social robots operating in populated environments. What makes the task difficult is that the appearance of humans in range data can change drastically as a function of body pose, distance to the sensor, self-occlusion and occlusion by other objects. In this paper we propose a novel approach to pedestrian detection in 3D range data based on supervised learning techniques to create a bank of classifiers for different height levels of the human body. In particular, our approach applies AdaBoost to train a strong classifier from geometrical and statistical features of groups of neighboring points at the same height. In a second step, the AdaBoost classifiers mutually enforce their evidence across different heights by voting into a continuous space. Pedestrians are finally found efficiently by mean-shift search for local maxima in the voting space. Experimental results carried out with 3D laser range data illustrate the robustness and efficiency of our approach even in cluttered urban environments. The learned people detector reaches a classification rate up to 96% from a single 3D scan.

In this paper, we are presenting a method to estimate terrain properties (such as small-scale geometry or surface friction) to improve the assessment of stability and the guiding of foot placement of legged robots in rough terrain. Haptic feedback, expressed through joint motor currents and ground contact force measurements that arises when prescribing a predefined motion was collected for a variety of ground samples (four different shapes and four different surface properties). Features were extracted from this data and used for training and classification by a multiclass AdaBoost machine learning algorithm. In a single leg testbed, the algorithm could correctly classify about 94% of the terrain shapes, and about 73% of the surface samples.

Most large-scale public environments provide direction signs to facilitate the orientation for humans and to find their way to a goal location in the environment. Thus, for a robot operating in the same environment, it would be beneficial to interpret such signs correctly for a safe and efficient navigation. In this work, we propose a novel approach to infer the meaning of direction signs and to use that for navigation, i.e., to find a mapping of a detected sign to a motion direction. Our method uses a hierarchical extension of the Implicit Shape Model framework called HISM that does not require any hand-labeled training data to detect the signs. On the lower level of this two-stage hierarchy, ISM is applied to image descriptors as in the standard approach. On the higher level, ISM operates on subparts of signs called tokens, using weights learned from data. The interpretation of the signs is inferred by associating navigation data to direction instructions. We conducted experiments from image data acquired in an airport terminal, aiming towards the implementation of a robotic guide, with promising results.

This paper presents a novel image based detection method for pedestrians at very small scales (between 16 × 20 and 32 × 40). We propose a set of new distinctive image features based on collections of local image gradients grouped by a superpixel segmentation. Features are collected and classified using AdaBoost. The positive classified features then vote for potential hypotheses that are collected using a mean shift mode estimation approach. The presented method overcomes the common limitations of a sliding window approach as well as those of standard voting approaches based on interest points. Extensive tests have been produced on a dataset with more than 20000 images showing the potential of this approach.

This paper presents a novel human detection method based on a Bayesian fusion approach using laser range data and camera images. Laser range data analysis groups data points with a novel graph cutting method. Therefore, it computes a belief to each cluster based on the evaluation of multidimensional features that describe geometrical properties. A person detection algorithm based on dense overlapping grid of Histograms of Oriented Gradients (HOG) is processed on the image area determined by each laser cluster. The selection of HOG features and laser features is obtained through a learning process based on a cascade of linear Support Vector Machines (SVM). A technique to obtain conditional probabilities from a cascade of SVMs is here proposed in order to combine the two information together. The resulting human detection consists in a rich information that takes into account the distance of the cluster and the confidence level of both detection methods. We demonstrate the performance of this work on real-world data and different environments.

This paper presents a novel people detection and tracking method based on a combined multimodal sensor approach that utilizes 2D and 3D laser range and camera data. Laser data points are clustered and classified with a set of geometrical features using an SVM AdaBoost method. The clusters define a region of interest in the image that is adjusted using the ground plane information extracted from the 3D laser. In this areas a novel vision based people detector based on Implicit Shape Model (ISM) is applied. Each detected person is tracked using a greedy data association technique and multiple Extended Kalman Filters that use different motion models. This way, the filter can cope with a variety of different motion patterns. The tracker is asynchronously updated by the detections from the laser and the camera data. Experiments conducted in real-world outdoor scenarios with crowds of pedestrians demonstrate the usefulness of our approach.