This article discusses the integration challenges and strategies for designing mobile robots, by focusing on the task-driven, optimal selection of hardware and software to balance safety, efficiency, and minimal usage of resources such as costs, energy, computational requirements, and weight. We emphasize the interplay between perception and motion planning in decision-making by introducing the concept of occupancy queries to quantify the perception requirements for sampling-based motion planners. Sensor and algorithm performance are evaluated using false negative rate and false positive rate across various factors such as geometric relationships, object properties, sensor resolution, and environmental conditions. By integrating perception requirements with perception performance, an integer linear programming approach is proposed for efficient sensor and algorithm selection and placement. This forms the basis for a co-design optimization that includes the robot body, motion planner, perception pipeline, and computing unit. We refer to this framework for solving the co-design problem of mobile robots as CODEI, short for co-design of embodied intelligence. A case study on developing an autonomous vehicle for urban scenarios provides actionable information for designers, and shows that complex tasks escalate resource demands, with task performance affecting choices of the autonomy stack. The study demonstrates that resource prioritization influences sensor choice: cameras are preferred for cost-effective and lightweight designs, while lidar sensors are chosen for better energy and computational efficiency.

This article discusses the integration challenges and strategies for designing mobile robots, by focusing on the task-driven, optimal selection of hardware and software to balance safety, efficiency, and minimal usage of resources such as costs, energy, computational requirements, and weight. We emphasize the interplay between perception and motion planning in decision-making by introducing the concept of occupancy queries to quantify the perception requirements for sampling-based motion planners. Sensor and algorithm performance are evaluated using false negative rate and false positive rate across various factors such as geometric relationships, object properties, sensor resolution, and environmental conditions. By integrating perception requirements with perception performance, an integer linear programming approach is proposed for efficient sensor and algorithm selection and placement. This forms the basis for a co-design optimization that includes the robot body, motion planner, perception pipeline, and computing unit. We refer to this framework for solving the co-design problem of mobile robots as CODEI, short for co-design of embodied intelligence. A case study on developing an autonomous vehicle for urban scenarios provides actionable information for designers, and shows that complex tasks escalate resource demands, with task performance affecting choices of the autonomy stack. The study demonstrates that resource prioritization influences sensor choice: cameras are preferred for cost-effective and lightweight designs, while lidar sensors are chosen for better energy and computational efficiency.

No abstract available.

In recent years, driving context perception has emerged as one of the key aspects to design driving assistance algorithms and user interfaces that are effective in adapting to different traffic situations or environments. To this aim, we introduce the Anonymous Driving Scene Perception (ADSP) Model, a novel deep neural network designed to classify anony-mous Controller Area Network (CAN)-bus data into multiple driving context domains. ADSP extends the idea of driving scene classification to time series signals, as previous works relied heavily on visual features. Our model achieved a multi -domain classification accuracy of 84.9% on our custom-built naturalistic data set, as a combination of 92.7% on road type classification and 90.1 % on binary traffic detection, performing 2.0% and 1.6% better than the state-of-the-art model for multivariate time series classification. Our work demonstrates the feasibility of driving scene classification from anonymous CAN-bus data, without collecting sensitive data from users (images or GPS).

Visual navigation for insect-scale robots is very challenging because in such a small scale, the size, weight, and power (SWaP) constraints do not appear to permit visual navigation techniques such as SLAM (Simultaneous Localization and Mapping) because they are likely to be too power-hungry. We propose to use a biology-inspired approach, which we term the bilinear optic flow approximation, that is more computationally efficient. We build on previous work that has shown that the bilinear approximation can be used for visual servoing. Here, we show that a bilinear approximator can be learned that is able to stabilize the heading of a robot while performing continuous forward motion in a corridor-shaped environment. This is a necessary capability for confined-space navigation that insect-sized robots are likely to perform. In this work, we describe the underlining methodology of the method and built a 2D visual simulation environment and omnidirectional camera model to validate our results.

Dynamic games feature a state-space complexity that scales superlinearly with the number of players. This makes this class of games often intractable even for a handful of players. We introduce the factorization process of dynamic games as a transformation leveraging the independence of players at equilibrium to build a leaner game graph. When applicable, it yields fewer nodes, fewer players per game node, hence much faster solutions. While for the general case checking for independence of players requires to solve the game itself, we observe that for dynamic games in the robotic domain there exist exact heuristics based on the spatio-temporal occupancy of the individual players. We validate our findings in realistic autonomous driving scenarios showing that already for a 4-player intersection we have a reduction of game nodes and solving time close to 99%.

Event cameras are bio-inspired sensors that differ from conventional frame cameras: Instead of capturing images at a fixed rate, they asynchronously measure per-pixel brightness changes, and output a stream of events that encode the time, location and sign of the brightness changes. Event cameras offer attractive properties compared to traditional cameras: high temporal resolution (in the order of μs), very high dynamic range (140 dB versus 60 dB), low power consumption, and high pixel bandwidth (on the order of kHz) resulting in reduced motion blur. Hence, event cameras have a large potential for robotics and computer vision in challenging scenarios for traditional cameras, such as low-latency, high speed, and high dynamic range. However, novel methods are required to process the unconventional output of these sensors in order to unlock their potential. This paper provides a comprehensive overview of the emerging field of event-based vision, with a focus on the applications and the algorithms developed to unlock the outstanding properties of event cameras. We present event cameras from their working principle, the actual sensors that are available and the tasks that they have been used for, from low-level vision (feature detection and tracking, optic flow, etc.) to high-level vision (reconstruction, segmentation, recognition). We also discuss the techniques developed to process events, including learning-based techniques, as well as specialized processors for these novel sensors, such as spiking neural networks. Additionally, we highlight the challenges that remain to be tackled and the opportunities that lie ahead in the search for a more efficient, bio-inspired way for machines to perceive and interact with the world.

In many situations it is either impossible or impractical to develop and evaluate agents entirely on the target domain on which they will be deployed. This is particularly true in robotics, where doing experiments on hardware is much more arduous than in simulation. This has become arguably more so in the case of learning-based agents. To this end, considerable recent effort has been devoted to developing increasingly realistic and higher fidelity simulators. However, we lack any principled way to evaluate how good a "proxy domain" is, specifically in terms of how useful it is in helping us achieve our end objective of building an agent that performs well in the target domain. In this work, we investigate methods to address this need. We begin by clearly separating two uses of proxy domains that are often conflated: 1) their ability to be a faithful predictor of agent performance and 2) their ability to be a useful tool for learning. In this paper, we attempt to clarify the role of proxy domains and establish new proxy usefulness (PU) metrics to compare the usefulness of different proxy domains. We propose the relative predictive PU to assess the predictive ability of a proxy domain and the learning PU to quantify the usefulness of a proxy as a tool to generate learning data. Furthermore, we argue that the value of a proxy is conditioned on the task that it is being used to help solve. We demonstrate how these new metrics can be used to optimize parameters of the proxy domain for which obtaining ground truth via system identification is not trivial.

We consider the problem of co-designing embodied intelligence as a whole in a structured way, from hardware components such as propulsion systems and sensors to software modules such as control and perception pipelines. We propose a principled approach to formulate and solve complex embodied intelligence co-design problems, leveraging a monotone co-design theory. The methods we propose are intuitive and integrate heterogeneous engineering disciplines, allowing analytical and simulation-based modeling techniques and enabling interdisciplinarity. We illustrate through a case study how, given a set of desired behaviors, our framework is able to compute Pareto efficient solutions for the entire hardware and software stack of a self-driving vehicle.

Plastic neural networks have the ability to adapt to new tasks. However, in a continual learning setting, the configuration of parameters learned in previous tasks can severely reduce the adaptability to future tasks. In particular, we show that, when using weight decay, weights in successive layers of a deep network may become "mutually frozen". This has a double effect: on the one hand, it makes the network updates more invariant to nuisance factors, providing a useful bias for future tasks. On the other hand, it can prevent the network from learning new tasks that require significantly different features. In this context, we find that the local input sensitivity of a deep model is correlated with its ability to adapt, thus leading to an intriguing trade-off between adaptability and invariance when training a deep model more than once. We then show that a simple intervention that "resets" the mutually frozen connections can improve transfer learning on a variety of visual classification tasks. The efficacy of "resetting" itself depends on the size of the target dataset and the difference of the pre-training and target domains, allowing us to achieve state-of-the-art results on some datasets.

As robotics matures and increases in complexity, it is more necessary than ever that robot autonomy research be reproducible. Compared to other sciences, there are specific challenges to benchmarking autonomy, such as the complexity of the software stacks, the variability of the hardware and the reliance on data-driven techniques, amongst others. In this paper, we describe a new concept for reproducible robotics research that integrates development and benchmarking, so that reproducibility is obtained "by design" from the beginning of the research/development processes. We first provide the overall conceptual objectives to achieve this goal and then a concrete instance that we have built: the DUCKIENet. One of the central components of this setup is the Duckietown Autolab, a remotely accessible standardized setup that is itself also relatively low-cost and reproducible. When evaluating agents, careful definition of interfaces allows users to choose among local versus remote evaluation using simulation, logs, or remote automated hardware setups. We validate the system by analyzing the repeatability of experiments conducted using the infrastructure and show that there is low variance across different robot hardware and across different remote labs.†

The behavior of self-driving cars must be compatible with an enormous set of conflicting and ambiguous objectives, from law, from ethics, from the local culture, and so on. This paper describes a new way to conveniently define the desired behavior for autonomous agents, which we use on the self-driving cars developed at nuTonomy, an Aptiv company. We define a “rulebook” as a pre-ordered set of “rules”, each akin to a violation metric on the possible outcomes (“realizations”). The rules are partially ordered by priority. The semantics of a rulebook imposes a pre-order on the set of realizations. We study the compositional properties of the rulebooks, and we derive which operations we can allow on the rulebooks to preserve previously-introduced constraints. While we demonstrate the application of these techniques in the self-driving domain, the methods are domain-independent.

One of the challenges of developing autonomous vehicles is planning in an inhabited environment under sensing uncertainty as well as limited perception and computational resources. Besides reasoning about the behaviour of traffic participants that are within the vehicles' field of view, safe autonomous driving also requires the vehicle to reason about possible traffic participants that might exist beyond its sensing horizon, and to adapt its driving behaviour accordingly. This paper describes an inference and motion planning pipeline that is able to guarantee passive safety (collisions are possible, but the autonomous vehicle will be at rest) with respect to hypothetical hidden agents that have not been observed yet. We also incorporate the vehicle's reaction time due to sensing and computational delays into the planning process; for example, we show how having a fast reaction time due to the availability of more computational resources leads to more aggressive trajectories, while a car with a larger reaction time will choose more relaxed trajectories that require less attention.

Typically, to enlarge the operating domain of an object detector, more labeled training data is required. We describe a method called wormhole learning, which allows to extend the operating domain without additional data, but only with temporary access to an auxiliary sensor with certain invariance properties. We describe the instantiation of this principle with a regular visible-light RGB camera as the main sensor, and an infrared sensor as the temporary sensor. We start with a pre-trained RGB detector; then we train the infrared detector based on the RGB-inferred labels; finally we re-train the RGB detector based on the infrared-inferred labels. After these two transfer-learning steps, the RGB detector has enlarged its operating domain by inheriting part of the invariance to illumination of the infrared sensor; in particular, the RGB detector is now able to see much better at night. We analyze the wormhole learning phenomenon by bounding the possible gain in accuracy using mutual information properties of the two sensors and considered operating domain.

Duckiepond is an education and research development environment that includes software systems, educational materials, and of a fleet of autonomous surface vehicles Duckieboat. Duckieboats are designed to be easily reproducible with parts from a 3D printer and other commercially available parts, with flexible software that leverages several open source packages. The Duckiepond environment is modeled after Duckietown and AI Driving Olympics environments: Duckieboats rely only on one monocular camera, IMU, and GPS, and perform all ML processing using onboard embedded computers. Duckiepond coordinates commonly used middlewares (ROS and MOOS) and containerized software packages in Docker, making it easy to deploy. The combination of learning-based methods together with classic methods enables important maritime missions: track and trail, navigation, and coordinate among Duckieboats to avoid collisions. Duckieboats have been operating in a man-made lake, reservoir and river environments. All software, hardware, and educational materials are openly available (https://robotx-nctu.github.io/duckiepond), with the goal of supporting research and education communities across related domains.

Duckietown is an open, inexpensive and flexible platform for autonomy education and research. The platform comprises small autonomous vehicles (“Duckiebots”) built from off-the-shelf components, and cities (“Duckietowns”) complete with roads, signage, traffic lights, obstacles, and citizens (duckies) in need of transportation. The Duckietown platform offers a wide range of functionalities at a low cost. Duckiebots sense the world with only one monocular camera and perform all processing onboard with a Raspberry Pi 2, yet are able to: follow lanes while avoiding obstacles, pedestrians (duckies) and other Duckiebots, localize within a global map, navigate a city, and coordinate with other Duckiebots to avoid collisions. Duckietown is a useful tool since educators and researchers can save money and time by not having to develop all of the necessary supporting infrastructure and capabilities. All materials are available as open source, and the hope is that others in the community will adopt the platform for education and research.

There is considerable freedom in choosing the sensors to be equipped on a robot. Currently many sensing technologies are available (radar, lidar, vision sensors, time-of-flight cameras, etc.). For each class, there are additional choices regarding the exact sensor parameters (spatial resolution, frame rate, etc.). Which sensor is best? In general, this question needs to be qualified. It depends on the task. In an estimation task, the answer depends on the prior for the signal. In a control task, the answer depends exactly on which are the sufficient statistics for computing the control signal. This paper shows that an ulterior qualification that needs to be made: the answer depends on the power available for sensing, even when the task is fixed. We define the “power-performance” curve as the performance attainable on a task for a given level of sensing power. We show that this approach is well suited to comparing a traditional CMOS sensor with the recently available “neuromorphic” sensors. We discuss estimation tasks with different priors for the signal. We find priors for which one sensor dominates the other and vice-versa, priors for which they are equivalent, and priors for which the answer depends on the power available. This shows that comparing sensors is a quite delicate problem. It also suggests that the optimal architecture might have more that one sensor, and would switch sensors on and off according to the performance level required instantaneously.

The agility of a robotic system is ultimately limited by the speed of its processing pipeline. The use of a Dynamic Vision Sensors (DVS), a sensor producing asynchronous events as luminance changes are perceived by its pixels, makes it possible to have a sensing pipeline of a theoretical latency of a few microseconds. However, several challenges must be overcome: a DVS does not provide the grayscale value but only changes in the luminance; and because the output is composed by a sequence of events, traditional frame-based visual odometry methods are not applicable. This paper presents the first visual odometry system based on a DVS plus a normal CMOS camera to provide the absolute brightness values. The two sources of data are automatically spatiotemporally calibrated from logs taken during normal operation. We design a visual odometry method that uses the DVS events to estimate the relative displacement since the previous CMOS frame by processing each event individually. Experiments show that the rotation can be estimated with surprising accuracy, while the translation can be estimated only very noisily, because it produces few events due to very small apparent motion.

Pose graph optimization is an elegant and efficient formulation for robot localization and mapping. Experimental evidence suggests that, in real problems, the set of measurements used to estimate robot poses is prone to contain outliers, due to perceptual aliasing and incorrect data association. While several related works deal with the rejection of outliers during pose estimation, the goal of this paper is to propose a grounded strategy for measurements selection, i.e., the output of our approach is a set of “reliable” measurements, rather than pose estimates. Because the classification in inliers/outliers is not observable in general, we pose the problem as finding the maximal subset of the measurements that is internally coherent. In the linear case, we show that the selection of the maximal coherent set can be (conservatively) relaxed to obtain a linear programming problem with ℓ1objective. We show that this approach can be extended to (nonlinear) planar pose graph optimization using similar ideas as our previous work on linear approaches to pose graph optimization. We evaluate our method on standard datasets, and we show that it is robust to a large number of outliers and different outlier generation models, while entailing the advantages of linear programming (fast computation, scalability).

Pose graph optimization from relative measurements is challenging because of the angular component of the poses: the variables live on a manifold product with nontrivial topology and the likelihood function is nonconvex and has many local minima. Because of these issues, iterative solvers are not robust to large amounts of noise. This paper describes a global estimation method, called multi-hypothesis orientation-from-lattice estimation in 2-D (MOIE2D), for the estimation of the nodes' orientation in a pose graph. We demonstrate that the original nonlinear optimization problem on the manifold product is equivalent to an unconstrained quadratic optimization problem on the integer lattice. Exploiting this insight, we show that, in general, the maximum likelihood estimate alone cannot be considered a reliable estimator. Therefore, MOIE2D returns a set of point estimates, for which we can derive precise probabilistic guarantees. Experiments show that the method is able to tolerate extreme amounts of noise, far above all noise levels of sensors used in applications. Using MOIE2D's output to bootstrap the initial guess of iterative pose graph optimization methods improves their robustness and makes them avoid local minima even for high levels of noise.

We consider the problem of planning motions in observations space, based on learned models of the dynamics that associate to each action a diffeomorphism of the observations domain. For an arbitrary set of diffeomorphisms, this problem must be formulated as a generic search problem. We adapt established algorithms of the graph search family. In this scenario, node expansion is very costly, as each node in the graph is associated to an uncertain diffeomorphism and corresponding predicted observations. We describe several improvements that ameliorate performance: the introduction of better image similarities to use as heuristics; a method to reduce the number of expanded nodes by preliminarily identifying redundant plans; and a method to pre-compute composite actions that make the search efficient in all directions.

At the current state of the art, the agility of an autonomous flying robot is limited by its sensing pipeline, because the relatively high latency and low sampling frequency limit the aggressiveness of the control strategies that can be implemented. To obtain more agile robots, we need faster sensing pipelines. A Dynamic Vision Sensor (DVS) is a very different sensor than a normal CMOS camera: rather than providing discrete frames like a CMOS camera, the sensor output is a sequence of asynchronous timestamped events each describing a change in the perceived brightness at a single pixel. The latency of such sensors can be measured in the microseconds, thus offering the theoretical possibility of creating a sensing pipeline whose latency is negligible compared to the dynamics of the platform. However, to use these sensors we must rethink the way we interpret visual data. This paper presents a method for low-latency pose tracking using a DVS and Active Led Markers (ALMs), which are LEDs blinking at high frequency (>1 KHz). The sensor's time resolution allows distinguishing different frequencies, thus avoiding the need for data association. This approach is compared to traditional pose tracking based on a CMOS camera. The DVS performance is not affected by fast motion, unlike the CMOS camera, which suffers from motion blur.

Diffeomorphisms dynamical systems are dynamical systems for which the state is an image and each command induce a diffeomorphism of the state. These systems can approximate the dynamics of robotic sensorimotor cascades well enough to be used for problems such as planning in observations space. Learning of an arbitrary diffeomorphism from pairs of images is an extremely high dimensional problem. This paper describes two improvements to the methods presented in previous work. The previous method had required O(ρ4) memory as a function of the desired resolution ρ, which, in practice, was the main limitation to the resolution of the diffeomorphisms that could be learned. This paper describes an algorithm based on recursive refinement that lowers the memory requirement to O(ρ2). Another improvement regards the estimation the diffeomorphism uncertainty, which is used to represent the sensor's limited field of view; the improved method obtains a more accurate estimation of the uncertainty by checking the consistency of a learned diffeomorphism and its independently learned inverse. The methods are tested on two robotic systems (a pan-tilt camera and a 5-DOF manipulator).

This paper presents a new intrinsic calibration method that allows us to calibrate a generic single-view point camera just by waving it around. From the video sequence obtained while the camera undergoes random motion, we compute the pairwise time correlation of the luminance signal for a subset of the pixels. We show that if the camera undergoes a random uniform motion, then the pairwise correlation of any pixels pair is a function of the distance between the pixel directions on the visual sphere. This leads to formalizing calibration as a problem of metric embedding from nonmetric measurements: We want to find the disposition of pixels on the visual sphere from similarities that are an unknown function of the distances. This problem is a generalization of multidimensional scaling (MDS) that has so far resisted a comprehensive observability analysis (can we reconstruct a metrically accurate embedding?) and a solid generic solution (how do we do so?). We show that the observability depends both on the local geometric properties (curvature) as well as on the global topological properties (connectedness) of the target manifold. We show that, in contrast to the euclidean case, on the sphere we can recover the scale of the points distribution, therefore obtaining a metrically accurate solution from nonmetric measurements. We describe an algorithm that is robust across manifolds and can recover a metrically accurate solution when the metric information is observable. We demonstrate the performance of the algorithm for several cameras (pin-hole, fish-eye, omnidirectional), and we obtain results comparable to calibration using classical methods. Additional synthetic benchmarks show that the algorithm performs as theoretically predicted for all corner cases of the observability analysis.

As animals move through the world in search of resources, they change course in reaction to both external sensory cues and internally-generated programs. Elucidating the functional logic of complex search algorithms is challenging because the observable actions of the animal cannot be unambiguously assigned to externally- or internally-triggered events. We present a technique that addresses this challenge by assessing quantitatively the contribution of external stimuli and internal processes. We apply this technique to the analysis of rapid turns ("saccades") of freely flying Drosophila melanogaster. We show that a single scalar feature computed from the visual stimulus experienced by the animal is sufficient to explain a majority (93%) of the turning decisions. We automatically estimate this scalar value from the observable trajectory, without any assumption regarding the sensory processing. A posteriori, we show that the estimated feature field is consistent with previous results measured in other experimental conditions. The remaining turning decisions, not explained by this feature of the visual input, may be attributed to a combination of deterministic processes based on unobservable internal states and purely stochastic behavior. We cannot distinguish these contributions using external observations alone, but we are able to provide a quantitative bound of their relative importance with respect to stimulus-triggered decisions. Our results suggest that comparatively few saccades in free-flying conditions are a result of an intrinsic spontaneous process, contrary to previous suggestions. We discuss how this technique could be generalized for use in other systems and employed as a tool for classifying effects into sensory, decision, and motor categories when used to analyze data from genetic behavioral screens.

Consider a differential-drive mobile robot equipped with an on-board exteroceptive sensor that can estimate its own motion, e.g., a range-finder. Calibration of this robot involves estimating six parameters: three for the odometry (radii and distance between the wheels) and three for the pose of the sensor with respect to the robot. After analyzing the observability of this problem, this paper describes a method for calibrating all parameters at the same time, without the need for external sensors or devices, using only the measurement of the wheel velocities and the data from the exteroceptive sensor. The method does not require the robot to move along particular trajectories. Simultaneous calibration is formulated as a maximum-likelihood problem and the solution is found in a closed form. Experimental results show that the accuracy of the proposed calibration method is very close to the attainable limit given by the Cramér-Rao bound.

One of the challenges in designing the next generation of robots operating in non-engineered environments is that there seems to be an infinite amount of causes that make the sensor data unreliable or actuators ineffective. In this paper, we discuss what faults are possible to detect using zero modeling effort: we start from uninterpreted streams of observations and commands, and without a prior knowledge of a model of the world. We show that in sensorimotor cascades it is possible to define static faults independently of a nominal model. We define an information-theoretic usefulness of a sensor reading and we show that it captures several kind of sensorimotor faults frequently encountered in practice. We particularize these ideas to models proposed in previous work as suitable candidates for describing generic sensorimotor cascades. We show several examples with camera and range-finder data, and we discuss a possible way to integrate these techniques in an existing robot software architecture.

The problem of bootstrapping consists in designing agents that can learn from scratch the model of their sensorimotor cascade (the series of robot actuators, the external world, and the robot sensors) and use it to achieve useful tasks. In principle, we would want to design agents that can work for any robot dynamics and any robot sensor(s). One of the difficulties of this problem is the fact that the observations are very high dimensional, the dynamics is nonlinear, and there is a wide range of “representation nuisances” to which we would want the agent to be robust. In this paper, we model the dynamics of sensorimotor cascades using diffeomorphisms of the sensel space. We show that this model captures the dynamics of camera and range-finder data, that it can be used for long-term predictions, and that it can capture nonlinear phenomena such as a limited field of view. Moreover, by analyzing the learned diffeomorphisms it is possible to recover the “linear structure” of the dynamics independently of the commands representation.

We consider the bootstrapping problem, which consists in learning a model of the agent's sensors and actuators starting from zero prior information, and we take the problem of servoing as a cross-modal task to validate the learned models. We study the class of sensors with bilinear dynamics, for which the derivative of the observations is a bilinear form of the control commands and the observations themselves. This class of models is simple, yet general enough to represent the main phenomena of three representative sensors (field sampler, camera, and range-finder), apparently very different from one another. It also allows a bootstrapping algorithm based on Hebbian learning, and a simple bioplausible control strategy. The convergence properties of learning and control are demonstrated with extensive simulations and by analytical arguments.

The bootstrapping problem consists in designing agents that learn a model of themselves and the world, and utilize it to achieve useful tasks. It is different from other learning problems as the agent starts with uninterpreted observations and commands, and with minimal prior information about the world. in this paper, we give a mathematical formalization of this aspect of the problem. We argue that the vague constrain of having “no prior information” can be recast as a precise algebraic condition on the agent: that its behavior is invariant to particular classes of nuisances on the world, which we show can be well represented by actions of groups (diffeomorphisms, permutations, linear transformations) on observations and commans. We then introduce the class of bilinear gradient dynamics sensors (BGDS) as a candidate for learning generic robotic sensorimotor cascades. We show how framing the problem as rejection of group nuisances allows a compact and modular analysis of typical preprocessing stages, such as learning the topology of the sensors. We demonstrate learning and using such models on real-word range-finder and camera date from publicly available datasets.

This paper presents a new method to estimate the relative motion of a vehicle from images of a single camera. The biggest problem in visual motion estimation is data association; matched points contain many outliers that must be detected and removed so that the motion can be estimated accurately. A very established method for robust motion estimation in the presence of outliers is the five-point RANSAC algorithm. Five-point RANSAC operates by generating motion hypotheses from randomly-sampled minimal sets of five-point correspondences. These hypotheses are then tested against all data points and the motion hypothesis that after a given number of iterations returns the largest number of inliers is taken as the solution to the problem. A typical drawback of RANSAC is that the number of iterations required to find a suitable solution grows exponentially with the number of outliers, often requiring thousands of iterations for typical data from urban environments. Another problem is that - due to its random nature - sometimes the found solution is not the “best” solution to the motion estimation problem. In this paper, we describe an algorithm for relative motion estimation in the presence of outliers, which does not rely on RANSAC. Contrary to RANSAC, motion hypotheses are not generated from randomly-sampled point correspondences, but from a “proposal distribution” that is built by exploiting the vehicle non-holonomic constraints. We show that not only is the proposed algorithm significantly faster than RANSAC, but that the returned solution may also be better in that it favors the underlying motion model of the vehicle, thus overcoming the typical limitations of RANSAC. Additionally, the proposed algorithm provides the likelihood of the motion estimate, which can be very useful in all those applications where a probability distribution of the position of the vehicle is required (e.g., SLAM). Finally, the performance of the proposed method is compared to that of the standard five-point RANSAC on real images collected from a vehicle moving in a cluttered, urban environment.

We consider the problem of purely visual pose stabilization (also known as servoing) of a second-order rigid-body system with six degrees of freedom: how to choose forces and torques, based on the current view and a memorized goal image, to steer the pose towards a desired one. Emphasis has been given to the bio-plausibility of the computation, in the sense that the control laws could be in principle implemented on the neural substrate of simple insects. We show that stabilizing laws can be realized by bilinear/quadratic operations on the visual input. This particular computational structure has several numerically favorable characteristics (sparse, local, and parallel), and thus permits an efficient engineering implementation. We show results of the control law tested on an indoor helicopter platform.

This paper presents Cramer-Rao bound-like inequalities for pose tracking, which is defined as the problem of recovering the robot displacement given two successive readings of a relative sensor. Computing the exact Fisher Information Matrix (FIM) for pose tracking is hard, because the state comprises the map, which is infinite-dimensional and unknown. This paper shows that the FIM for pose tracking can be bounded by a function of the FIM for localization on a known map, thereby reducing the analysis to a finite-dimensional problem. The resulting bounds are independent of the map prior and representation. The results are valid for any relative sensor; the experimental verification is done for the particular case of pose tracking using range-finders (scan matching).

This paper presents HSM3D, an algorithm for global rigid 6DOF alignment of 3D point clouds. The algorithm works by projecting the two input sets into the Radon/Hough domain, whose properties allow to decompose the 6DOF search into a series of fast one-dimensional cross-correlations. No planes or other particular features must be present in the input data, and the algorithm is provably complete in the case of noise-free input. The algorithm has been experimentally validated on publicly available data sets.

We recently introduced HSM3D, an algorithm to solve the six dimensional scan-matching problem without relying on features in the input, and whose solution does not depend on initial guesses. Building upon these new findings, in this manuscript we present a more detailed experimental study of the algorithm we proposed. In particular, we show how to improve the algorithm's performance also when matching point clouds produced by stereo cameras, given that this kind of input invalidates some of the assumptions we formerly identified in order to accelerate HSM3D's performance. We also show that by incorporating color information into the the algorithm it is possible to reduce the number of sporadic outliers in the solution set, thus providing a more reliable algorithm.

This paper describes PLICP, an ICP (iterative closest/corresponding point) variant that uses a point-to-line metric, and an exact closed-form for minimizing such metric. The resulting algorithm has some interesting properties: it converges quadratically, and in a finite number of steps. The method is validated against vanilla ICP, IDC (iterative dual correspondences), and MBICP (Metric-Based ICP) by reproducing the experiments performed in Minguez et al. (2006). The experiments suggest that PLICP is more precise, and requires less iterations. However, it is less robust to very large initial displacement errors. The last part of the paper is devoted to purely algorithmic optimization of the correspondence search; this allows for a significant speed-up of the computation. The source code is available for download.

Modeling robot motion planning with uncertainty in a Bayesian framework leads to a computationally intractable stochastic control problem. We seek hypotheses that can justify a separate implementation of control, localization and planning. In the end, we reduce the stochastic control problem to path- planning in the extended space of poses x covariances; the transitions between states are modeled through the use of the Fisher information matrix. In this framework, we consider two problems: minimizing the execution time, and minimizing the final covariance, with an upper bound on the execution time. Two correct and complete algorithms are presented. The first is the direct extension of classical graph-search algorithms in the extended space. The second one is a back-projection algorithm: uncertainty constraints are propagated backward from the goal towards the start state.

For a differential-drive mobile robot equipped with an on-board range sensor, there are six parameters to calibrate: three for the odometry (radii and distance between the wheels), and three for the pose of the sensor with respect to the robot frame. This paper describes a method for calibrating all six parameters at the same time, without the need for external sensors or devices. Moreover, it is not necessary to drive the robot along particular trajectories. The available data are the measures of the angular velocities of the wheels and the range sensor readings. The maximum-likelihood calibration solution is found in a closed form.

We present a new algorithm for solving the global localization problem called Frozen-Time Smoother (FTS). Time is 'frozen', in the sense that the belief always refers to the same time instant, instead of following a moving target, like Monte Carlo Localization does. This algorithm works in the case in which global localization is formulated as a smoothing problem, and a precise estimate of the incremental motion of the robot is usually available. These assumptions correspond to the case when global localization is used to solve the loop closing problem in SLAM. We compare FTS to two Monte Carlo methods designed with the same assumptions. The experiments suggest that a naive implementation of the FTS is more efficient than an extremely optimized equivalent Monte Carlo solution. Moreover, the FTS has an intrinsic laziness: it does not need frequent updates (scans can be integrated once every many meters) and it can process data in arbitrary order. The source code and datasets are available for download.

Intense efforts to define a common structure in robotic applications, both from a conceptual and from an implementation point of view, have been carried out in the last years and several frameworks have been realized for helping in developing robotic applications. However, due to the diversity of these applications, as well as of the research groups involved, a common framework is still far from being accepted. In this paper we focus on modularity and re-usability, as major features for robotic applications. We thus characterize existing frameworks for robot software development through the choices made on concurrent execution of modules and information sharing among them and we present OpenRDK, a modular framework focused on rapid development of distributed robotic systems. OpenRDK has been designed and developed with many years of experience following userspsila advice and has been successfully used for the development of many diverse applications with different kinds of robots. After such an extensive test, OpenRDK is now an open source project.

Existing methods for estimating the covariance of the ICP (iterative closest/corresponding point) algorithm are either inaccurate or are computationally too expensive to be used online. This paper proposes a new method, based on the analysis of the error function being minimized. It considers that the correspondences are not independent (the same measurement being used in more than one correspondence), and explicitly utilizes the covariance matrix of the measurements, which are not assumed to be independent either. The validity of the approach is verified through extensive simulations: it is more accurate than previous methods and its computational load is negligible. The ill-posedness of the surface matching problem is explicitly tackled for under-constrained situations by performing an observability analysis; in the analyzed cases the method still provides a good estimate of the error projected on the observable manifold.

The covariance of every unbiased estimator is bounded by the Cramer-Rao lower bound, which is the inverse of Fisher's information matrix. This paper shows that, for the case of localization with range-finders, Fisher's matrix is a function of the expected readings and of the orientation of the environment's surfaces at the sensed points. The matrix also offers a mathematically sound way to characterize under-constrained situations as those for which it is singular: in those cases the kernel describes the direction of maximum uncertainty. This paper also introduces a simple model of unstructured environments for which the Cramer-Rao bound is a function of two statistics of the shape of the environment: the average radius and a measure of the irregularity of the surfaces. Although this model is not valid for all environments, it allows for some interesting qualitative considerations. As an experimental validation, this paper reports simulations comparing the bound with the actual performance of the ICP (iterative closest/corresponding point) algorithm. Finally, it is discussed the difficulty in extending these results to find a lower bound for accuracy in scan matching and SLAM.

We describe an interpretation of scan matching as a probability distribution approximation problem and we propose an algorithm that, employing a particle approximation to the target distribution, can take advantage of the knowledge of the evolution model and provide an estimate of the matching uncertainty. Experiments show it can work in unstructured environments, it is reliable to severe sensor occlusions and it handles under constrained situations gracefully

Scan matching is used as a building block in many robotic applications, for localization and simultaneous localization and mapping (SLAM). Although many techniques have been proposed for scan matching in the past years, more efficient and effective scan matching procedures allow for improvements of such associated problems. In this paper we present a new scan matching method that, exploiting the properties of the Hough domain, allows for combining advantages of dense scan matching algorithms with feature-based ones.