Estimating the 3D pose of desktop objects is crucial for applications such as robotic manipulation. Many existing approaches to this problem require a depth map of the object for both training and prediction, which restricts them to opaque, lambertian objects that produce good returns in an RGBD sensor. In this paper we forgo using a depth sensor in favor of raw stereo input. We address two problems: first, we establish an easy method for capturing and labeling 3D keypoints on desktop objects with an RGB camera; and second, we develop a deep neural network, called KeyPose, that learns to accurately predict object poses using 3D keypoints, from stereo input, and works even for transparent objects. To evaluate the performance of our method, we create a dataset of 15 clear objects in five classes, with 48K 3D-keypoint labeled images. We train both instance and category models, and show generalization to new textures, poses, and objects. KeyPose surpasses state-of-the-art performance in 3D pose estimation on this dataset by factors of 1.5 to 3.5, even in cases where the competing method is provided with ground-truth depth. Stereo input is essential for this performance as it improves results compared to using monocular input by a factor of 2. We will release a public version of the data capture and labeling pipeline, the transparent object database, and the KeyPose models and evaluation code. Project website: https://sites.google.com/corp/view/keypose.

Instrumenting and collecting annotated visual grasping datasets to train modern machine learning algorithms can be extremely time-consuming and expensive. An appealing alternative is to use off-the-shelf simulators to render synthetic data for which ground-truth annotations are generated automatically. Unfortunately, models trained purely on simulated data often fail to generalize to the real world. We study how randomized simulated environments and domain adaptation methods can be extended to train a grasping system to grasp novel objects from raw monocular RGB images. We extensively evaluate our approaches with a total of more than 25,000 physical test grasps, studying a range of simulation conditions and domain adaptation methods, including a novel extension of pixel-level domain adaptation that we term the GraspGAN. We show that, by using synthetic data and domain adaptation, we are able to reduce the number of real-world samples needed to achieve a given level of performance by up to 50 times, using only randomly generated simulated objects. We also show that by using only unlabeled real-world data and our GraspGAN methodology, we obtain real-world grasping performance without any real-world labels that is similar to that achieved with 939,777 labeled real-world samples.

While Iterative Closest Point (ICP) algorithms have been successful at aligning 3D point clouds, they do not take into account constraints arising from sensor viewpoints. More recent beam-based models take into account sensor noise and viewpoint, but problems still remain. In particular, good optimization strategies are still lacking for the beam-based model. In situations of occlusion and clutter, both beam-based and ICP approaches can fail to find good solutions. In this paper, we present both an optimization method for beambased models and a novel framework for modeling observation dependencies in beam-based models using over-segmentations. This technique enables reasoning about object extents and works well in heavy clutter. We also make available a ground-truth 3D dataset for testing algorithms in this area.

We present a method for detecting 3D objects using multi-modalities. While it is generic, we demonstrate it on the combination of an image and a dense depth map which give complementary object information. It works in real-time, under heavy clutter, does not require a time consuming training stage, and can handle untextured objects. It is based on an efficient representation of templates that capture the different modalities, and we show in many experiments on commodity hardware that our approach significantly outperforms state-of-the-art methods on single modalities.

Feature matching is at the base of many computer vision problems, such as object recognition or structure from motion. Current methods rely on costly descriptors for detection and matching. In this paper, we propose a very fast binary descriptor based on BRIEF, called ORB, which is rotation invariant and resistant to noise. We demonstrate through experiments how ORB is at two orders of magnitude faster than SIFT, while performing as well in many situations. The efficiency is tested on several real-world applications, including object detection and patch-tracking on a smart phone.

We present a novel and general optimisation framework for visual SLAM, which scales for both local, highly accurate reconstruction and large-scale motion with long loop closures. We take a two-level approach that combines accurate pose-point constraints in the primary region of interest with a stabilising periphery of pose-pose soft constraints. Our algorithm automatically builds a suitable connected graph of keyposes and constraints, dynamically selects inner and outer window membership and optimises both simultaneously. We demonstrate in extensive simulation experiments that our method approaches the accuracy of offline bundle adjustment while maintaining constant-time operation, even in the hard case of very loopy monocular camera motion. Furthermore, we present a set of real experiments for various types of visual sensor and motion, including large scale SLAM with both monocular and stereo cameras, loopy local browsing with either monocular or RGB-D cameras, and dense RGB-D object model building.

We present an approach for navigation in hybrid maps consisting of a topological graph overlaid with local occupancy grids. The topological graph is built on top of a graph SLAM system, which can be efficiently optimized even for very large environments. The novel feature of our system is that it navigates locally using local metric maps, while the overall plan is formed on the topological graph. Unlike many current SLAM methods, we never reconstruct a full occupancy grid of the environment for localization or path planning. We show that our method generates near-optimal plans, and deals gracefully with changes to the map.

Many popular problems in robotics and computer vision including various types of simultaneous localization and mapping (SLAM) or bundle adjustment (BA) can be phrased as least squares optimization of an error function that can be represented by a graph. This paper describes the general structure of such problems and presents g2o, an open-source C++ framework for optimizing graph-based nonlinear error functions. Our system has been designed to be easily extensible to a wide range of problems and a new problem typically can be specified in a few lines of code. The current implementation provides solutions to several variants of SLAM and BA. We provide evaluations on a wide range of real-world and simulated datasets. The results demonstrate that while being general g2o offers a performance comparable to implementations of state of-the-art approaches for the specific problems.

In this paper we address the topic of feature extraction in 3D point cloud data for object recognition and pose identification. We present a novel interest keypoint extraction method that operates on range images generated from arbitrary 3D point clouds, which explicitly considers the borders of the objects identified by transitions from foreground to background. We furthermore present a feature descriptor that takes the same information into account. We have implemented our approach and present rigorous experiments in which we analyze the individual components with respect to their repeatability and matching capabilities and evaluate the usefulness for point feature based object detection methods.

The on-chip fabrication and manipulation of microstructures are expected to be applied for single cell analysis system such as cell manipulation and measurement tools. In this paper, we previously present a methodology for fabricating and assembling microstructures inside a microfluidic channel. By the illumination of patterned UV-ray through the mask under a microscope, microstructures with arbitrary shape are made of the photo-crosslinkable resin inside microfluidic device. The microstructures are fabricated at the desired place inside microfluidic channel and manipulated by optical tweezers. Based on the technique which can manipulate multiple points simultaneously by high-speed scanning of a single laser with galvanometer mirror, a rotational microstructure made of a microgear and a rotation axis is assembled and rotated. We also report two methods of solution replacement inside microfluidic channel which reduces viscosity of solvent in order to improve manipulation performance. By adjusting the concentration of photo-crosslinkable resin and replacing solution components, the viscosity of solvent inside channel can be changed. The manipulation speed of the rotational microstructure increases when the viscosity of solvent decreases, because the viscosity resistance for the movement of microstructure is weaker inside lower viscosity solvent. We fabricate rotational microstructures inside lower viscosity solvent and evaluate the movement efficiency compared with microstructures inside former high viscosity solvent.

Sparse Bundle Adjustment (SBA) is a method for simultaneously optimizing a set of camera poses and visible points. It exploits the sparse primary structure of the problem, where connections exist just between points and cameras. In this paper, we implement an efficient version of SBA for systems where the secondary structure (relations among cameras) is also sparse. The method, which we call Sparse SBA (sSBA), integrates an efficient method for setting up the linear subproblem with recent advances in direct sparse Cholesky solvers. sSBA outperforms the current SBA standard implementation on datasets with sparse secondary structure by at least an order of magnitude, while also being more efficient on dense datasets.

Passive stereo vision is widely used as a range sensing technology in robots, but suffers from dropouts: areas of low texture where stereo matching fails. By supplementing a stereo system with a strong texture projector, dropouts can be eliminated or reduced. This paper develops a practical stereo projector system, first by finding good patterns to project in the ideal case, then by analyzing the effects of system blur and phase noise on these patterns, and finally by designing a compact projector that is capable of good performance out to 3m in indoor scenes. The system has been implemented and has excellent depth precision and resolution, especially in the range out to 1.5m.

This paper describes a navigation system that allowed a robot to complete 26.2 miles of autonomous navigation in a real office environment. We present the methods required to achieve this level of robustness, including an efficient Voxel-based 3D mapping algorithm that explicitly models unknown space. We also provide an open-source implementation of the algorithms used, as well as simulated environments in which our results can be verified.

We describe an autonomous robotic system capable of navigating through an office environment, opening doors along the way, and plugging itself into electrical outlets to recharge as needed. We demonstrate through extensive experimentation that our robot executes these tasks reliably, without requiring any modification to the environment. We present robust detection algorithms for doors, door handles, and electrical plugs and sockets, combining vision and laser sensors. We show how to overcome the unavoidable shortcoming of perception by integrating compliant control into manipulation motions. We present a visual-differencing approach to high-precision plug-insertion that avoids the need for high-precision hand-eye calibration.

Service robots deployed in domestic environments generally need the capability to deal with articulated objects such as doors and drawers in order to fulfill certain mobile manipulation tasks. This however, requires, that the robots are able to perceive the articulation models of such objects. In this paper, we present an approach for detecting, tracking, and learning articulation models for cabinet doors and drawers without using artificial markers. Our approach uses a highly efficient and sampling-based approach to rectangle detection in depth images obtained from a self-developed active stereo system. The robot can use the generative models learned for the articulated objects to estimate their articulation type, their current configuration, and to make predictions about possible configurations not observed before. We present experiments carried out on real data obtained from our active stereo system. The results demonstrate that our technique is able to learn accurate articulation models. We furthermore provide a detailed error analysis based on ground truth data obtained in a motion capturing studio.

Pose graphs have become a popular representation for solving the simultaneous localization and mapping (SLAM) problem. A pose graph is a set of robot poses connected by nonlinear constraints obtained from observations of features common to nearby poses. Optimizing large pose graphs has been a bottleneck for mobile robots, since the computation time of direct nonlinear optimization can grow cubically with the size of the graph. In this paper, we propose an efficient method for constructing and solving the linear subproblem, which is the bottleneck of these direct methods. We compare our method, called Sparse Pose Adjustment (SPA), with competing indirect methods, and show that it outperforms them in terms of convergence speed and accuracy. We demonstrate its effectiveness on a large set of indoor real-world maps, and a very large simulated dataset. Open-source implementations in C++, and the datasets, are publicly available.

Prominent feature point descriptors such as SIFT and SURF allow reliable real-time matching but at a computational cost that limits the number of points that can be handled on PCs, and even more on less powerful mobile devices. A recently proposed technique that relies on statistical classification to compute signatures has the potential to be much faster but at the cost of using very large amounts of memory, which makes it impractical for implementation on low-memory devices. In this paper, we show that we can exploit the sparseness of these signatures to compact them, speed up the computation, and drastically reduce memory usage. We base our approach on Compressive Sensing theory. We also highlight its effectiveness by incorporating it into two very different SLAM packages and demonstrating substantial performance increases.

The typical SLAM mapping system assumes a static environment and constructs a map that is then used without regard for ongoing changes. Most SLAM systems, such as FastSLAM, also require a single connected run to create a map. In this paper we present a system of visual mapping, using only input from a stereo camera, that continually updates an optimized metric map in large indoor spaces with movable objects: people, furniture, partitions, etc. The system can be stopped and restarted at arbitrary disconnected points, is robust to occlusion and localization failures, and efficiently maintains alternative views of a dynamic environment. It operates completely online at a 30 Hz frame rate.

Many indoor robotics systems use laser rangeflnders as their primary sensor for mapping, localization, and obstacle avoidance. The cost and power of such systems is a major roadblock to the deployment of low-cost, efficient consumer robot platforms for home use. In this paper, we describe a compact, planar laser distance sensor (LDS) that has capabilities comparable to current laser scanners: 3 cm accuracy out to 6 m, 10 Hz acquisition, and 1 degree resolution over a full 360 degree scan. The build cost of this device, using COTS electronics and custom mechanical tooling, is under $30.

We present a fast integrated approach for online segmentation of images for outdoor robots. A compact color and texture descriptor has been developed to describe local color and texture variations in an image. This descriptor is then used in a two-stage fast clustering framework using K-means to perform online segmentation of natural images. We present results of applying our descriptor for segmenting a synthetic image and compare it against other state-of-the-art descriptors. We also apply our segmentation algorithm to the task of detecting natural paths in outdoor images. The whole system has been demonstrated to work online alongside localization, 3D obstacle detection, and planning.

Many successful indoor mapping techniques employ frame-to-frame matching of laser scans to produce detailed local maps as well as the closing of large loops. In this paper, we propose a framework for applying the same techniques to visual imagery. We match visual frames with large numbers of point features, using classic bundle adjustment techniques from computational vision, but we keep only relative frame pose information (askeleton). The skeleton is a reduced nonlinear system that is a faithful approximation of the larger system and can be used to solve large loop closures quickly, as well as forming a backbone for data association and local registration. We illustrate the workings of the system with large outdoor datasets (10 km), showing large-scale loop closure and precise localization in real time.

Many successful indoor mapping techniques employ frame-to-frame matching of laser scans to produce detailed local maps, as well as closing large loops. In this paper, we propose a framework for applying the same techniques to visual imagery, matching visual frames with large numbers of point features. The relationship between frames is kept as a nonlinear measurement, and can be used to solve large loop closures quickly. Both monocular (bearing-only) and binocular vision can be used to generate matches. Other advantages of our system are that no special landmark initialization is required, and large loops can be solved very quickly.

We consider the problem of autonomous navigation in unstructured outdoor terrains using vision sensors. The goal is for a robot to come into a new environment, map it and move to a given goal at modest speeds (1 m/sec). The biggest challenges are in building good maps and keeping the robot well localized as it advances towards the goal. In this paper, we concentrate on showing how it is possible to build a consistent, globally correct map in real time, using efficient precise stereo algorithms for map making and visual odometry for localization. While we have made advances in both localization and mapping using stereo vision, it is the integration of the techniques that is the biggest contribution of the research. The validity of our approach is tested in blind experiments, where we submit our code to an independent testing group that runs and validates it on an outdoor robot

Efficient exploration of unknown environments is a fundamental problem in mobile robotics. We present an approach to distributed multirobot mapping and exploration. Our system enables teams of robots to efficiently explore environments from different, unknown locations. In order to ensure consistency when combining their data into shared maps, the robots actively seek to verify their relative locations. Using shared maps, they coordinate their exploration strategies to maximize the efficiency of exploration. This system was evaluated under extremely realistic real-world conditions. An outside evaluation team found the system to be highly efficient and robust. The maps generated by our approach are consistently more accurate than those generated by manually measuring the locations and extensions of rooms and objects

The two dominant forms of SLAM are based on Extended Kalman Filtering and Consistent Pose Estimation. We show that these are particular subsets of a more general view of the SLAM problem, in which variables representing all robot poses and features are kept. The general technique of variable reduction is a unifying view of these methods that is mathematically sound, and which enables us to explore other interesting and computationally compelling forms for solving SLAM problems.

An important assumption underlying virtually all approaches to multi-robot exploration is prior knowledge about their relative locations. This is due to the fact that robots need to merge their maps so as to coordinate their exploration strategies. The key step in map merging is to estimate the relative locations of the individual robots. This paper presents a novel approach to multi-robot map merging under global uncertainty about the robot's relative locations. Our approach uses an adapted version of particle filters to estimate the position of one robot in the other robot's partial map. The risk of false-positive map matches is avoided by verifying match hypotheses using a rendezvous approach. We show how to seamlessly integrate this approach into a decision-theoretic multi-robot coordination strategy. The experiments show that our sample-based technique can reliably find good hypotheses for map matches. Furthermore, we present results obtained with two robots successfully merging their maps using the decision-theoretic rendezvous strategy.

A set of robots mapping an area can potentially combine their information to produce a distributed map more efficiently than a single robot alone. We describe a general framework for distributed map building in the presence of uncertain communication. Within this framework, we then present a technical solution to the key decision problem of determining relative location within partial maps.

Panoramic cameras, which image a very large field of view, are useful devices for mobile robots that must move rapidly and securely in their environments. Recent panoramic cameras present a very wide field of view from a single viewpoint. A single viewpoint is useful in mobile robotics for a number of reasons, including perspective reprojection and stereo analysis. However, the requirement of single viewpoint for panoramic cameras restricts the optical and geometrical design of these devices. In this paper we present a method for approximating a single viewpoint in panoramic devices that allows much greater freedom in design. We illustrate the method with a compact catadioptric device using a spherical mirror and standard optics, and apply it to perspective reprojection. The resultant panoramic camera has been integrated as a surveillance device on a small mobile robot.

We have assembled programmed, and demonstrated a robot-mounted motion-detection radar suitable for through-wall operation. Our radar, which employs of pulse-Doppler techniques, is designed to look through building walls and locate moving targets. The radar's signal-processing algorithms use both time-domain and frequency-domain clues to classify detected motion as arising from: (1) the ambient background; (2) mechanical motion; or (3) human activity. In the case of human motion, our routines also attempt to identify the occupant's particular activity, e.g., resting, walking, talking.

Despite many decades of research into mobile robot control, reliable, high-speed motion in complicated, uncertain environments remains an unachieved goal. In this paper we present a solution to realtime motion control that can competently maneuver a robot at optimal speed even as it explores a new region or encounters new obstacles. The method uses a navigation function to generate a gradient field that represents the optimal (lowest-cost) path to the goal at every point in the workspace. Additionally, we present an integrated sensor fusion system that allows incremental construction of an unknown or uncertain environment. Under modest assumptions, the robot is guaranteed to get to the goal in an arbitrary static unexplored environment, as long as such a path exists. We present preliminary experiments to show that the gradient method is better than expert human controllers in both known and unknown environments.

Stereo sequences promise to be a powerful method for segmenting images for applications such as tracking human figures. We present a method of statistical background modeling for stereo sequences that improves the reliability and sensitivity of segmentation in the presence of object clutter. The dynamic version of the method, called gated background adaptation, can reliably learn background statistics in the presence of corrupting foreground motion. The method has been used with a simple head discriminator to detect and track people using a stereo head mounted on a pan/tilt platform. It runs at video rates using standard PC hardware.

Real-time or near real-time range images are available from a variety of sources.Many proposed uses for these devices require significant post-processing to extract relevant 3-D information.We propose a new method based on a generalization of contour maps that has several advantages over current methods in tasks such as terrain reconstruction and obstacle avoidance.The method is illustrated with examples from obstacle avoidance on a mobile robot.

Localization is the process of updating the pose of a robot in an environment, based on sensor readings. In this experimental study, we compare two methods for localization of indoor mobile robots: Markov localization, which uses a probability distribution across a grid of robot poses; and scan matching, which uses Kalman filtering techniques based on matching sensor scans. Both these techniques are dense matching methods, that is, they match dense sets of environment features to an a priori map. To arrive at results for a range of situations, we utilize several different types of environments, and add noise to both the dead-reckoning and the sensors. Analysis shows that, roughly, the scan-matching techniques are more efficient and accurate, but Markov localization is better able to cope with large amounts of noise. These results suggest hybrid methods that are efficient, accurate and robust to noise.

Mobile robots, if they are to perform useful tasks and become accepted in open environments, must be fully autonomous. Autonomy has many different aspects ; here the focus is on three central ones: the ability to attend to another agent, to take advice about the environment, and to carry out assigned tasks. All three involve complex sensing and planning operations on the part of the robot, including the use of visual tracking of humans, coordination of motor controls, and planning. It is shown how these capabilities are integrated in the Saphira architecture, using the concepts of coordination of behaviour, coherence of modelling, and communication with other agents.

Reasoning about defaults—implications that typically hold, but which may have exceptions—is an important part of commonsense reasoning. We present some parts of a theory of defaults, concentrating on distinctions between various subtle ways in which defaults can be defeated, and on inferences which seem plausible but which are not correct in all cases. To represent this theory in a formal system, it is natural to use the epistemic concept of self‐belief. We show how to express the theory by a local translation into autoepistemic logic, which contains the requisite epistemic operators.

A prerequisite to a theory of the way agents understand speech acts is a theory of how their beliefs and intentions are revised as a consequence of events. This process of attitude revision is an interesting domain for the application of nonmonotonic reasoning because speech acts have a conventional aspect that is readily represented by defaults, but that interacts with an agent's beliefs and intentions in many complex ways that may override the defaults. Perrault has developed a theory of speech acts, based on Rieter's default logic, that captures the conventional aspect; it does not, however, adequately account for certain easily observed facts about attitude revision resulting from speech acts. A natural theory of attitude revision seems to require a method of stating preferences among competing defaults. We present here a speech act theory, formalized in hierarchic autoepistemic logic (a refinement of Moore's autoepistemic logic), in which revision of both the speaker's and hearer's attitudes can be adequately described. As a collateral benefit, efficient automatic reasoning methods for the formalism exist. The theory has been implemented and is now being employed by an utterance-planning system.

Default logic is a formal means of reasoning about defaults: what normally is the case, in the absence of contradicting information. Autoepistemic logic, on the other hand, is meant to describe the consequences of reasoning about ignorance: what must be true if a certain fact is not known. Although the motivation and formal character of these two systems are different, a closer analysis shows that they share a common trait, which is the indexical nature of certain elements in the theory. In this paper we compare the expressive power of the two systems. First, we give an effective translation of default logic into autoepistemic logic; default theories can thus be embedded into autoepistemic logic. We also present a more surprising result: the reverse translation is also possible, so that every set of sentences in autoepistemic logic can be effectively rewritten as a default theory. The formal equivalence of these two differing systems is thus established. This analysis gives an interpretive semantics to default logic, and yields insight into the nature of defaults in autoepistemic reasoning.

Americanae nace como un proyecto conjunto que surge dentro de la Red Europea de Información y Documentación sobre América Latina (REDIAL), y que ha afrontado la Biblioteca de la Agencia Española de Cooperación Internacional para el Desarrollo (AECID). Esta nueva biblioteca virtual hace más accesibles los libros digitales de tema americanista a los investigadores y usuarios interesados de cualquier parte del mundo.