Compared to polynomial splines, rational functions are known to be more efficient and well-behaved data fitting models. However, due to the potential presence of zeros in their denominator, rational functions tend to yield notoriously hard optimization problems. In this work, we present a novel least squares method for 6D pose prediction that employs rational regression. Our method can accommodate fixed data points and is able to circumvent the occurrence of zeros for rational quadratic interpolants. We demonstrate the suitability of rational quadratics for pose prediction by applying our approach to real data from the feature tracking stage of a real-time visual SLAM system and showing that it yields far more stable predictions when compared to state-of-the-art rational and polynomial spline methods.

This paper presents the first absolute pose estimation approach tailored to Light Field cameras. It builds on the observation that the ratio between the disparity arising in different sub-aperture images and their corresponding baseline is constant. Hence, we augment the 2D pixel coordinates with the corresponding normalised disparity to obtain the Light Field feature. This new representation reduces the effect of noise by aggregating multiple projections and allows for linear estimation of the absolute pose of a Light Field camera using the well-known Direct Linear Transformation algorithm. We evaluate the resulting absolute pose estimates with extensive simulations and experiments involving real Light Field datasets, demonstrating the competitive performance of our linear approach. Furthermore, we integrate our approach in a state-of-the-art Light Field Structure from Motion pipeline and demonstrate accurate multi-view 3D reconstruction.

The perspective-n-point (PnP) problem is of fundamental importance in computer vision. A global optimality condition for PnP that is independent of a particular rotation parameterization was recently developed by Nakano. This paper puts forward a direct least squares, algebraic PnP solution that extends Nakano's work by combining his optimality condition with the modified Rodrigues parameters (MRPs) for parameterizing rotation. The result is a system of polynomials that is solved using the Gröbner basis approach. An MRP vector has twice the rotational range of the classical Rodrigues (i.e., Cayley) vector used by Nakano to represent rotation. The proposed solution provides strong guarantees that the full rotation singularity associated with MRPs is avoided. Furthermore, detailed experiments provide evidence that our solution attains accuracy that is indistinguishable from Nakano's Cayley-based method with a moderate increase in computational cost.

Future autonomous spacecraft rendezvous with uncooperative or unprepared objects will be enabled by vision-based navigation, which imposes great computational challenges. Targeting short duration missions in low Earth orbit, this paper develops high-performance avionics supporting custom computer vision algorithms of increased complexity for satellite pose tracking. At algorithmic level, we track 6D pose by rendering a depth image from an object mesh model and robustly matching edges detected in the depth and intensity images. At system level, we devise an architecture to exploit the structure of commercial system-on-chip FPGAs, i.e., Zynq7000, and the benefits of tightly coupling VHDL accelerators with CPU-based functions. At implementation level, we employ our custom HW/SW co-design methodology and an elaborate combination of digital circuit design techniques to optimize and map efficiently all functions to a compact embedded device. Providing significant performance per watt improvement, the resulting VBN system achieves a throughput of 10-14 FPS for 1 Mpixel images, with only 4.3 watts mean power and 1U size, while tracking ENVISAT in real-time with only 0.5% mean positional error.

This paper presents a large scale, metric Structure from Motion (SfM) pipeline for generalised cameras with overlapping fields-of-view, and demonstrates it using Light Field (LF) images. We build on recent developments in algorithms for absolute and relative pose recovery for generalised cameras and couple them with multi-view triangulation in a robust framework that advances the state-of-the-art on 3D reconstruction from LFs in several ways. First, our framework can recover the scale of a scene. Second, it is concerned with unordered sets of LF images, meticulously determining the order in which images should be considered. Third, it can scale to datasets with hundreds of LF images. Finally, it recovers 3D scene structure while abstaining from triangulating using very small baselines. Our approach outperforms the state-of-the-art, as demonstrated by real-world experiments with variable size datasets.

Increased mobile autonomy is a vital requisite for future planetary exploration rovers. Stereo vision is a key enabling technology in this regard, as it can passively reconstruct in three dimensions the surroundings of a rover and facilitate the selection of science targets and the planning of safe routes. Nonetheless, accurate dense stereo algorithms are computationally demanding. When executed on the low-performance, radiation-hardened CPUs typically installed on rovers, slow stereo processing severely limits the driving speed and hence the science that can be conducted in situ . Aiming to decrease execution time while increasing the accuracy of stereo vision embedded in future rovers, this article proposes HW/SW co-design and acceleration on resource-constrained, space-grade FPGAs. In a top-down approach, we develop a stereo algorithm based on the space sweep paradigm, design its parallel HW architecture, implement it with VHDL, and demonstrate feasible solutions even on small-sized devices with our multi-FPGA partitioning methodology. To meet all cost, accuracy, and speed requirements set by the European Space Agency for this system, we customize our HW/SW co-processor by design space exploration and testing on a Mars-like dataset. Implemented on Xilinx Virtex technology, or European NG-MEDIUM devices, the FPGA kernel processes a 1,120 × 1,120 stereo pair in 1.7s−3.1s, utilizing only 5.4−9.3 LUT6 and 200−312 RAMB18. The proposed system exhibits up to 32× speedup over desktop CPUs, or 2,810× over space-grade LEON3, and achieves a mean reconstruction error less than 2cm up to 4m depth. Excluding errors exceeding 2cm (which are less than 4% of the total), the mean error is under 8mm.

Absolute orientation estimation is the determination of the similarity transformation between two sets of corresponding 3D points, a task arising frequently in computer vision and robotics. We have recently proposed an absolute orientation algorithm based on the Fast Optimal Attitude Matrix (FOAM) algorithm from astronautics and demonstrated that it is more efficient computationally compared to widely-used approaches involving costly eigenand singular-value matrix decompositions. In this work, we compare our FOAM-based solution with several more algorithms derived from attitude estimation techniques and show that further computational savings are possible by employing an algorithm grounded on the Optimal Linear Attitude Estimator (OLAE) method.

Estimating the pose of orbiting satellites is a key prerequisite for supporting autonomous proximity operations in space. This work presents a monocular 3D tracking algorithm that tracks edges with the aid of an arbitrary 3D mesh model assumed to capture a satellite's shape. The proposed tracker propagates a pose hypothesis between successive frames, using it first to render a depth image and then refining it according to partial matches established between depth and intensity edges. Edge matching relies on fast, local 1D searches along the depth gradient direction. The tracker does not require any pre-processing of the 3D model nor does it make any assumptions regarding its characteristics, as is often the case for other approaches. It is also robust to parts of the tracked satellite being out of view, occluded, shadowed or visually undetected. Experimental results evaluating the accuracy of the tracker and comparing it with established techniques are also included.

We introduce T-LESS, a new public dataset for estimating the 6D pose, i.e. translation and rotation, of texture-less rigid objects. The dataset features thirty industry-relevant objects with no significant texture and no discriminative color or reflectance properties. The objects exhibit symmetries and mutual similarities in shape and/or size. Compared to other datasets, a unique property is that some of the objects are parts of others. The dataset includes training and test images that were captured with three synchronized sensors, specifically a structured-light and a time-of-flight RGB-D sensor and a high-resolution RGB camera. There are approximately 39K training and 10K test images from each sensor. Additionally, two types of 3D models are provided for each object, i.e. a manually created CAD model and a semi-automatically reconstructed one. Training images depict individual objects against a black background. Test images originate from twenty test scenes having varying complexity, which increases from simple scenes with several isolated objects to very challenging ones with multiple instances of several objects and with a high amount of clutter and occlusion. The images were captured from a systematically sampled view sphere around the object/scene, and are annotated with accurate ground truth 6D poses of all modeled objects. Initial evaluation results indicate that the state of the art in 6D object pose estimation has ample room for improvement, especially in difficult cases with significant occlusion. The T-LESS dataset is available online at cmp:felk:cvut:cz/t-less.

The absolute orientation problem arises often in vision and robotics. Despite that robust algorithmic solutions exist for quite some time, they all rely on matrix factorizations such as eigen or singular value decomposition. These factorizations are relatively expensive to compute, therefore might become a performance bottleneck when absolute orientation needs to be repeatedly computed on low-end hardware. The issue is exacerbated by implementations relying on standard numerical software libraries like LAPACK, since the linear algebra factorization routines they include are optimized for large matrices and thus are not the most efficient choice for small ones. Based on an attitude estimation algorithm originating from astronautics, this paper proposes a direct, factorization-free solution to the absolute orientation problem that is both computationally efficient and numerically accurate. Results from an experimental comparison with established methods demonstrate its superior performance.

Future Mars exploration missions rely heavily on high-mobility autonomous rovers equipped with sophisticated scientific instruments and possessing advanced navigational capabilities. Increasing their navigation velocity and localization accuracy is essential for enabling these rovers to explore large areas on Mars. Contemporary Mars rovers move slowly, partially due to the long execution time of complex computer vision algorithms running on their slow space-grade CPUs. This paper exploits the advent of high-performance space-grade field-programmable gate arrays (FPGAs) to accelerate the navigation of future rovers. Specifically, it focuses on visual odometry (VO) and performs HW/SW codesign to achieve one order of magnitude faster execution and improved accuracy. Conforming to the specifications of the European Space Agency, we build a proof-of-concept system on an HW/SW platform with processing power resembling that to be available onboard future rovers. We develop a codesign methodology adapted to the rover's specifications, design parallel architectures, and customize several feature extraction, matching, and motion estimation algorithms. We implement and evaluate five distinct HW/SW pipelines on a Virtex6 FPGA and a 150 MIPS CPU. We provide a detailed analysis of their cost-time-accuracy tradeoffs and quantify the benefits of employing FPGAs for implementing VO. Our solution achieves a speedup factor of 16× over a CPU-only implementation, handling a stereo image pair in less than 1 s, with a 1.25% mean positional error after a 100 m traverse and an FPGA cost of 54 K LUTs and 1.46-MB RAM.

Despite their ubiquitous presence, texture-less objects present significant challenges to contemporary visual object detection and localization algorithms. This paper proposes a practical method for the detection and accurate 3D localization of multiple texture-less and rigid objects depicted in RGB-D images. The detection procedure adopts the sliding window paradigm, with an efficient cascade-style evaluation of each window location. A simple pre-filtering is performed first, rapidly rejecting most locations. For each remaining location, a set of candidate templates (i.e. trained object views) is identified with a voting procedure based on hashing, which makes the method's computational complexity largely unaffected by the total number of known objects. The candidate templates are then verified by matching feature points in different modalities. Finally, the approximate object pose associated with each detected template is used as a starting point for a stochastic optimization procedure that estimates accurate 3D pose. Experimental evaluation shows that the proposed method yields a recognition rate comparable to the state of the art, while its complexity is sub-linear in the number of templates.

Visual Odometry (VO) is a proven technology for planetary exploration rovers, facilitating their localization with a small error over medium-sized trajectories. However, due to VO's incremental mode of operation, its estimation error accumulates over time, resulting in considerable drift for long trajectories. This paper proposes a global localization method that counters VO drift by matching boulders extracted from overhead and ground images and using them periodically to relocalize the rover and refine VO estimates. The performance of the proposed method is evaluated with the aid of overhead imagery of different resolutions. Experimental results demonstrate that a very terse representation, consisting of approximate boulder locations only, suffices for significantly improving the accuracy of VO over long traverses.

We present a new method for tracking the 3D position, global orientation and full articulation of human hands. Following recent advances in model-based, hypothesize-and-test methods, the high-dimensional parameter space of hand configurations is explored with a novel evolutionary optimization technique specifically tailored to the problem. The proposed method capitalizes on the fact that samples from quasi-random sequences such as the Sobol have low discrepancy and exhibit a more uniform coverage of the sampled space compared to random samples obtained from the uniform distribution. The method has been tested for the problems of tracking the articulation of a single hand (27D parameter space) and two hands (54D space). Extensive experiments have been carried out with synthetic and real data, in comparison with state of the art methods. The quantitative evaluation shows that for cases of limited computational resources, the new approach achieves a speed-up of four (single hand tracking) and eight (two hands tracking) without compromising tracking accuracy. Interestingly, the proposed method is preferable compared to the state of the art either in the case of limited computational resources or in the case of more complex (i.e., higher dimensional) problems, thus improving the applicability of the method in a number of application domains.

Metric rectification has important applications to topics such as single-view metrology, camera calibration, optical character recognition and texture extraction or synthesis. This paper proposes a method for estimating the rectification homography of a world plane that pertains to a single view containing the images of at least two circles in general position. The method does not presuppose that the circles radii, the camera's intrinsics or the plane's vanishing line are known and can simultaneously accommodate multiple circles. Comparative experimental results demonstrate the approach's efficacy.

Bundle adjustment constitutes a large, nonlinear least-squares problem that is often solved as the last step of feature-based structure and motion estimation computer vision algorithms to obtain optimal estimates. Due to the very large number of parameters involved, a general purpose least-squares algorithm incurs high computational and memory storage costs when applied to bundle adjustment. Fortunately, the lack of interaction among certain subgroups of parameters results in the corresponding Jacobian being sparse, a fact that can be exploited to achieve considerable computational savings. This article presents sba, a publicly available C/C++ software package for realizing generic bundle adjustment with high efficiency and flexibility regarding parameterization.

This paper proposes a feature-based method for recovering the relative positions of the viewpoints of a set of panoramic images for which no a priori order information is available, along with certain structure information regarding the imaged environment. The proposed approach operates incrementally, employing the Levenshtein distance to deduce the spatial proximity of image viewpoints and thus determine the order in which images should be processed. The Levenshtein distance also provides matches between images, from which their underlying environment points can be recovered. Recovered points that are visible in multiple views permit the localization of more views which in turn allow the recovery of more points. The process repeats until all views have been localized. Periodic refinement of the reconstruction with the aid of bundle adjustment, distributes the reconstruction errors among images. The method is demonstrated on several unordered sets of panoramic images obtained in an indoor environment.

This paper concerns a method for tracking multiple blobs exhibiting certain color distributions in images acquired by a possibly moving camera. The method encompasses a collection of techniques that enable modeling and detecting the blobs possessing the desired color distribution(s), as well as inferring their temporal association across image sequences. Appropriately colored blobs are detected with a Bayesian classifier, which is bootstrapped with a small set of training data. Then, an online iterative training procedure is employed to refine the classifier using additional training images. Online adaptation of color probabilities is used to enable the classifier to cope with illumination changes. Tracking over time is realized through a novel technique, which can handle multiple colored blobs. Such blobs may move in complex trajectories and occlude each other in the field of view of a possibly moving camera, while their number may vary over time. A prototype implementation of the developed system running on a conventional Pentium IV processor at 2.5 GHz operates on 320/spl times/240 live video in real time (30Hz). It is worth pointing out that currently, the cycle time of the tracker is determined by the maximum acquisition frame rate that is supported by our IEEE 1394 camera, rather than the latency introduced by the computational overhead for tracking blobs.

y are being acquired. Furthermore, it does not rely upon the presence in the environment of fiducial markers or special calibration objects. A brief overview of our approach is given in the next section. More detailed descriptions can be found in [1] and online at http://www. ics.forth.gr/lourakis/camtrack/. THE APPROACH Our method is based on a feature-based 3D plane tracking technique, which permits the estimation of the homographies induced by a 3D plane between successive image pairs. At the core of plane tracking lies a homography "chaining" operation that is applied to triplets of consecutive images through a sliding time window and exploits the constraint that all images of a planar surface acquired by a rigidly moving observer depend upon the same 3D geometry. Since the tracked plane is not required to be physically present in the scene, a virtual one can be used instead. Plane tracking is achieved by matching between images the 2D projections of points from all over the scen

In order to obtain optimal 3D structure and viewing parameter estimates, bundle adjustment is often used as the last step of feature-based structure and motion estimation algorithms. Bundle adjustment involves the formulation of a large scale, yet sparse minimization problem, which is traditionally solved using a sparse variant of the Levenberg-Marquardt optimization algorithm that avoids storing and operating on zero entries. This paper argues that considerable computational benefits can be gained by substituting the sparse Levenberg-Marquardt algorithm in the implementation of bundle adjustment with a sparse variant of Powell's dog leg non-linear least squares technique. Detailed comparative experimental results provide strong evidence supporting this claim.

We present a computationally efficient method for estimating the trifocal tensor corresponding to three images acquired by a freely moving camera. The proposed method represents projective space through a "plane + parallax" decomposition and employs a novel technique for estimating the homographies induced by a virtual 3D plane between successive image pairs. Knowledge of these homographies allows the corresponding camera projection matrices to be expressed in a common projective frame and, therefore, to be recovered directly. The trifocal tensor can then be recovered in a straightforward manner from the estimated projection matrices. Sample experimental results demonstrate that the method performs considerably faster compared to a state of the art method, without a serious loss in accuracy.

We address the problem of tracking the 3D position and orientation of a camera, using the images it acquires while moving freely in unmodeled, arbitrary environments. This task has a broad spectrum of useful applications in domains such as augmented reality and video post production. Most of the existing methods for vision-based camera tracking are designed to operate in a batch, off-line mode, assuming that the whole video sequence to be tracked is available before tracking commences. Typically, such methods operate noncausally, processing video frames backwards and forwards in time as they see fit. Furthermore, they resort to optimization in very high dimensional spaces, a process that is computationally intensive. For these reasons, batch methods are inapplicable to tracking in online, time-critical applications such as video see-through augmented reality. This paper puts forward a novel feature-based approach for camera tracking. The proposed approach operates on images continuously as they are acquired, has realistic computational requirements and does not require modifications of the environment. Sample experimental results demonstrating the feasibility of the approach on video images are also provided

Many vision tasks rely upon the identification of sets of corresponding features among different images. This paper presents a method that, given some corresponding features in two stereo images, matches them with features extracted from a second stereo pair captured from a distant viewpoint. The proposed method is based on the assumption that the viewed scene contains two planar surfaces and exploits geometric constraints that are imposed by the existence of these planes to first transfer and then match image features between the two stereo pairs. The resulting scheme handles point and line features in a unified manner and is capable of successfully matching features extracted from stereo pairs that are acquired from considerably different viewpoints. Experimental results are presented, which demonstrate that the performance of the proposed method compares favorably to that of epipolar and tensor-based approaches.

Plane detection is a prerequisite to a wide variety of vision tasks. This paper proposes a novel method that exploits results from projective geometry to automatically detect planes using two images. Using a set of point and line features that have been matched between images, the method exploits the fact that every pair of a 3D line and a 3D point defines a plane and utilizes an iterative voting scheme for identifying coplanar subsets of the employed feature set. The method does not require camera calibration, circumvents the 3D reconstruction problem, is robust to the existence of mismatched features and is applicable either to stereo or motion sequence images. Sample results from the application of the proposed method to real imagery are also provided. 1

Several vision tasks rely upon the availability of sets of corresponding features among images. This paper presents a method which, given some corresponding features in two stereo images, addresses the problem of matching them with features extracted from a second stereo pair captured from a distant viewpoint. The proposed method is based on the assumption that the viewed scene contains two planar surfaces and exploits geometric constraints that are imposed by the existence of these planes to predict the location of image features in the second stereo pair. The resulting scheme handles point and line features in a unified manner and is capable of successfully matching features extracted from stereo pairs acquired from considerably different viewpoints. Experimental results from a prototype implementation demonstrate the effectiveness of the approach.

To avoid collisions, mobile robots need a measure for assessing the proximity of obstacles. A measure that is well suited to this purpose is provided by the time-to-contact. In this paper, a novel method for estimating the time-to-contact is presented. The method is based on the assumption that the robot is moving on a locally planar ground, parts of which are visible in the acquired images. Using the optical flow field induced by the robot's motion, the time-to-contact with points on the ground is first estimated and then the concept of planar parallax is exploited to recover the time-to-contact with obstacle points. The computation of high order derivatives of the image flow, which are known to be very difficult to estimate accurately, is completely avoided. Moreover, no knowledge of the egomotion is necessary. Experimental results from the application of the method on real and simulated flow fields are also reported.

Feature matching is a prerequisite to a wide variety of vision tasks. This paper presents a method that addresses the problem of matching disparate views of coplanar points and lines in a unified manner. The proposed method employs a randomized search strategy combined with the two-line two-point projective invariant to derive small sets of possibly matching points and lines. These candidate matches are then verified by recovering the associated plane homography, which is further used to predict more matches. The resulting scheme is capable of successfully matching features extracted from views that differ considerably, even in the presence of large numbers of outlying features. Experimental results from the application of the method to indoor and aerial images indicate its effectiveness and robustness. 1 Introduction A fundamental problem in computer vision, that appears in different forms in tasks such as discrete motion estimation, feature-based stereo, object recognition, image r...

This paper considers a specific problem of visual perception of motion, tamely the problem of visual detection of independent 3D motion. Most of the existing techniques for solving this problem rely on restrictive assumptions about the environment, the observer's motion, or both. Moreover, they are based on the computation of a dense optical flow field, which amounts to solving the ill-posed correspondence problem. In this work independent motion detection is formulated as a problem of robust parameter estimation applied to the visual input acquired by a rigidly moving observer. The proposed method automatically selects a planar surface in the scene and the residual planar parallax normal flow field with respect to the motion of this surface is computed at two successive time! instants. The two resulting normal flow fields are then combined in a linear model. The parameters of this model are related to the parameters of self-motion (ego-motion) and their robust estimation leads to a segmentation of the scene based on 3D motion. The method avoids a complete solution to the correspondence problem by selectively matching subsets of image points and by employing normal flow fields. Experimental results demonstrate the effectiveness of the proposed method in detecting independent motion in scenes with large depth variations and unrestricted observer motion.

An approach towards providing advanced navigational support to robotic wheelchair platforms is presented. In order to avoid any modifications to the environment, we propose an approach that employs computer vision techniques which facilitate space perception and navigation. Computer vision has not been introduced to date in rehabilitation robotics, since the former is not mature enough to meet the needs of this sensitive application. However, in the proposed approach, stable techniques are exploited that facilitate reliable, automatic navigation to any point in the visible environment. Preliminary results obtained from its implementation on a laboratory robotic platform indicate its usefulness and flexibility.

This paper presents a methodology for the detection of objects that move independently of the observer in a 3D dynamic environment. Independent 3D motion detection is formulated as a problem of robust regression applied to visual input acquired by a binocular, rigidly moving observer. The qualitative analysis of images acquired by a parallel stereo configuration yields a segmentation of a scene into depth layers. A depth layer consists of points of the 3D space with almost constant depth from the observer. Robust regression in the form of Least Median of Squares estimation is applied within each depth layer in order to segment the latter into coherently moving regions. Finally, a combination stage is applied across all layers in order to come up with an integrated view of independent motion in the whole 3D scene. In contrast to other existing approaches for independent motion detection which are based on the ill-posed problem of optical flow computation, the proposed methodology relies...

This paper presents a method for the detection of objects that move independently of the observer in a 3D dynamic scene, Independent motion detection is achieved through processing of stereoscopic image sequences acquired by a binocular, rigidly moving observer. A weak assumption is made about the observer's motion (egomotion), namely that the direction of the translational and rotational components of egomotion are constant in small image patches. This assumption facilitates the extraction of qualitative information about depth from motion, while additional qualitative depth information is independently computed from image stereo pairs acquired by the binocular vision system. Robust regression in the form of least median of squares estimation is applied within each image patch to test for consistency between the depth functions computed from motion and stereo. Possible inconsistencies signal the presence of independently moving objects. In contrast to other existing approaches for independent motion detection, which are based on the ill-posed problem of optical flow computation, the proposed method relies on normal flow fields for both stereo and motion processing. By exploiting local constraints of qualitative nature, the problem of independent motion detection is approached directly, without relying on a solution to the general structure from motion problem. Experimental results indicate that the proposed method is both effective and robust.