In this paper, we revisit trajectory-based action detection in a potent and non-uniform way. Improved trajectories have been proven to be an effective model for motion description in action recognition. In temporal action localization, however, this approach is not efficiently exploited. Trajectory features extracted from uniform video segments result in significant performance degradation due to two reasons: (a) during uniform segmentation, a significant amount of noise is often added to the main action and (b) partial actions can have negative impact in classifier's performance. Since uniform video segmentation seems to be insufficient for this task, we propose a two-step supervised non-uniform segmentation, performed in an online manner. Action proposals are generated using either 2D or 3D data, therefore action classification can be directly performed on them using the standard improved trajectories approach. We experimentally compare our method with other approaches and we show improved performance on a challenging online action detection dataset.

The article describes a pipeline that receives as input a sequence of stereo images, and outputs the camera motion and a Piecewise-Planar Reconstruction (PPR) of the scene. The pipeline, named Piecewise-Planar StereoScan (PPSS), works as follows: the planes in the scene are detected for each stereo view using semi-dense depth estimation; the relative pose is computed by a new closed-form minimal algorithm that only uses point correspondences whenever plane detections do not fully constrain the motion; the camera motion and the PPR are jointly refined by alternating between discrete optimization and continuous bundle adjustment; and, finally, the detected 3D planes are segmented in images using a new framework that handles low texture and visibility issues. PPSS is extensively validated in indoor and outdoor datasets, and benchmarked against two popular point-based SfM pipelines. The experiments confirm that plane-based visual odometry is resilient to situations of small image overlap, poor texture, specularity, and perceptual aliasing where the fast LIBVISO2 [1] pipeline fails. The comparison against VisualSfM+CMVS/PMVS [2] , [3] shows that, for a similar computational complexity, PPSS is more accurate and provides much more compelling and visually pleasant 3D models. These results strongly suggest that plane primitives are an advantageous alternative to point correspondences for applications of SfM and 3D reconstruction in man-made environments.

The article concerns the automatic calibration of a camera with radial distortion from a single image. It is known that, under the mild assumption of square pixels and zero skew, lines in the scene project into circles in the image, and three lines suffice to calibrate the camera up to an ambiguity between focal length and radial distortion. The calibration results highly depend on accurate circle estimation, which is hard to accomplish because lines tend to project into short circular arcs. To overcome this problem, we show that, given a short circular arc edge, it is possible to robustly determine a line that goes through the center of the corresponding circle. These lines, henceforth called Lines of Circle Centres (LCCs), are used in a new method that detects sets of parallel lines and estimates the calibration parameters, including the center and amount of distortion, focal length, and camera orientation with respect to the Manhattan frame. Extensive experiments in both semi-synthetic and real images show that our algorithm outperforms state-of-the-art approaches in unsupervised calibration from a single image, while providing more information.

Action recognition using dense trajectories is a popular concept. However, many spatio-temporal characteristics of the trajectories are lost in the final video representation when using a single Bag-of-Words model. Also, there is a significant amount of extracted trajectory features that are actually irrelevant to the activity being analyzed, which can considerably degrade the recognition performance. In this paper, we propose a human-tailored trajectory extraction scheme, in which trajectories are clustered using information from the human pose. Two configurations are considered; first, when exact skeleton joint positions are provided, and second, when only an estimate thereof is available. In both cases, the proposed method is further strengthened by using the concept of local Bag-of-Words, where a specific codebook is generated for each skeleton joint group. This has the advantage of adding spatial human pose awareness in the video representation, effectively increasing its discriminative power. We experimentally compare the proposed method with the standard dense trajectories approach on two challenging datasets.

This paper revisits the problem of human action recognition from skeleton joint locations, and analyses the tradeoff of sampling the joint space with respect to the recognition performance and computational complexity. The provided insights led to the design of a new algorithm for automatically selecting the most appropriate set of joints for each action. During the training stage, the approach applies a guided joint sampling strategy for learning different SVM classifiers, selecting the classifier that maximizes confidence and ambiguity metrics. Experimental results on three action datasets show that pre-selecting the most varying skeleton joints for each action dramatically reduces the computational complexity while keeping competitive recognition rates.

We propose a simple and useful idea based on cross-ratio constraint for wide-baseline matching and 3D reconstruction. Most existing methods exploit feature points and planes from images. Lines have always been considered notorious for both matching and reconstruction due to the lack of good line descriptors. We propose a method to generate and match new points using virtual lines constructed using pairs of keypoints, which are obtained using standard feature point detectors. We use cross-ratio constraints to obtain an initial set of new point matches, which are subsequently used to obtain line correspondences. We develop a method that works for both calibrated and uncalibrated camera configurations. We show compelling line-matching and large-scale 3D reconstruction.

Instead of measuring photo-similarity, SymStereo is a stereo vision algorithm that uses new cost functions to measure symmetry differences between pairs of images. In this paper we propose the acceleration of a complete signal processing pipeline for generating 3D volumes based on dense SymStereo. The outputs here generated achieve superior reconstruction quality namely for slant based scenarios, so typical in autonomous systems, that have to capture pairs of images and perform moving decisions in real-time. In particular, we analyse several parallelization strategies for the compute-intensive aggregation procedure using different parameters and evaluate a trade-off between processing time, and higher precision of the calculated depths and quality of the final reconstructed 3D volume. The developed parallel pipeline allows to process more than 4.5 volumes per second for high resolution images using commodity GPUs, which conveniently suits its application in a variety of robotics systems.

SymStereo is a new algorithm used for stereo estimation. Instead of measuring photo-similarity, it proposes novel cost functions that measure symmetry for evaluating the likelihood of two pixels being a match. In this work we propose a parallel approach of the LogN matching cost variant of SymStereo capable of processing pairs of images in real-time for depth estimation. The power of the graphics processing units utilized allows exploring more efficiently the bank of log-Gabor wavelets developed to analyze symmetry, in the spectral domain. We analyze tradeoffs and propose different parameter-izations of the signal processing algorithm to accommodate image size, dimension of the filter bank, number of wavelets and also the number of disparities that controls the space density of the estimation, and still process up to 53 frames per second (fps) for images with size 288 × 384 and up to 3 fps for 768 × 1024 images.

Odometry consists in using data from a moving sensor to estimate change in position over time. It is a crucial step for several applications in robotics and computer vision. This paper presents a novel approach for estimating the relative motion between suc-cessive RGB-D frames that uses plane-primitives instead of point features. The planes in the scene are extracted and the motion estimation is cast as a plane-to-plane registra-tion problem with a closed-form solution. Point features are only extracted in the cases where the plane surface configuration is insufficient to determine motion with no ambi-guity. The initial estimate is refined in a photo-geometric optimization step that takes full advantage of the plane detection and simultaneous availability of depth and visual appearance cues. Extensive experiments show that our plane-based approach is as accu-rate as state-of-the-art point-based approaches when the camera displacement is small, and significantly outperforms them in case of wide-baseline and/or dynamic foreground. 1

This article presents a new global approach for detecting vanishing points and groups of mutually orthogonal vanishing directions using lines detected in images of man-made environments. These two multi-model fitting problems are respectively cast as Uncapacited Facility Location (UFL) and Hierarchical Facility Location (HFL) instances that are efficiently solved using a message passing inference algorithm. We also propose new functions for measuring the consistency between an edge and a putative vanishing point, and for computing the vanishing point defined by a subset of edges. Extensive experiments in both synthetic and real images show that our algorithms outperform the state-of-the-art methods while keeping computation tractable. In addition, we show for the first time results in simultaneously detecting multiple Manhattan-world configurations that can either share one vanishing direction (Atlanta world) or be completely independent.

Estimating the amount and center of distortion from lines in the scene has been addressed in the literature by the so-called ``plumb-line'' approach. In this paper we propose a new geometric method to estimate not only the distortion parameters but the entire camera calibration (up to an ``angular'' scale factor) using a minimum of 3 lines. We propose a new framework for the unsupervised simultaneous detection of natural image of lines and camera parameters estimation, enabling a robust calibration from a single image. Comparative experiments with existing automatic approaches for the distortion estimation and with ground truth data are presented.

Many robotic systems combine cameras with Laser Rangefinders (LRF) for simultaneously achieving multi-purpose visual sensing and accurate depth recovery. Employing a single sensor modality for accomplishing both goals is an appealing proposition because it enables substantial savings in equipment, and tends to decrease the overall complexity of the system. This article explores the possibility of replacing LRF by passive stereo vision for reconstructing the scene along a 2D scan plane. We present a new stereo algorithm that is specifically tailored for the purpose. The algorithm recovers the depth along the scan plane using a symmetry-based matching cost (SymStereo), and refines the raw estimates by applying dynamic programming, followed by a Markov Random Field (MRF) that decides if the reconstructed contour is a line or not. We report for the first time comparative experiments between Stereo Rangefinding (SRF) and LRF. The results are encouraging by showing that SRF can be a plausible alternative to LRF in several application scenarios. Moreover, since SRF also enables independent depth estimates along multiple scan planes with arbitrary orientation, being the only constraint that the scan plane intersects the stereo baseline, it is an important benefit that can be decisive for many robotic applications.