Long term visual localization has to conquer the problem of matching images with dramatic photometric changes caused by different seasons, natural and man-made illumination changes, etc. Visual localization at night plays a vital role in many applications like autonomous driving and augmented reality, for which extracting keypoints and descriptors with robustness to day-night illumination changes has became the bottleneck. This paper proposes an adversarial learning based solution to harvest from the weakly domain labels of day and night images, along with the point level correspondences among day time images, to achieve robust local feature extraction and description across day-night images. The key idea is to learn a discriminator to distinguish whether a feature map is generated from the day or night images, and simultaneously to adjust the parameters of feature extraction network so as to fool the discriminator. After adversarial training of the discriminator and feature extraction network, the feature extraction network finally reaches a stable status so that the extracted feature maps are robust to day-night photometric changes, based on which day-night domain invariant keypoints and descriptors can be extracted. Compared to existing local feature learning methods, it only requires an additional set of easily captured night images to improve the domain invariance of learned features. Experiments on two challenging benchmarks show the effectiveness of proposed method. In addition, this paper revisits the widely used image matching metrics on HPatches and finds that recall of different methods is highly related to their relative localization performance.

Extracting robust and discriminative local features from images plays a vital role for long term visual localization, whose challenges are mainly caused by the severe appearance differences between matching images due to the day-night illuminations, seasonal changes, and human activities. Existing solutions resort to jointly learning both keypoints and their descriptors in an end-to-end manner, leveraged on large number of annotations of point correspondence which are harvested from the structure from motion and depth estimation algorithms. While these methods show improved performance over non-deep methods or those two-stage deep methods, i.e., detection and then description, they are still struggled to conquer the problems encountered in long term visual localization. Since the intrinsic semantics are invariant to the local appearance changes, this paper proposes to learn semantic-aware local features in order to improve robustness of local feature matching for long term localization. Based on a state of the art CNN architecture for local feature learning, i.e., ASLFeat, this paper leverages on the semantic information from an off-the-shelf semantic segmentation network to learn semantic-aware feature maps. The learned correspondence-aware feature descriptors and semantic features are then merged to form the final feature descriptors, for which the improved feature matching ability has been observed in experiments. In addition, the learned semantics embedded in the features can be further used to filter out noisy keypoints, leading to additional accuracy improvement and faster matching speed. Experiments on two popular long term visual localization benchmarks (Aachen Day and Night v1.1, Robotcar Seasons) and one challenging indoor benchmark (InLoc) demonstrate encouraging improvements of the localization accuracy over its counterpart and other competitive methods.

Despite the fact that Second Order Similarity (SOS) has been used with significant success in tasks such as graph matching and clustering, it has not been exploited for learning local descriptors. In this work, we explore the potential of \sos in the field of descriptor learning by building upon the intuition that a positive pair of matching points should exhibit similar distances with respect to other points in the embedding space. Thus, we propose a novel regularization term, named Second Order Similarity Regularization (SOSR), that follows this principle. By incorporating SOSR into training, our learned descriptor achieves state-of-the-art performance on several challenging benchmarks containing distinct tasks ranging from local patch retrieval to structure from motion. Furthermore, by designing a von Mises-Fischer distribution based evaluation method, we link the utilization of the descriptor space to the matching performance, thus demonstrating the effectiveness of our proposed SOSR. Extensive experimental results, empirical evidence, and in-depth analysis are provided, indicating that SOSR can significantly boost the matching performance of the learned descriptor.

Cayley-Klein metric is a specific kind of non-Euclidean metric in projective space. Recently, it has been introduced into metric learning with encouraging performance when dealing with computer vision tasks. However, the original Cayley-Klein metric learning methods with conventional pairwise and triplet-wise constraints, which are fixed bound based constraints, may not perform well when the intra-and inter-class variations of data distribution become complex. Pairwise constraints restrict the distance between samples of a similar pair to be lower than a fixed upper bound, and the distance between samples of a dissimilar pair higher than a fixed lower bound. Triplet-wise constraints restrict the distance between samples of a similar pair to be smaller than that between a pair of samples from different classes. In this paper, we propose a novel Cayley-Klein metric learning method (CKseML) with adaptive shrinkage-expansion pairwise constraints. CKseML is very effective in learning metric from data with complex distributions. Our experimental results demonstrate that CKseML achieves better performance than the original Cayley-Klein metric learning methods.

Deep convolutional neural networks (CNN) has achieved state-of-the-art result on traffic sign classification, which plays a key role in intelligent transportation system. However, it usually requires a large number of labeled training data, which is not always available, to guarantee a good performance. In this paper, we propose to synthesize traffic sign images by generative adversarial networks (GANs). It takes a standard traffic sign template and a background image as input to the generative network in GANs, where the template defines which class of traffic sign to include and the background image controls the visual appearance of the synthetic images. Experiments show that our method could generate more realistic traffic sign images than the conventional image synthesis method. Meanwhile, by adding the synthesis images to train a typical CNN for traffic sign classification, we obtained a better accuracy.

Although traffic sign recognition has been studied for many years, most existing works are focused on the symbol-based traffic signs. This paper proposes a new data-driven system to recognize all categories of traffic signs, which include both symbol-based and text-based signs, in video sequences captured by a camera mounted on a car. The system consists of three stages, traffic sign regions of interest (ROIs) extraction, ROIs refinement and classification, and post-processing. Traffic sign ROIs from each frame are first extracted using maximally stable extremal regions on gray and normalized RGB channels. Then, they are refined and assigned to their detailed classes via the proposed multi-task convolutional neural network, which is trained with a large amount of data, including synthetic traffic signs and images labeled from street views. The post-processing finally combines the results in all frames to make a recognition decision. Experimental results have demonstrated the effectiveness of the proposed system.

The research focus of designing local patch descriptors has gradually shifted from handcrafted ones (e.g., SIFT) to learned ones. In this paper, we propose to learn high performance descriptor in Euclidean space via the Convolutional Neural Network (CNN). Our method is distinctive in four aspects: (i) We propose a progressive sampling strategy which enables the network to access billions of training samples in a few epochs. (ii) Derived from the basic concept of local patch matching problem, we empha-size the relative distance between descriptors. (iii) Extra supervision is imposed on the intermediate feature maps. (iv) Compactness of the descriptor is taken into account. The proposed network is named as L2-Net since the output descriptor can be matched in Euclidean space by L2 distance. L2-Net achieves state-of-the-art performance on the Brown datasets [16], Oxford dataset [18] and the newly proposed Hpatches dataset [11]. The good generalization ability shown by experiments indicates that L2-Net can serve as a direct substitution of the existing handcrafted descriptors. The pre-trained L2-Net is publicly available.

This paper aims to build robust feature descriptors by exploring intensity order information in a patch. To this end, the local intensity order pattern (LIOP) and the overall intensity order pattern (OIOP) are proposed to effectively encode intensity order information of each pixel in different aspects. Specifically, LIOP captures the local ordinal information by using the intensity relationships among all the neighbouring sampling points around a pixel, while OIOP exploits the coarsely quantized overall intensity order of these sampling points. These two kinds of patterns are then separately aggregated into different ordinal bins, leading to two kinds of feature descriptors. Furthermore, as these two kinds of descriptors could encode complementary ordinal information, they are combined together to obtain a discriminative and compact mixed intensity order pattern descriptor. All these descriptors are constructed on the basis of relative relationships of intensities in a rotationally invariant way, making them be inherently invariant to image rotation and any monotonic intensity changes. Experimental results on image matching and object recognition are encouraging, demonstrating the superiorities of our descriptors over the state of the art.

Traffic sign recognition plays an important role in driver assistant systems and intelligent autonomous vehicles. Its real-time performance is highly desirable in addition to its recognition performance. This paper aims to deal with real-time traffic sign recognition, i.e., localizing what type of traffic sign appears in which area of an input image at a fast processing time. To achieve this goal, we first propose an extremely fast detection module, which is 20 times faster than the existing best detection module. Our detection module is based on traffic sign proposal extraction and classification built upon a color probability model and a color HOG. Then, we harvest from a convolutional neural network to further classify the detected signs into their subclasses within each superclass. Experimental results on both German and Chinese roads show that both our detection and classification methods achieve comparable performance with the state-of-the-art methods, with significantly improved computational efficiency.

In this paper, we aim to explore the role of prior knowledge for pedestrian detection. The main idea is to integrate human body priors into the design of features. To this end, we propose the symmetric features and cross-channel features so as to capture the specific information of human body. Experimental results demonstrate that our detector achieves state-of-the-art performance. What’s more, the evaluation results on subsets of Caltech-USA show that our detector performs best at medium scale and therefore has great potential to be integrated into real-world applications.

Cayley-Klein metric is a kind of non-Euclidean metric suitable for projective space. In this paper, we introduce it into the computer vision community as a powerful metric and an alternative to the widely studied Mahalanobis metric. We show that besides its good characteristic in non-Euclidean space, it is a generalization of Mahalanobis metric in some specific cases. Furthermore, as many Mahalanobis metric learning, we give two kinds of Cayley-Klein metric learning methods: MMC Cayley-Klein metric learning and LMNN Cayley-Klein metric learning. Experiments have shown the superiority of Cayley-Klein metric over Mahalanobis ones and the effectiveness of our Cayley-Klein metric learning methods.

Establishing robust visual correspondences is a fundamental component of many computer vision applications. However, it is very challenging to obtain high quality features while maintaining a low computational cost. This paper aims to tackle this problem by adopting a novel Fast Robust Invariant Feature (FRIF) for both feature detection and description. The basic idea is to employ a fast approximated LoG detector to select scale-invariant keypoints and incorporate local pattern and inter-pattern information to construct distinctive binary descriptors. A comprehensive evaluation on standard dataset shows that FRIF achieves quite a high performance with a computation time comparable to state-of-the-art real-time features.

This paper proposes a novel method for interest region description which pools local features based on their intensity orders in multiple support regions. Pooling by intensity orders is not only invariant to rotation and monotonic intensity changes, but also encodes ordinal information into a descriptor. Two kinds of local features are used in this paper, one based on gradients and the other on intensities; hence, two descriptors are obtained: the Multisupport Region Order-Based Gradient Histogram (MROGH) and the Multisupport Region Rotation and Intensity Monotonic Invariant Descriptor (MRRID). Thanks to the intensity order pooling scheme, the two descriptors are rotation invariant without estimating a reference orientation, which appears to be a major error source for most of the existing methods, such as Scale Invariant Feature Transform (SIFT), SURF, and DAISY. Promising experimental results on image matching and object recognition demonstrate the effectiveness of the proposed descriptors compared to state-of-the-art descriptors.

In this paper, a weighted similarity-invariant linear algorithm for camera calibration with rotating 1D objects is proposed. First, we propose a new estimation method for computing the relative depth of the free endpoint on the 1D object and prove its robustness against noise compared with those used in previous literature. The introduced estimator is invariant to image similarity transforms, resulting in a similarity-invariant linear calibration algorithm which is slightly more accurate than the well-known normalized linear algorithm. Then, we use the reciprocals of the standard deviations of the estimated relative depths from different images as the weights on the constraint equations of the similarity-invariant linear calibration algorithm, and propose a weighted similarity-invariant linear calibration algorithm with higher accuracy. Experimental results on synthetic data as well as on real image data show the effectiveness of our proposed algorithm.

A novel local image descriptor is proposed in this paper, which combines intensity orders and gradient distributions in multiple support regions. The novelty lies in three aspects: 1) The gradient is calculated in a rotation invariant way in a given support region; 2) The rotation invariant gradients are adaptively pooled spatially based on intensity orders in order to encode spatial information; 3) Multiple support regions are used for constructing descriptor which further improves its discriminative ability. Therefore, the proposed descriptor encodes not only gradient information but also information about relative relationship of intensities as well as spatial information. In addition, it is truly rotation invariant in theory without the need of computing a dominant orientation which is a major error source of most existing methods, such as SIFT. Results on the standard Oxford dataset and 3D objects have shown a significant improvement over the state-of-the-art methods under various image transformations.

This paper presents a novel method for feature description based on intensity order. Specifically, a Local Intensity Order Pattern(LIOP) is proposed to encode the local ordinal information of each pixel and the overall ordinal information is used to divide the local patch into subregions which are used for accumulating the LIOPs respectively. Therefore, both local and overall intensity ordinal information of the local patch are captured by the proposed LIOP descriptor so as to make it a highly discriminative descriptor. It is shown that the proposed descriptor is not only invariant to monotonic intensity changes and image rotation but also robust to many other geometric and photometric transformations such as viewpoint change, image blur and JEPG compression. The proposed descriptor has been evaluated on the standard Oxford dataset and four additional image pairs with complex illumination changes. The experimental results show that the proposed descriptor obtains a significant improvement over the existing state-of-the-art descriptors.

AID object is a segment with several known-distance markers, and calibration methods with ID objects are more flexible than those with 2D/3D objects. Under the pinhole camera model, it is proved that the calibration with free-moving ID objects is not possible. For a central catadioptric camera setup, can the camera be calibrated by a ID object under general motions? In this paper, we prove that a central catadioptric camera can indeed be calibrated, and propose a catadioptric camera calibration method using ID objects undertaking general motions. The proposed method consists of two steps. Firstly, the principal point is calculated with geometric invariants under catadioptric camera model; Secondly, we use images of ID object to calibrate the focal lengths, skew factor and mirror parameter. The method needs neither prior knowledge of catadioptric parameters nor conic fitting, and it is linear, which makes it easy to implement. Experiments demonstrate its usefulness and stability.

A novel method for line matching is proposed. The basic idea is to use tentative point correspondences, which can be easily obtained by keypoint matching methods, to significantly improve line matching performance, even when the point correspondences are severely contaminated by outliers. When matching a pair of image lines, a group of corresponding points that may be coplanar with these lines in 3D space is firstly obtained from all corresponding image points in the local neighborhoods of these lines. Then given such a group of corresponding points, the similarity between this pair of lines is calculated based on an affine invariant from one line and two points. The similarity is defined on the basis of median statistic in order to handle the problem of inevitable incorrect correspondences in the group of point correspondences. Furthermore, the relationship of rotation between the reference and query images is estimated from all corresponding points to filter out those pairs of lines which are obviously impossible to be matches, hence speeding up the matching process as well as further improving its robustness. Extensive experiments on real images demonstrate the good performance of the proposed method as well as its superiority to the state-of-the-art methods.

Line matching plays an important role in many applications, such as image registration, 3D reconstruction, object recognition and video understanding. However, compared with other features (such as point, region) matching, it has made little progress in recent years. In this paper, we investigate the problem of matching line segments automatically only from their neighborhood appearance, without any constraints or priori knowledge. A novel line descriptor called HLD descriptor is proposed for this purpose, which is constructed by the following three steps: (1) Line support region (LSR) is defined, then divided into a series of overlapped sub-regions with the same size; (2) Line description matrix (DM) is formed by characterizing each sub-region into a vector; (3) HLD descriptor is built by computing mean and standard deviation of DM column vectors. Experimental results show that HLD descriptor is highly distinctive and very robust for line matching on real images.

Visual metrology methods with radial distorted images usually require a radial distortion model and a pre-calibration. In this paper, we propose a novel 3D metrology algorithm with at least three uncalibrated radial distorted images, and also derive a 2D metrology algorithm with at least two uncalibrated radial distorted images. The algorithm does not require a radial distortion model or calibrating camera intrinsic parameters except for radial distortion center, which can be usually known as a prior or computed easily, and correspondences of control points with known coordinates. The algorithm is of high accuracy, and robust to noise due to no requirement for specific radial distortion models. Experimental results show the feasibility and accuracy of the algorithm.

Trifocal tensor encapsulates the geometric constraints between three views. It plays an important role in computer vision. However elements in measurement matrix of existing linear trifocal tensor estimation algorithms are products of several measurement data, which can amplify measurement error. The factorization algorithm of trifocal tensor estimation is presented, which can overcome this shortcoming. To overcome the deficiency that the extended solution space of factorization algorithm may result in an unstable solution, some modifications are adopted. Synthetic and real image data experiments show that the proposed algorithm is accurate and robust. This provides a new way different from normalized linear algorithm to improve performance of linear algorithm.

A new image feature called Harris feature vector is defined in this paper, which effectively describes the image gradient distribution. By computing the mean and the standard deviation of the Harris feature vector in key point neighborhood, a novel descriptor for key points matching is constructed, which is invariant to image rigid transformation and linear intensity change. Experimental evidence suggests that the novel descriptor has a good adaptability to slight view point changing, JPEG compression as well as nonlinear change of intensity.

It is well known that in order to calibrate a single camera with a one-dimensional (1D) calibration object, the object must undertake some constrained motions, in other words, it is impossible to calibrate a single camera if the object motion is of general one. For a multi-camera setup, i.e., when the number of camera is more than one, can the cameras be calibrated by a 1D object under general motions? In this work, we prove that all cameras can indeed be calibrated and a calibration algorithm is also proposed and experimentally tested. In contrast to other multi-camera calibration method, no one calibrated "base" camera is needed. In addition, we show that for such multi-camera cases, the minimum condition of calibration and critical motions are similar to those of calibrating a single camera with 1D calibration object.

In this paper, a novel algorithm for the fundamental matrix estimation, called factorized 8-point algorithm, is presented. The factorized 8-point algorithm is composed of three steps: (1) The measurement matrix in the traditional 8-point algorithm is decomposed into two factor matrices; (2) By introducing some auxiliary variables, a new linear minimization problem is formed, where every element of its associated measurement matrix is simply either a measurement datum or a constant; (3) The fundamental matrix is determined by solving this minimization problem by a least squares method. Like the traditional 8-point algorithm and Hartley's normalized 8-point algorithm, the factorized 8-point algorithm is also completely linear. But unlike the normalized 8-point algorithm, the factorized 8-point algorithm does not need any pre-normalization step. Since every element of the measurement matrix in the factorized 8-point algorithm is a measurement datum or a constant, no amplification of measurement error is involved; the factorized 8-point algorithm can boost effectively the robustness of the estimation. Large numbers of experiments show that the factorized 8-point algorithm consistently outperforms the traditional 8-point algorithm. In addition, although the factorized 8-point algorithm is specially designed for fundamental matrix estimation, its basic principle can be generalized to other estimation problems in computer vision, such as camera projection matrix estimation, homography estimation, focus of expansion estimation, and trifocal tensor estimation.

In this paper we propose and study a simple trinocular rectification method in which stratification to projective and affine components gives the rectifying homographies in a closed form. The class of trinocular rectifications which has 6 DOF is parametrized by an independent set of parameters with a geometric meaning. This offers the possibility to minimize rectification distortion in a natural way. It is shown experimentally on real data that our algorithm performs the rectification task correctly. As shown on groundtruth data using Confidently Stable Matching, trinocular matching significantly improves disparity map density and mismatch error, both depending on texture strength. Matching results on real complex scenes are reported. 1

In the literature, the PnP problem is indistinguishably defined as either to determine the distances of the control points from the camera's optical center or to determine the transformation matrices from the object-centered frame to the camera-centered frame. We show that these two definitions are generally not equivalent. In particular, we prove that, if the four control points are not coplanar, the upper bound of the P4P problem under the distance-based definition is 5 and also attainable, whereas the upper bound of the P4P problem under the transformation-based definition is only 4. Finally, we study the conditions under which at least two, three, four, and five different positive solutions exist in the distance based noncoplanar P4P problem.