Image based modeling and laser scanning are two commonly used approaches in large-scale architectural scene reconstruction nowadays. In order to generate a complete scene reconstruction, an effective way is to completely cover the scene using ground and aerial images, supplemented by laser scanning on certain regions with low textures and complicated structures. Thus, the key issue is to accurately calibrate cameras and register laser scans in a unified framework. To this end, we proposed a three-step pipeline for complete scene reconstruction by merging images and laser scans. First, images are captured around the architecture in a multiview and multiscale way and are feed into a structure-from-motion (SfM) pipeline to generate SfM points. Then, based on the SfM result, the laser scanning locations are automatically planned by considering textural richness, structural complexity of the scene and spatial layout of the laser scans. Finally, the images and laser scans are accurately merged in a coarse-to-fine manner. Experimental evaluations on two ancient Chinese architecture datasets demonstrate the effectiveness of our proposed complete scene reconstruction pipeline.

Most existing binary descriptors are only based on intensity and simply compare averaged intensities of one sample pair to obtain binary output. To address this issue, we propose a novel ring-sampling pattern based binary descriptor encoding both intensity and gradient. For intensity coding, a ring-sampling pattern is presented to define a number of sample points and their neighboring points given an interest point. The intensity difference between two sample points is obtained by performing the comparisons of the intensities of their corresponding neighboring points directly. Further, a majority based voting strategy is employed to obtain compact representation (1 bit or 3 bits) based on all these intensity difference among neighboring points. As for gradient coding, the gradient orientation histogram is computed for each sample point, and the gradient difference of two sample points is obtained by comparing the gradient magnitude on each orientation bin. The raw binary descriptor is constructed by concentrating the intensity and gradient differences of all the sample pairs, and two feature selection strategies are proposed to obtain the final compact descriptor, named as Features Combined Binary Descriptor based on Voted Ring-Sampling Pattern (BDVRP). The experimental results on the tasks of object recognition and image matching demonstrate the superiority and the effectiveness of the proposed descriptor, with comparison to the state-of-the-art hand-crafted binary descriptors.

Social influence analysis in microblogging services has attracted much attention in recent years. However, most previous studies were focused on measuring users' (topical) influence. Little effort has been made to discern and quantify how a user is influenced. Specifically, the fact that user i retweets a tweet from author j could be either because i is influenced by j (i.e., j is a topical authority), or simply because he is “influenced” by the content (interested in the content). To mine such influence patterns, we propose a novel Bayesian factorization model, dubbed Influence Beta-Poisson Factorization (IBPF). IBPF jointly factorizes the retweet data and tweet content to quantify latent topical factors of user preference, author influence and content influence. It generates every retweet record according to the sum of two causing terms: one representing author influence, and the other one derived from content influence. To control the impact of the two terms, for each user IBPF generates a probability for each latent topic by Beta distribution, indicating how strongly the user cares about the topical authority of the author. We develop an efficient variational inference algorithm for IBPF. We demonstrate the efficacy of IBPF on two public microblogging datasets.

This paper performs a comprehensive and comparative evaluation of the state-of-the-art local features for the task of image-based 3D reconstruction. The evaluated local features cover the recently developed ones by using powerful machine learning techniques and the elaborately designed handcrafted features. To obtain a comprehensive evaluation, we choose to include both float type features and binary ones. Meanwhile, two kinds of datasets have been used in this evaluation. One is a dataset of many different scene types with groundtruth 3D points, containing images of different scenes captured at fixed positions, for quantitative performance evaluation of different local features in the controlled image capturing situation. The other dataset contains Internet scale image sets of several landmarks with a lot of unrelated images, which is used for qualitative performance evaluation of different local features in the free image collection situation. Our experimental results show that binary features are competent to reconstruct scenes from controlled image sequences with only a fraction of processing time compared to using float type features. However, for the case of a large scale image set with many distracting images, float type features show a clear advantage over binary ones. Currently, the most traditional SIFT is very stable with regard to scene types in this specific task and produces very competitive reconstruction results among all the evaluated local features. Meanwhile, although the learned binary features are not as competitive as the handcrafted ones, learning float type features with CNN is promising but still requires much effort in the future.

Pairwise image matching plays a vital role in Structure-from-Motion (SfM). Though the image-retrieval method accelerates the matching process, the number of neighbors is usually hard to determine. Insufficient feature matches could break the completeness of reconstructed scene, while redundant pairs may bring in many erroneous ones. In this paper, we propose a progressive SfM method to tackle the completeness, robustness and efficiency problems in a united framework, where two loops are contained. The outer loop is a feature matching loop, where the orthogonal MSTs (maximum spanning trees) of the image similarity graph is iteratively selected to perform the image matching. The inner loop is an incremental camera calibration loop, where the initial camera poses in each iteration are inherited from those calibrated in the last one. By progressively performing the image matching and calibration, we find both loops converge fast and a large number of redundant pairs are excluded. Experiments demonstrate the superior performance of our method in terms of both efficiency and robustness on various image datasets, and our method also has a large potential to tackle the ambiguity problems in SfM.

Multi-view data is common in real-world datasets, where different views describe distinct perspectives. To better summarize the consistent and complementary information in multi-view data, researchers have proposed various multi-view representation learning algorithms, typically based on factorization models. However, most previous methods were focused on shallow factorization models which cannot capture the complex hierarchical information. Although a deep multi-view factorization model has been proposed recently, it fails to explicitly discern consistent and complementary information in multi-view data and does not consider conceptual labels. In this work we present a semi-supervised deep multi-view factorization method, named Deep Multi-view Concept Learning (DMCL). DMCL performs nonnegative factorization of the data hierarchically, and tries to capture semantic structures and explicitly model consistent and complementary information in multi-view data at the highest abstraction level. We develop a block coordinate descent algorithm for DMCL. Experiments conducted on image and document datasets show that DMCL performs well and outperforms baseline methods.

For images with distortions or repetitive patterns, the existing matching methods usually work well just on one of the two kinds of images. In this paper, we present novel triangle guidance and constraints (TGC)-based feature matching method, which can achieve good results on both kinds of images. We first extract stable matched feature points and combine these points into triangles as the initial matched triangles, and triangles combined by feature points are as the candidates to be matched. Then, triangle guidance based on the connection relationship via the shared feature point between the matched triangles and the candidates is defined to find the potential matching triangles. Triangle constraints, specially the location of a vertex relative to the inscribed circle center of the triangle, the scale represented by the ratio of corresponding side lengths of two matching triangles and the included angles between the sides of two triangles with connection relationship, are subsequently used to verify the potential matches and obtain the correct ones. Comparative experiments show that the proposed TGC can increase the number of the matched points with high accuracy under various image transformations, especially more effective on images with distortions or repetitive patterns due to the fact that the triangular structure are not only stable to image transformations but also provides more geometric constraints.

Most color images obtained from the real world usually contain complex areas, such as nature scene images, remote sensing images, and medical images. All these type of images are very difficult to be separated accurately and automatically for complex color and structures included. In this paper, we focus on detecting hybrid cues of color image to segment complex scene in a bottom-up framework. The main idea of the proposed segmentation method is based on a two-step procedure: 1) a reasonable superpixels computing method is conducted and 2) a Mumford-Shah (M-S) optimal merging model is proposed for presegment suerpixels. First, a set of seed pixels is positioned at the lowest texture energy map computed from structure tensor diffusion features. Next, we implement a growing procedure to extract superpixels from selected seed pixels with color and texture cues. After that, a color-texture histograms feature is defined to measure similarity between regions, and an M-S optimal merging process is executed by comparing the similarity of adjacent regions with standard deviation constraints to get final segmentation. Extensive experiments are conducted on the Berkeley segmentation database, some remote sensing images, and medical images. The results of experiments have verified that the segmentation effectiveness of the proposed method in segmenting complex scenes and indicated that it is more robust and accurate than conventional methods.

This study investigates the problem of constructing binary descriptor and develops a novel binary descriptor called simple tri‐bit binary descriptor (STBD) based on a simple sampling pattern (SSP) and a tri‐value binarisation strategy (TBS). First, an SSP is proposed, in which sample points are divided into two groups according to the distance from the pattern centre and smoothed by different circular filters. Then, to make the descriptor adaptive to the matched images, a selection strategy which directly employs detected keypoints as training data is introduced to select 256 point pairs with low correlation from initial pairs. Finally, a modified TBS method is presented to properly refine intensity comparison results. Experiments show that the proposed STBD can perform well and is robust to various transformations, except for scale change.

Image distortions and repetitive patterns widely exist in real images, which results in that feature matching is still a challenging problem though great progress has been made recently. This study presents a matching method, called guidance‐constraints method (GCM), which has obvious advantages in resolving problems of matching features on images with image distortions or repetitive patterns. In GCM, feature points are paired and connection compatibility is introduced to describe the relative geometric relations among features, and then potential matches are found by using the defined geometric guidance and are verified by using the defined geometric constraints. Experimental evaluation shows that the proposed GCM can significantly improve both the number of correct matches and correct ratio under various image transformations, especially more effective on images with distortions or containing repetitive patterns.

On the basis of feature points pairing, a scale‐invariant feature matching method is proposed in this study. The distance between two features is used to compute feature pairs' support region size, which is different from the methods using detectors to provide information to find the support region. Moreover, to achieve rotation invariance, a sub‐region division method based on intensity order is introduced. For comparison to the popular descriptors scale‐invariant feature transform and speeded‐up robust features, the authors also choose the detected points by difference of Gaussian and fast Hessain detectors as feature points to start the authors' method. Additional experiments compare the reported method with similar proposed methods, such as Tell's and Fan's. The experimental results show that the authors' proposed descriptor outperforms the popular descriptors under various image transformations, especially on images with scale and viewpoint transformations.

This study presents a novel texture descriptor for curve matching, called ternary contextualised histogram pattern (TCHP), which is based on the intensity order histogram and local ternary homogeneity patterns. Local ternary homogeneity patterns are firstly adopted to represent the texture features of the curve's neighbourhood, which ensures the robustness and distinctiveness of the descriptor. The proposed TCHP is constructed by three steps: firstly, the curve support region without assigning a dominant orientation is determined and then partitioned into several ordinal bins according to the intensity permutation; then, ternary contextualised histogram (TCH) feature of each point is generated by computing the statistics of the predefined ternary homogeneity patterns; finally, TCHP is achieved by accumulating the TCHs of points in each order bin. Experiments show TCHP can effectively characterise the texture features of the curve's neighbourhood and performs robust to image rotation, viewpoint change and illumination change.

Feature description for local image patch is widely used in computer vision. While the conventional way to design local descriptor is based on expert experience and knowledge, learning-based methods for designing local descriptor become more and more popular because of their good performance and data-driven property. This paper proposes a novel data-driven method for designing binary feature descriptor, which we call receptive fields descriptor (RFD). Technically, RFD is constructed by thresholding responses of a set of receptive fields, which are selected from a large number of candidates according to their distinctiveness and correlations in a greedy way. Using two different kinds of receptive fields (namely rectangular pooling area and Gaussian pooling area) for selection, we obtain two binary descriptors RFDR and RFDG .accordingly. Image matching experiments on the well-known patch data set and Oxford data set demonstrate that RFD significantly outperforms the state-of-the-art binary descriptors, and is comparable with the best float-valued descriptors at a fraction of processing time. Finally, experiments on object recognition tasks confirm that both RFDR and RFDG successfully bridge the performance gap between binary descriptors and their floating-point competitors.

Local image descriptors are one of the key components in many computer vision applications. Recently, binary descriptors have received increasing interest of the community for its efficiency and low memory cost. The similarity of binary descriptors is measured by Hamming distance which has equal emphasis on all elements of binary descriptors. This paper improves the performance of binary descriptors by learning a weighted Hamming distance for binary descriptors with larger weights assigned to more discriminative elements. What is more, the weighted Hamming distance can be computed as fast as the Hamming distance on the basis of a pre-computed look-up-table. Therefore, the proposed method improves the matching performance of binary descriptors without sacrificing matching speed. Experimental results on two popular binary descriptors (BRIEF [1] and FREAK [2]) validate the effectiveness of the proposed method.

Curve matching plays an important role in pattern recognition, computer vision and image understanding. In several past years, this problem has been studied mainly based on the curve contour, while only little progress has been made using the texture feature of the curve's neighborhood. This paper develops a novel texture-based curve matching method called IOCD, which consists of three steps: (1) Curve support region (CSR) without assigning a dominant orientation is first determined; (2) CSR is equally partitioned into several order bins according to the overall intensity order; (3) The feature vector is computed based on the local intensity order mapping. Experiments prove that the proposed IOCD performs robust to image rotation, viewpoint change, illumination change, blur, noise and JPEG compress. The application of image mosaic further identifies IOCD can achieve good matching performance.

Line matching plays an important role in many applications, such as image registration, 3D reconstruction, object recognition and video understanding. However, compared with other features (such as point, region) matching, it has made little progress in recent years. In this paper, we investigate the problem of matching line segments automatically only from their neighborhood appearance, without any constraints or priori knowledge. A novel line descriptor called HLD descriptor is proposed for this purpose, which is constructed by the following three steps: (1) Line support region (LSR) is defined, then divided into a series of overlapped sub-regions with the same size; (2) Line description matrix (DM) is formed by characterizing each sub-region into a vector; (3) HLD descriptor is built by computing mean and standard deviation of DM column vectors. Experimental results show that HLD descriptor is highly distinctive and very robust for line matching on real images.