We address the challenges of local feature matching under large scale and rotation changes by focusing on keypoint positions. First, we propose a novel module called similarity normalization (SN). This module normalizes keypoint positions to remove translation, rotation and scale differences between image pairs. By performing positional encoding on these normalized positions, a network incorporating with SN can effectively avoid encoding largely different positions into descriptors from the two images. Second, we apply strong data augmentation (DA) that includes large scale and rotation, whereas existing matchers ignore such DA and are overfitted to upright image pairs. In our experiments, SN and DA improve the performance for image pairs with large scale and rotation differences. Additionally, the combination of SN with DA leads to further performance improvements.

We propose a novel pipeline called Loc-PL that uses both points and lines for indoor visual localization in dense colored point cloud. Loc-PL utilizes the spatially complementary relationship between points and lines to address challenging indoor issues. There are two successive camera pose estimation modules. The first improves robustness against repetitive patterns by considering the geometric consistency of points and lines. The second utilizes points and lines to refine poses by Perspective-m-Point-n-Line (PmPnL) and circumvents unstable localization due to locally concentrated matches caused by less-textured environments. The modules use different schemes to obtain line correspondences; the first finds line matches using RANSAC, which is effective for image pairs with large viewpoint gaps, and the second utilizes rendered images from dense point cloud to get them by feature line matching. In addition, we develop a simple but effective module for evaluating the correctness of camera poses using matched point distances across two images. The experimental results on a large dataset, InLoc, show that Loc-PL achieves the state-of-the-art in four out of six scores.

This paper proposes minimal solutions to uncalibrated two-view geometry with known epipoles. Exploiting the epipoles, we can reduce the number of point correspondences needed to find the fundamental matrix together with the intrinsic parameters: the focal length and the radial lens distortion. We define four cases by the number of available epipoles and unknown intrinsic parameters, then derive a closed-form solution for each case formulated as a higher-order polynomial in a single variable. The proposed solvers are more numerically stable and faster by orders of magnitude than the conventional 6- or 7-point algorithms. Moreover, we demonstrate by experiments on the human pose dataset that the proposed method can solve two-view geometry even with 2D human pose, of which point localization is noisier than general feature point detectors.

This paper presents an action segmentation method utilizing multiple features on the basis of a novel intermediate fusion module, named Mutual Cross Fusion Module (MCFM). The proposed method analyzes multiple features on each feature’s classifier stream. MCFM recalibrates the target feature in the middle of the classifier stream from the knowledge of the other features in contrast to the existing module of the previous method, which utilizes a joint representation learned from all features for the recalibration of the target feature. MCFM integrates the knowledge of multiple features without biassing the knowledge toward one of the multiple features. We compare the proposed method with the state-of-the-art methods on two public datasets: GTEA and 50Salads. The proposed method outperforms the state-of-the-art methods in terms of frame-wise accuracy, edit distance, and F1-score by 2.0, 1.6, and 2.9 points, respectively.

In this paper, we introduce a new video dataset for action segmentation, the BRIO-TA (BRIO Toy Assembly) dataset, which is designed to simulate operations in factory assembly. In contrast with existing datasets, BRIO-TA consists of two types of scenarios: normal work processes and anomalous work processes. Anomalies are further categorized into incorrect processes, omissions, and abnormal durations. The subjects in the videos are asked to perform either normal work or one of the three anomalies, and all video frames are manually annotated into 23 action classes. In addition, we propose a new metric called anomaly section accuracy (ASA) for evaluating the detection accuracy of anomalous segments in a video. With the new dataset and metric, we report that the state-of-the-art methods show a significantly low ASA, while they work for normal work segments. Demo videos are available at https://github.com/Tarmo-moriwaki/BRIO-TA_sample and the full dataset will be released after publication.

This paper proposes a new algebraic constraint for the planar homography estimation to ensure transformations between two convex quadrilaterals. The new constraint is derived by utilizing a projective invariance of an ellipse, i.e. an ellipse is projected as an ellipse in other views under a physically plausible homography. The invariance is expressed by a quadratic inequality about a homography matrix, therefore, the quadratic constraint can be incorporated with a direct linear method that can be solved as a generalized eigenvalue problem. We demonstrate by experiments that both LO-RANSAC and M-estimator with the proposed constraint are more accurate and robust to outliers than LO-RANSAC with the standard 4-point DLT method.

In this paper, we propose a camera calibration method for surveillance cameras that uses the image projection of parallel 3D line segments of the same length. We assume that vertical line segments are perpendicular to the ground plane and their bottom end points are on the ground plane. Under this assumption, the camera parameters can be directly determined from at least two line segments without estimating vanishing points. By extending the minimal solution, we devise a closed-form solution to the least squares case with more than two line segments. Lens distortion is jointly optimized in bundle adjustment. Evaluation of synthetic data showed that the optimal depression angle of a camera is around 50 degrees. In real data evaluation, we used the joints of pedestrians as vertical line segments. The experimental results on public datasets showed that the proposed method used with a human pose detector can accurately calibrate wide-angle cameras that have radial distortion.

This paper proposes a novel fast and robust homography estimation by adaptively controlling the threshold of graduated non-convexity (GNC). Based on the fact that GNC is a variant of deterministic annealing, we provide a new method for updating the inlier threshold at each GNC iteration by utilizing the statistical properties of residuals of potential inliers. Contrary to RANSAC, our approach gives the same unique parameter for a single input due to without random sampling. Moreover, computational time increases linearly against outlier ratio changes, whereas RANSAC increases exponentially. Synthetic data evaluation shows that the proposed method is more robust and faster than RANSAC for highly contaminated data containing more than 80% outliers. Additionally, we demonstrate that our method works on severe real images that the state-of-the-art RANSAC method fails.

This paper proposes a novel sampling method for accelerating RANSAC family on 2D homography estimation. From the initial set of matched points, the proposed method generates a promising reduced subset having higher inlier ratio than the initial set by utilizing pixel values of the matches. Regarding pairs of the pixel value as two dimensional scattered points, we estimate the global brightness consistency of the pixel values. Then, points that violate the global brightness consistency are removed from the initial point set. Incorporating the proposed method with RANSAC and USAC, we demonstrate that the number of iterations and computational time are both significantly reduced by orders of magnitude while maintaining accuracy of homography estimation.

Crack development in concrete structures starts at the micro-crack stage and proceeds to the macro-crack stage due to repeated cyclic loading, like ongoing vehicles on bridges. Automatic detection of early stage cracks is required for both safety and economic reasons. We present an automatic crack detection method that scans a captured concrete area and provides a pixel-wise localization of both visible macro-cracks and early stage micro-cracks from video sequences. The key component in the proposed method is a spatial-temporal non-linear filtering on framewise dense 2D motion field combined with Conditional Random Fields based crack localization refinement. We evaluate our method against labeled ground truth data provided by an expert crack inspector. Experimental results show that our method can produce high accuracy automatic crack localization having F1 score improvement of 0.14-0.22 compared to conventional image based detectors. The proposed method is also shown to detect cracks at an earlier stage which enables early preventive measures for repair operations.

The perspective-n-point (PnP) problem, which estimates 3D rotation and translation of a calibrated camera from n pairs of known 3D points and corresponding 2D image points, is a classical problem but still fundamental in the computer vision community. It is well studied that the PnP problem can be solved by at least three points [1]. If n ≥ 4, the PnP problem becomes a nonlinear problem where the number of the solutions depend on n and the shape of the scene. This paper proposes an efficient, scalable, and globally optimal DLS method parameterized by Cayley representation, which has been regarded as a unsuitable parametrization due to its singularity. First we derive a new optimality condition without Lagrange multipliers. Letting pi = [xi,yi,zi] be an i-th 3D point and mi = [ui,vi,1] be the corresponding calibrated image point in homogeneous coordinates, the PnP problem can be formulated as a nonlinear optimization

When human cannot perceive the inconsistency of artificial shadows which are not physically correct, they are acceptable as “perceptually-correct” shadows. This paper focuses on the simplification of light-source models for generating the perceptually-correct artificial shadows. First, we conducted subjective evaluations to obtain knowledge about the human perception of the shadows. Then the knowledge was applied to control the resolution of the light-source map to generate perceptually-correct artificial shadows. Comparative studies among artificial and real shadows justified perceptually correctness. All experiments were done using still images, not videos. Our research becomes a reference to determine the resolution of light-source map in an MR scene.