Techniques for identifying potholes on road surfaces aim at developing strategies for real-time or offline identification of potholes, to support real-time control of a vehicle (for driver assistance or autonomous driving) or offline data collection for road maintenance. For these reasons, research around the world has comprehensively explored strategies for the identification of potholes on roads. This paper starts with a brief review of the field; it classifies developed strategies into several categories. We, then, present our contributions to this field by implementing strategies for automatic identification of potholes. We developed and studied two techniques based on stereo-vision analysis of road environments ahead of the vehicle; we also designed two models for deep-learning-based pothole detection. An experimental evaluation of those four designed methods is provided, and conclusions are drawn about particular benefits of these methods.

This paper highlights the role of ground manifold modeling for stixel calculations; stixels are medium-level data representations used for the development of computer vision modules for self-driving cars. By using single-disparity maps and simplifying ground manifold models, calculated stixels may suffer from noise, inconsistency, and false-detection rates for obstacles, especially in challenging datasets. Stixel calculations can be improved with respect to accuracy and robustness by using more adaptive ground manifold approximations. A comparative study of stixel results, obtained for different ground-manifold models (e.g., plane-fitting, line-fitting in v-disparities or polynomial approximation, and graph cut), defines the main part of this paper. This paper also considers the use of trinocular stereo vision and shows that this provides options to enhance stixel results, compared with the binocular recording. Comprehensive experiments are performed on two publicly available challenging datasets. We also use a novel way for comparing calculated stixels with ground truth. We compare depth information, as given by extracted stixels, with ground-truth depth, provided by depth measurements using a highly accurate LiDAR range sensor (as available in one of the public datasets). We evaluate the accuracy of four different ground-manifold methods. The experimental results also include quantitative evaluations of the tradeoff between accuracy and run time. As a result, the proposed trinocular recording together with graph-cut estimation of ground manifolds appears to be a recommended way, also considering challenging weather and lighting conditions.

Currently, self-localization of vehicles is primarily carried out by GPS receivers of different price and performance classes, which are permanently installed in vehicles. Known limitations of GPS-based localization include signal disturbances due to multipath propagation, as it may occur in narrow street canyons or tree alleys, as well as signal losses in tunnels or entrances to buildings, for example. It may happen that horizontal GPS channels still support a useful 2D positioning with low uncertainties while the GPS altitude channel has very large deviations or error uncertainties. Additional sensors (IMU or cameras) help to detect sections of trajectories where the GPS signal becomes unusable. Based on a traditional navigation strategy (i.e. integration of IMU and GPS), we also study the possible integration of visual odometry obtained from a stereo camera system. The paper reports about a large-scale project studying multi-sensor integration for very accurate and robust self-localization of vehicles.

We propose the use of a reliable confidence map for multi-layer stixel segmentation; our confidence map uses a calibrated collinear trinocular vision model. It is generated from three conjugate synchronized stereo images for evaluating the consistency of disparity values. The evaluation measure is referred to as transitivity error in disparity space. Multi-layer stixels are commonly generated from a single disparity map which make them merely dependent on the applied stereo matcher. A multi-map fusion is proposed to achieve more reliable stixel segmentation for disparity values. Moreover, another advantage of our work is to provide a new and effective ground-detection technique (ground-manifold detection) which benefits from the confidence map. Experimental results show a significant improvement on average of 12.6% using our method compared with conventional stixels detected by binocular vision only.

Fairy circles are circular patches of barren soil forming large clusters in the arid grasslands of Southern Africa (especially in Namibia) and Western Australia. Fairy circles are clearly visible in aerial images shown in applications such as Google Maps, and they can be recorded using sensors mounted on drones in very high image and video resolution for ecological studies aiming at understanding the origin of these patterns. Traditional analysis of fairy circles is done by manual digitising and counting. We also showed recently that, despite being challenging, traditional computer vision methods enabled the detection of fairy circles. To improve fairy circle detection and localization automatically in aerial images, we here present the use of a convolutional neural network (CNN). The results suggest that new methods using CNNs outperform other methods in terms of accuracy.

The paper proposes a new single image dehazing method based on a convolutional neural network. Our method directly learns an end to end mapping between the haze images and their corresponding haze layers (i.e. residual images between haze images and non-haze images). A convolutional neural network takes the haze image as an input and the residual image as an output. Then, a recovered dehazed image can be obtained by removing the residual from the haze image. Residual learning allows the network to directly estimate the initial haze layer with relatively high learning rates, which reduce computational complexity and speed-up the convergence process. Since the initial haze layer is only approximate, we use a guided filter to refine this layer to avoid halos and block artefacts, which makes the recovered image more similar to a real scene. The algorithms are tested on fog images with different fog densities. Comparisons are provided with other dehazing algorithms. Experiments demonstrate that the proposed algorithm outperforms state-of-the-art methods on both synthetic and real-world images, qualitatively and quantitatively.

This article is published with

This paper proposes a fast and reliable method for anomaly detection and localization in video data showing crowded scenes. Time-efficient anomaly localization is an ongoing challenge and subject of this paper. We propose a cubicpatch- based method, characterised by a cascade of classifiers, which makes use of an advanced feature-learning approach. Our cascade of classifiers has two main stages. First, a light but deep 3D auto-encoder is used for early identification of "many" normal cubic patches. This deep network operates on small cubic patches as being the first stage, before carefully resizing remaining candidates of interest, and evaluating those at the second stage using a more complex and deeper 3D convolutional neural network (CNN). We divide the deep autoencoder and the CNN into multiple sub-stages which operate as cascaded classifiers. Shallow layers of the cascaded deep networks (designed as Gaussian classifiers, acting as weak single-class classifiers) detect "simple" normal patches such as background patches, and more complex normal patches are detected at deeper layers. It is shown that the proposed novel technique (a cascade of two cascaded classifiers) performs comparable to current top-performing detection and localization methods on standard benchmarks, but outperforms those in general with respect to required computation time.

This paper considers robust vehicle logo recognition (without aiming at accurate location) for intelligent transportation systems. We propose a recognition-before-location framework for multi-scale vehicle logos which exploits a directional SIFT flow parsing method. We extract dense SIFT descriptors of different standard vehicle logos. An improved matching method is proposed to obtain a directional SIFT flow from standard logo models for vehicle images. Our vehicle logo recognition algorithm is based on dense SIFT matching energy and SIFT flow consistency. We verify the accuracy of vehicle logo recognition and the robustness for multi-scale logo images on various real data.

In this article a new strategy for single-image super-resolution is proposed. A selective sparse coding strategy based on patch sharpness is assumed to be invariant for patch resolution. This sharpness criterion is used at training stage to classify image patches into different clusters. It is suggested that the use of coupled dictionary learning, with a mapping function can improve the representation quality. By this strategy clustered dictionaries are designed along with a mapping function for each cluster which can provide the coupling link between low-resolution and high-resolution image patches. During the reconstruction, image patch sharpness is used as a criterion for the selection of a clustered dictionary and the mapping function. The high-resolution patches are recovered by high-resolution cluster dictionary atoms and the mapping function with sparse representation coefficients from low resolution patches. The algorithm is tested over a set of benchmark images from different data sets. Peak-signal-to-noise ratio and structural-similarity-index measures indicate that the given algorithm is competitive in general with existing baseline algorithms. The proposed algorithm performs better for images with high-frequency components.

Recently LiDAR-camera systems have rapidly emerged in many applications. The integration of laser range-finding technologies into existing vision systems enables a more comprehensive understanding of 3D structure of the environment. The advantage, however, relies on a good geometrical calibration between the LiDAR and the image sensors. In this paper we consider visual odometry, a discipline in computer vision and robotics, in the context of recently emerging online sensory calibration studies. By embedding the online calibration problem into a LiDAR-monocular visual odometry technique, the temporal change of extrinsic parameters can be tracked and compensated effectively.

In compressive tracking algorithms, a feature reduction projection matrix is constructed by using compressed sensing theory. Target and non-target objects are discriminated by using a naive Bayesian classifier. Such an algorithm may ensure accuracy of target tracking in real-time. But it is not adaptive for tracking with respect to scales and rotations. In this paper, we propose a novel adaptive algorithm based on feature point matching for tracking objects which appear with various changes. We combine weight-average and improved compressive tracking algorithms together for tracking objects, then calculate the corresponding feature points between two subsequent frames of the same object for obtaining the target changes related to various scales and rotations. Our experimental results show that the improved algorithm effectively improves the accuracy of target tracking and ensures adaptability of the tracking algorithm.

The Hough transform is a popular technique used in the field of image processing and computer vision. With a Hough transform technique, not only the normal angle and distance of a line but also the line-segment's length and midpoint (centroid) can be extracted by analysing the voting distribution around a peak in the Hough space. In this paper, a method based on minimum-entropy analysis is proposed to extract the set of parameters of a line segment. In each column around a peak in Hough space, the voting values specify probabilistic distributions. The corresponding entropies and statistical means are computed. The line-segment's normal angle and length are simultaneously computed by fitting a quadratic polynomial curve to the voting entropies. The line-segment's midpoint and normal distance are computed by fitting and interpolating a linear curve to the voting means. The proposed method is tested on simulated images for detection accuracy by providing comparative results. Experimental results on real-world images verify the method as well. The proposed method for line-segment detection is both accurate and robust in the presence of quantization error, background noise, or pixel disturbances.

Avoiding high computational costs and calibration issues involved in stereo-vision-based algorithms, this paper proposes real-time monocular-vision-based techniques for simultaneous vehicle detection and inter-vehicle distance estimation, in which the performance and robustness of the system remain competitive, even for highly challenging benchmark datasets. This paper develops a collision warning system by detecting vehicles ahead and, by identifying safety distances to assist a distracted driver, prior to occurrence of an imminent crash. We introduce adaptive global Haar-like features for vehicle detection, tail-light segmentation, virtual symmetry detection, intervehicle distance estimation, as well as an efficient single-sensor multifeature fusion technique to enhance the accuracy and robustness of our algorithm. The proposed algorithm is able to detect vehicles ahead at both day or night and also for short- and long-range distances. Experimental results under various weather and lighting conditions (including sunny, rainy, foggy, or snowy) show that the proposed algorithm outperforms state-of-the-art algorithms.

The paper proposes an advanced driver-assistance system that correlates the driver's head pose to road hazards by analyzing both simultaneously. In particular, we aim at the prevention of rear-end crashes due to driver fatigue or distraction. We contribute by three novel ideas: Asymmetric appearance-modeling, 2D to 3D pose estimation enhanced by the introduced Fermat-point transform, and adaptation of Global Haar (GHaar) classifiers for vehicle detection under challenging lighting conditions. The system defines the driver's direction of attention (in 6 degrees of freedom), yawning and head-nodding detection, as well as vehicle detection, and distance estimation. Having both road and driver's behaviour information, and implementing a fuzzy fusion system, we develop an integrated framework to cover all of the above subjects. We provide real-time performance analysis for real-world driving scenarios.

Current research on traffic flow prediction mainly concentrates on generating accurate prediction results based on intelligent or combined algorithms but ignores the interpretability of the prediction model. In practice, however, the interpretability of the model is equally important for traffic managers to realize which road segment in the road network will affect the future traffic state of the target segment in a specific time interval and when such an influence is expected to happen. In this paper, an interpretable and adaptable spatiotemporal Bayesian multivariate adaptive-regression splines (ST-BMARS) model is developed to predict short-term freeway traffic flow accurately. The parameters in the model are estimated in the way of Bayesian inference, and the optimal models are obtained using a Markov chain Monte Carlo (MCMC) simulation. In order to investigate the spatial relationship of the freeway traffic flow, all of the road segments on the freeway are taken into account for the traffic prediction of the target road segment. In our experiments, actual traffic data collected from a series of observation stations along freeway Interstate 205 in Portland, OR, USA, are used to evaluate the performance of the model. Experimental results indicate that the proposed interpretable ST-BMARS model is robust and can generate superior prediction accuracy in contrast with the temporal MARS model, the parametric model autoregressive integrated moving averaging (ARIMA), the state-of-the-art seasonal ARIMA model, and the kernel method support vector regression.

In this study, we have generalized our previous tool for assisting a safety engineer in assessing collision trajectories by extending from colliding objects with constant velocity to more general variable velocity ones. We have also highlighted that a linear system cannot be relied upon for handling a colliding object with variable velocity. To deal with such trajectories, past observations are weighted depending on velocities at those locations; priority is given to locations with reduced velocity. Based on this hypothesis, we have shown that the weighted system outperforms a linear one. The benefit is that it always issues a timely warning, even if the trajectory of the colliding object keeps on changing over time.

The report about the ECCV 2012 Robust Vision Challenge summarized strengths and weaknesses of the winning stereo matcher (Iterative Semi-Global Matching = iSGM). In this paper we discuss whether two variants of a Belief Propagation (BP) matcher can cope better with events such as sun flare or missing temporal consistency, where iSGM showed weaknesses (similar to the original SGM). The two variants are defined by the use of a census data cost function on a 5 × 5 window and either a linear or a quadratic truncated smoothness function. An evaluation on data with ground truth showed better results for the linear smoothness function. The BP matcher with the linear smoothness function provided then also better matching results (compared to iSGM) on some of the test sequences (e.g. images with sun flare). The third-eye approach was used for the performance evaluation.

Hardware platforms with limited processing power are often incapable of running dense stereo analysis algorithms at acceptable speed. Sparse algorithms provide an alternative but generally lack in accuracy. To overcome this predicament, we present an efficient sparse stereo analysis algorithm that applies a dense consistency check, leading to accurate matching results. We further improve matching accuracy by introducing a new feature detector based on FAST, which exhibits a less clustered feature distribution. The new feature detector leads to a superior performance of our stereo analysis algorithm. Performance evaluation shows that the proposed stereo matching system achieves processing rates above 200 frames per second on a commodity dual core CPU, and faster than video frame-rate processing on a low-performance embedded platform. The stereo matching results prove to be superior to those obtained with ordinary sparse matching algorithms.

Semi-global matching is a popular choice for applications where dense and robust stereo estimation is required and real-time performance is crucial. It therefore plays an important role in vision-based driver assistance systems. The strength of the algorithm comes from the integration of multiple 1D energy paths which are minimized along eight different directions across the image domain. The contribution of this paper is twofold. First, a thorough evaluation of stereo matching quality is performed when the number of accumulation paths is reduced. Second, an alteration of semi-global matching is proposed that operates only on half of the image domain without losing disparity resolution. The evaluation is performed on four real-world driving sequences of 400 frames each, as well as on 396 frames of a synthetic image sequence where sub-pixel accurate ground truth is available. Results indicate that a reduction of accumulation paths is a very good option to improve the run-time performance without losing significant quality, even on sub-pixel level. Furthermore, operating semiglobal matching only on half the image yields almost identical results to the corresponding full path integration. This approach yields the potential to further speed up the runtime and could also be exploited for other alterations of the algorithm.

In this study we examine three road-modeling methods, which we evaluate on seven stereo matching algorithms. The road-modeling methods we consider are a B-spline modeling technique based on region-growing and two versions of the popular v-disparity approach. The used stereo algorithms are variations or different parameterizations of belief propagation, graph cut and semi-global matching. We examine the performance of the possible combinations of modeling methods and stereo algorithms by comparing the deviation towards a reference road profile. Two evaluation sequences are used, of which one features switched on windscreen wipers that are visible in the the recorded imagery. Our findings are that the examined B-spline modeling method provides the best results in most cases. In terms of the modeled distance, belief propagation is the most suitable stereo matching method, which also performs best with the wiper sequence. Semi global matching achieves smaller model deviations, but with drastically reduced modeled distances.

A significant component of driver assistance systems (DAS) is lane detection, and has been studied since the 1990s. However, improving and generalizing lane detection solutions proved to be a challenging task until recently. A (physical) lane is defined by road boundaries or various kinds of lane marks, and this is only partially applicable for modeling the space an ego-vehicle is able to drive in. This paper proposes a concept of (virtual) corridor for modeling this space. A corridor depends on information available about the motion of the ego-vehicle, as well as about the (physical) lane. This paper also suggests a modified version of Euclidean Distance Transform (EDT), named Row Orientation Distance Transform (RODT), to facilitate the detection of corridor boundary points. Then, boundary selection and road patch extension are applied as post-processing. Moreover, this paper also informs about the possible application of corridor for driver assistance. Finally, experiments using images from highways and urban roads with some challenging road situations are presented, illustrating the effectiveness of the proposed corridor detection algorithm. Comparison of lane and corridor on a public dataset is also provided.

The support vector machine (SVM) of statistical learning theory was successfully applied in various fields, but still suffers from noise sensitivity originating from the fact that all the data points are treated equally. To relax this problem, the SVM was extended into a fuzzy SVM (FSVM) by the introduction of fuzzy memberships. The FSVM also has been further extended in two ways, by adopting a different objective function with the help of domain-specific knowledge, or by employing a different membership calculation method. In this paper we follow the second approach by proposing a new membership calculation method using a fuzzy k nearest neighbor classifier (F-KNN). Although there are already several membership calculation methods to enhance the performance of the FSVM, one problem in those methods is that they assume a specific data distribution. The F-KNN does not assume any data distribution, which helps the proposed method to accommodate various data distributions in real world problems. The proposed algorithm was applied to an insect footprint classification problem, and results verify the effectiveness of the method.

Panoramic images and panoramic cameras (or sensors) are of increasing importance for various applications in computer vision, computer graphics, visualization, and robotics. Various panoramic image capturing sensors have been developed for different purposes. But many of the sensing devices do not support stereo visualization. This paper reviews a methodology for stereo panorama acquisition using a widely available digital matrix camera. Two sensor parameters, off-axis distance and principle angle, are critical to achieving high quality stereo visualization. In this paper, a new camera calibration method will be presented, which supports accurate estimations of these two essential parameters. To our best knowledge, there is no other camera calibration method published so far for such a purpose.

In this paper we define and study digital manifolds of arbitrary dimension, and provide (in particular)a general theoretical basis for curve or surface tracing in picture analysis. The studies involve properties such as one-dimensionality of digital curves and (n-1)-dimensionality of digital hypersurfaces that makes them discrete analogs of corresponding notions in continuous topology. The presented approach is fully based on the concept of adjacency relation and complements the concept of dimension as common in combinatorial topology. This work appears to be the first one on digital manifolds based ona graph-theoretical definition of dimension. In particular, in the n-dimensional digital space, a digital curve is a one-dimensional object and a digital hypersurface is an (n-1)-dimensional object, as it is in the case of curves and hypersurfaces in the Euclidean space. Relying on the obtained properties of digital hypersurfaces, we propose a uniform approach for studying good pairs defined by separations and obtain a classification of good pairs in arbitrary dimension. We also discuss possible applications of the presented definitions and results.

This paper presents a class of Fourier transform based approaches for height from gradients (HFG) problem, which is to reconstruct the 3D surface height of an object from its gradients. The HFG problem results from quite a few research areas in computer vision such as shape from shading, photometric stereo, shape from texture, shape from contours and so on. The proposed methods have some distinct advantages. Firstly, the derivation process of the algorithms has generality, and can be used for more functional dealing with additional constraints. Secondly, they are noniterative so that boundary conditions are not needed. In addition, their robustness to noisy gradient estimates can be improved by choosing associated weighting parameters. Experimental results using synthetic and real images are also presented.

This paper compares previously published length estimators in image analysis having digitized curves as input. The evaluation uses multigrid convergence (theoretical results and measured speed of convergence) and further measures as criteria. This paper also suggests a new gradient-based method for length estimation, and combines a previously proposed length estimator for straight segments with a polygonalization method.

We consider simple digital curves in a 3D orthogonal grid as special polyhedrally bounded sets. These digital curves model digitized curves or arcs in three-dimensional Euclidean space. The length of such a simple digital curve is defined to be the length of the minimum-length polygonal curve fully contained and complete in the tube of this digital curve. So far, no algorithm was known for the calculation of such a shortest polygonal curve. The paper provides an iterative algorithmic solution for approximating the minimum-length polygon of a given simple digital space-curve. The theoretical foundations of this algorithm are presented as well as experimental results.

We consider simple digital curves in a 3D orthogonal grid as special polyhedrally bounded sets. These digital curves model digitized curves or arcs in three-dimensional euclidean space. The length of such a simple digital curve is defined to be the length of the minimum-length polygonal curve fully contained and complete in the tube of this digital curve. So far no algorithm was known for the calculation of such a shortest polygonal curve. The paper provides an iterative algorithmic solution, including a presentation of its foundations and of experimental results.

This paper compares two techniques for measuring the length of a digital curve. Both techniques (digital straight segment approximation, minimum length polygon) are known to be convergent estimators. Theoretical convergence results are cited. The main focus is on experimental evaluation: several measures are defined, applied and discussed. Test sets are digitized for consecutive resolutions.

We consider estimates of worst-case bounds for quantization errors in calculating moments of arbitrary order. New estimates are provided which are based on Huxley's theorem and a general definition of families of sets, propagating an estimate for zero-order moments towards estimates for moments of arbitrary order.

This paper describes on-going research about integration techniques for 3D surface reconstruction. Our goal is to establish simple and robust 3D models for the growing needs of visual communication. Introducing and analyzing some typical techniques, we propose a new method for generating complete 3D models of complex objects-combining "shape from occluding contours" and photometric stereo techniques.