Object detection and classification are key processes in advanced driver-assistance systems. The existing object detection and classification methods are effective in normal daylight conditions. However, the performance of these methods deteriorates in adverse driving conditions, such as those involving low light, illumination changes, and nighttime conditions. To overcome these limitations, several feature-based algorithms have been developed that introduce local features, such as local binary pattern, local tetra pattern, and local density encoding, for adverse driving conditions. However, these local patterns cannot effectively address the noise in real driving conditions because the relationship between the neighboring pixels cannot be comprehensively encoded. To solve these problems, this study developed a robust feature-based method by introducing a triangular-pattern-based sigmoid function to effectively encode and establish the robust feature of neighboring pixels in the local region. The performance of the proposed pattern is evaluated by integrating it into state-of-the-art object detection algorithms. The proposed method significantly increases the vehicle detection ratio of YOLOv5s by 11.7% for an intersection over a union of 0.5 in difficult driving conditions for the CCD dataset. Moreover, the detection ratios of the proposed method are comparable to those of other state-of-the-art object detection methods such as Retina, Faster RCNN, and Deformable DETR over various datasets such as KITTI, COCO, HCI, and CCD. Additionally, the proposed algorithm is implemented on a Raspberry Pi-based autonomous car system to evaluate its performance during real driving conditions. Our proposed method supports robust input feature extraction and can thus be used to enhance the performance of the existing obstacle detection and classification systems.

Stereo matching is a challenging research topic in driving assistance systems. Existing stereo matching methods work well under normal day-light conditions. However, they fail to operate under adverse driving conditions, such as at night and during snowfall. This paper proposes a robust stereo matching framework using both deep-learning-based features and feature engineering. The proposed method investigates the benefits of features based on feature engineering and deep learning for solving stereo matching problems. Robust feature engineering is proposed for handling specific driving under adverse weather conditions, and a robust feature based on deep learning is considered for handling unspecific driving under extreme weather conditions. The proposed study has shown significantly improved accuracy by 8.31% for the state-of-the-art census based on semi-global matching under the reflection regions using the KITTI Stereo 2012 benchmark. Moreover, the experimental results demonstrate that the proposed system obtains more stable results than existing stereo methods based on deep learning on various stereo datasets, such as the Middlebury, EISAT, HCI, and CCD datasets.

In this paper, we present a deep learning model for high-accuracy, high-speed detection of vehicle taillights in traffic. The model consists of three major modules: the lane detector, the car detector, and the taillight detector. Unlike most previously proposed algorithms where hand-coded schemes are used, we have adopted a data-driven approach. This data-driven scheme was implemented in both the car and taillight detection modules. First, we used an intricately designed lane detection module, then we adopted the Recurrent Rolling Convolution (RRC) architecture and tracking mechanism for detecting car boundaries. Subsequently, we used the same RRC architecture to extract the taillight regions of the detected cars. The lane detection and car detection modules improve both the speed and detection rate of the final taillight detection. The robustness of our model was verified using datasets from Sungkyunkwan University (SKKU) as well as the Karlsruhe Institute of Technology and Toyota Technological Institute (KITTI). Our model works well even in hostile conditions. It achieves detection rates as high as 99% in testing with the SKKU dataset. When using the KITTI 2D Object dataset, the model achieves a taillight detection rate of 86%. The model achieves 100% taillight detection rate on a certain, small subset of the KITTI Tracking dataset.

While it is a hot issue whether cars could drive by themselves in emergent situations without any kind of human interference, pedestrian detection is the key technology in autonomous driving cars. Though current pedestrian detection technologies have come to a point in which they are accurate in normal conditions and surroundings, existent systems are inaccurate in harsh situations, such as when there are too many pedestrians, when there is too much light or when it is too dark, or when it is raining or snowing heavily. This problem may be solved by integrating deep learning and combining a new type of local pattern with the RGB raw image as input, instead of using just the RGB image as input. We will introduce a new type of local pattern called Triangular Patterns, which is effective in extracting more detailed and stable features from local regions. Here in this paper, we propose a pedestrian detection system in which deep learning is used, along with combining the RGB raw image with Triangular Patterns for input.

Traditional deep learning-based vehicle detection methods are often designed using a pyramid of filters with multiple scales and sizes; therefore, the processing time is slow due to the large number of scales used and because the classifier runs at all scales. Recently, a deep learning-based region proposal network was introduced to detect vehicles that only employ the network one time regardless of the size of the input image. In object detection, deep learning-based region proposal networks have achieved state-of-the-art performance in terms of accuracy. These systems achieve a very high accuracy under normal driving conditions; however, their performance decreases under difficult driving conditions such as in snow, rain, or fog. In addition, the current state-of-the-art system-based region proposal networks still fail to satisfy the real-time requirements of the driving assistant systems. More recently, the identification of local patterns has been shown to improve the performance of the traditional deep-learning systems; hence, this paper investigates local patterns in region proposal networks to improve their accuracy. Depth information is also investigated to improve the processing time of current region proposal networks. Our experimental results show that the proposed system obtains better performance than the state-of-the-art object region detection systems in terms of both accuracy and processing time.

Car detection is a crucial issue in self-driving cars. Numerous in-traffic car detection models have been proposed, each of which exhibits its own strengths and weaknesses; the high detection speeds of some models are not accompanied by high precision, while the precision of other models is shadowed by insufficient speeds. Our main goal in this paper is to introduce a model that utilizes the Recurrent Rolling Convolution (RRC). The model gives promising results on detection speed and precision, thereby mitigating the weaknesses of previously proposed models, which is exhibited in our extensive experiment.

Pedestrian detection is fundamental challenge for computer vision which requires localizing objects within an image. Convolutional neural networks are widely used in object recognition. However, ordinal convolutional methods using sliding window as the input for networks require time to run an entire image and can only handle a fixed size window image. We propose to using a region proposal based in the V-disparity method to obtain prospect regions, instead of the original scanning methods to obtain the object regions. The region proposals from the V-disparity will be fed as the input for a convolutional neural network(CNN). We also extend the CNN for more effective task object detection. In our model, CNN are combined with recursive neural networks to learn features and classify color images. The convolutional neural network layer learns low-level features of the input image. By using CNN, the learned features can represent highly variable objects in the input image. The learned features after the convolutional layer are then given as inputs to a recursive neural network (RNN) to compose higher order features. The RNN is a multiple, fixed-tree recursive which can combine convolution and pooling into one efficient hierarchical operation.

The performance of stereo matching algorithms strongly depends on the quality of the stereo data/matching cost. Most state-of-the-art data costs require expert knowledge for the design of a transformation function, such as census for handling gray-level changes monotonically, adaptive normalized cross correlation for handling Lambertian cases, guided filtering for preserving edge information, and local density encoding for handling illumination differences. However, it is difficult to design a complex transformation function to handle unknown factors that often occur in driving conditions such as snow, rain, and sun. Therefore, this paper has investigated the deep learning strategy to develop a novel stereo matching cost model without using much expert knowledge. Experimental results show that the proposed deep learning model obtains better results than the state-of-the-art stereo matching cost as judged by the standard KITTI benchmark, Middlebury, and HCI datasets.

This paper introduces a general framework for detection, recognition, and tracking preceding vehicles and pedestrians based on a deep learning approach. The proposed framework combines a novel deep learning approach with the use of multiple sources of local patterns and depth information to yield robust on-road vehicle and pedestrian detection, recognition, and tracking. The proposed system is first based on robust obstacle detection to identify obstacles appearing along the road that are likely to be vehicles and pedestrians, implemented as an efficient adaptive U-V disparity algorithm. Second, the results from the obstacle detection stage are input into a novel vehicle and pedestrian recognition system based on a deep learning model that processes multiple sources of depth information and local patterns. Finally, the results from the recognition stage are used to track detected vehicles or pedestrians in the next frame by means of a proposed tracking and validation model. The proposed framework has been thoroughly evaluated by inputting several vehicle and pedestrian data sets that were collected under various driving conditions. Experimental results show that this framework provides robust vehicle and pedestrian detection, recognition, and tracking with high accuracy, and also satisfies the real-time requirements of driver assistance systems.

We propose a novel fuzzy encoding pattern that fuzzily encodes the relative orders between pixel pairs. An image window is divided into disjoint neighboring pixel sets for the window's center pixel, and the relative order is established not only between the center pixel and its neighbors but also between the pixel pairs in each neighboring pixel set. The relative orders are fuzzily encoded to extract more detailed information from a local structure. We successfully apply the pattern as a matching cost function for stereo correspondence under severe radiometric variations. We conduct experiments using the proposed matching cost function and compare it with functions employing the census transform, supporting local binary pattern, and adaptive normalized cross correlation, as well as a mutual information-based matching cost function, using different stereo data sets. Compared with the census transform, the proposed function reduces the error from 33.1% to 16.9% in the Middlebury data set and from 17.6% to 9.5% in the Kitti data set. The experimental results indicate that the proposed function is superior to the state-of-the-art functions under radiometric variations. In addition, the proposed function is faster than recently developed functions, such as the adaptive normalized cross correlation, a mutual information-based function, and support local binary pattern.

Real-world stereo images are inevitably affected by radiometric differences, including variations in exposure, vignetting, lighting, and noise. Stereo images with severe radiometric distortion can have large radiometric differences and include locally nonlinear changes. In this paper, we first introduce an adaptive orthogonal integral image, which is an improved version of an orthogonal integral image. After that, based on matching by tone mapping and the adaptive orthogonal integral image, we propose a robust and accurate matching cost function that can tolerate locally nonlinear intensity distortion. By using the adaptive orthogonal integral image, the proposed matching cost function can adaptively construct different support regions of arbitrary shapes and sizes for different pixels in the reference image, so it can operate robustly within object boundaries. Furthermore, we develop techniques to automatically estimate the values of the parameters of our proposed function. We conduct experiments using the proposed matching cost function and compare it with functions employing the census transform, supporting local binary pattern, and adaptive normalized cross correlation, as well as a mutual information-based matching cost function using different stereo data sets. By using the adaptive orthogonal integral image, the proposed matching cost function reduces the error from 21.51% to 15.73% in the Middlebury data set, and from 15.9% to 10.85% in the Kitti data set, as compared with using the orthogonal integral image. The experimental results indicate that the proposed matching cost function is superior to the state-of-the-art matching cost functions under radiometric variation.

Stereo correspondence is challenging under realistic conditions due to uncontrolled factors that affect input images, including illumination inconsistencies and radiometric variations. Many local and global models have been suggested to address these problems; however, their performance is often degraded due to the assumption of color consistency between the left and right images. Therefore, we present a new local pattern, local density encoding, for stereo matching measurements to improve the performance of existing stereo methods. Our experimental results indicate that the proposed method is less sensitive to illumination changes and radiometric variations. Moreover, in the cases with normal and severe illumination changes, the proposed method is more robust than state-of-the-art data costs.

The local binary pattern (LBP) and its variants have been widely investigated in many image processing and computer vision applications due to their robust ability to capture local image structures and their computational simplicity. The existing LBPs extract local structure information by establishing a relationship between the central pixel and its adjacent pixels. However, most LBPs miss the relationship among all of the pixels in the local region. Therefore, this paper proposes a novel model to establish this relationship by introducing a support LBP. The proposed model improves the performance of the existing LBP methods and results in lower sensitivity to illumination changes and radiometric variations. Moreover, the proposed model has been successfully investigated in two applications: disparity map generation and texture classification. For disparity map generation, the proposed model reduces the root mean square (RMS) error by 23.6% (in Baby1 dataset, Middlebury), and 16.58% (in Aloe dataset, Middlebury) as compared with the standard LBP under radiometric variation conditions. Moreover, the proposed model reduces the RMS by 28.11% as compared with the standard LBP under the Gaussian noise condition in the ESATS dataset. For texture classification applications, the proposed model improves the classification results from 96.26% to 98.13% on the Outext database, from 88.03% to 91.41% on the Xu database, and from 94.00% to 96.67% on the KTH-TIPS database as compared with the completed LBP.

High dynamic range conditions are major obstacles to the implementation of practical stereovision systems in real scenes. We address this problem by introducing an adaptive local ternary-derivative pattern (ALTDP) which is a fusion of the local ternary pattern (LTP) and local derivative pattern (LDP). We make three main contributions in this study: (i) ALTDP encodes more detail information than LDP by extending to eight directions; (ii) ALDTP is better at discriminating and less sensitive to noise in uniform regions with three-value encoding (−1,0,1) without using a pre-defined threshold; and (iii) ALTDP significantly improves the performance of hierarchical belief propagation (BP) by substituting ALTDP data cost for the different intensity data cost. Moreover, our proposed method performs slightly better than LBP and LDP with three datasets: synthetic sequences (set 2) in the EISATS dataset, bright differences sequences (set 5) in the EISATS dataset, and the bumblebee xb3 dataset.

Spatiotemporal stereo matching has attracted much interest over the last few years. The problem is to maintain the temporally consistent disparity of video sequences and to avoid the flickering-artifacts occurring in consecutive disparity maps. In this paper, we propose a constant time spatiotemporal local stereo matching based on the information permeability method that was recently proposed and achieves good stereo results for static image pairs. The proposed three-pass aggregation method takes into account multiple preceding and following frames of the video instead of a single image pair as in the conventional method. Consequently, the temporal disparity consistency is enforced without requiring an explicit motion estimation as several spatiotemporal approaches have done. Experiments showed that our method improves the disparity consistency compared to the conventional information permeability method, and that it outperforms existing spatiotemporal stereo matching techniques.

A new approach for vehicle detection and distance estimation based on stereo vision and evolutionary algorithm (SEA) is described in this paper. First, we reuse our recent work on FPGA implementation of census-based correlations for stereo matching. Next, the SEA uses the gray scale left image and disparity information obtained from the FPGA system to detect the preceding vehicle and estimate its distance. This paper introduces an effective fitness function that allows our proposed method to have an improved performance and higher accuracy when compared with the existing evolutionary algorithm (EA) based methods. A new crossover type, tourna-ment crossover, is introduced to reduce the convergence time of our proposed. This paper also introduces a new approach for estimating the fitness function parameters. This estimation differs from the traditional EA because these parameters were generally created via experiments. Moreover, the processing time and accuracy of SEA can be improved by converting the global search to the local search with V disparity map. The robust experiments have proved that SEA successfully detects vehicles in front and sustains noise from different objects appearing along the road. The detection range is 10m-140m, the detection rate is 95% and the average processing-time is approximately 31 ms/frame on CPU. These results prove that SEA is suitable for a real-time system.