Online camera-to-ground (C2G) calibration dyna-mically estimates the nonrigid transformation between the camera and the road surface, Traditional static calibration methods are sensitive to variations in tire pressure, vehicle loading volume, and diverse road surfaces. Existing online approaches depend on road elements or photometric consistency between overlapping images, requiring continuous target detection and synchronized cameras for calibration. We propose an online monocular C2G calibration method that operates independently of predefined targets. Our approach employs a coarse-to-fine strategy for ground feature extraction and integrates wheel odometry to refine C2G calibration parameters through nonlinear optimization. To ensure accuracy despite nonrigid transformations during driving, we establish stop criteria to identify reliable calibration parameters and introduce quantitative metrics for performance evaluation. Our algorithm has been validated through both simulation and real-world data experiments, proving its effectiveness and superiority over current state-of-the-art techniques.

This paper presents an automated monocular-camera-based computer vision system for autonomous self-backing-up a vehicle towards a trailer, by continuously estimating the 3D trailer coupler position and feeding it to the vehicle control system, until the alignment of the tow hitch with the trailers coupler. This system is made possible through our proposed distance-driven Multiplexer-CNN method, which selects the most suitable CNN using the estimated coupler-to-vehicle distance. The input of the multiplexer is a group made of a CNN detector, trackers, and 3D localizer. In the CNN detector, we propose a novel algorithm to provide a presence confidence score with each detection. The score reflects the existence of the target object in a region, as well as how accurate is the 2D target detection. We demonstrate the accuracy and efficiency of the system on a large trailer database. Our system achieves an estimation error of 1.4 cm when the ball reaches the coupler, while running at 18.9 FPS on a regular PC.

We propose a vision-based highway border detection algorithm using structured Hough voting. Our approach takes advantage of the geometric relationship between highway road borders and highway lane markings. It uses a strategy where a number of trained road border and lane marking detectors are triggered, followed by Hough voting to generate corresponding detection of the border and lane marking. Since the initially triggered detectors usually result in large number of positives, conventional frame-wise Hough voting is not able to always generate robust border and lane marking results. Therefore, we formulate this problem as a joint detection-and-tracking problem under the structured Hough voting model, where tracking refers to exploiting inter-frame structural information to stabilize the detection results. Both qualitative and quantitative evaluations show the superiority of the proposed structured Hough voting model over a number of baseline methods.

In order to be deployed in real-world driving environments, self-driving cars must be able to recognize and respond to exceptional road conditions, such as highway workzones, because such unusual events can alter previously known traffic rules and road geometry. In this paper, we present a set of computer vision methods that recognize, through identification of workzone signs, the bounds of a highway workzone and temporary changes in highway driving environments. Through testing using video data about highway workzones recorded under various weather conditions, our approach was able to perfectly identify the boundaries of workzones and robustly detect a majority of driving condition changes. In addition to these tests, we evaluated, using a mock workzone setup, the usefulness of our workzone recognition systems' outputs for safe-guarding a self-driving car.

Distracted driving is one of the main causes of vehicle collisions in the United States. Passively monitoring a driver's activities constitutes the basis of an automobile safety system that can potentially reduce the number of accidents by estimating the driver's focus of attention. This paper proposes an inexpensive vision-based system to accurately detect Eyes Off the Road (EOR). The system has three main components: 1) robust facial feature tracking; 2) head pose and gaze estimation; and 3) 3-D geometric reasoning to detect EOR. From the video stream of a camera installed on the steering wheel column, our system tracks facial features from the driver's face. Using the tracked landmarks and a 3-D face model, the system computes head pose and gaze direction. The head pose estimation algorithm is robust to nonrigid face deformations due to changes in expressions. Finally, using a 3-D geometric analysis, the system reliably detects EOR.

We address the problem of detecting rear-view (obstacle free) ground surface using a vehicle production camera. This task is considerably more challenging than general front-view road detection, as the associated challenges widely range from low picture quality, fisheye distortion and large objects, to the absence of useful priors such as vanishing points and road structure. Regarding the challenges, we propose a feature that can simultaneously capture local appearance and context information. In addition, the task suffers from strong appearance variations such as shadows and ground markers. Therefore, we propose a novel conditional random field (CRF) model which includes hidden states indicating confident nodes and propagate their confidence to neighboring nodes. We show that our proposed feature and model can jointly achieve robustness against large objects and shadows/markers, showing excellent detection performance under low quality inputs.

Raindrops on vehicles' windshields can degrade the performance of in-vehicle vision systems. In this paper, we present a novel approach that detects and removes raindrops in the captured image when using a single in-vehicle camera. When driving in light or moderate rainy conditions, raindrops appear as small circlets on the windshield in each image frame. Therefore, by analyzing the color, texture and shape characteristics of raindrops in images, we first identify possible raindrop candidates in the regions of interest (ROI), which are small locally salient droplets in a raindrop saliency map. Then, a learning-based verification algorithm is proposed to reduce the number of false alarms (i.e., clear regions mis-detected as raindrops). Finally, we fill in the regions occupied by the raindrops using image inpainting techniques. Numerical experiments indicate that the proposed method is capable of detecting and reducing raindrops in various rain and road scenarios. We also quantify the improvement offered by the proposed method over the state-of-the-art algorithms aimed at the same problem and the benefits to the in-vehicle vision applications like clear path detection.

Detecting objects in shadows is a challenging task in computer vision. For example, in clear path detection application, strong shadows on the road confound the detection of the boundary between clear path and obstacles, making clear path detection algorithms less robust. Shadow removal, relies on the classification of edges as shadow edges or non-shadow edges. We present an algorithm to detect strong shadow edges, which enables us to remove shadows. By analyzing the patch-based characteristics of shadow edges and non-shadow edges (e.g., object edges), the proposed detector can discriminate strong shadow edges from other edges in images by learning the distinguishing characteristics. In addition, spatial smoothing is used to further improve shadow edge detection. Numerical experiments show convincing results that shadows on the road are either removed or attenuated with few visual artifacts, which benefits the clear path detection. In addition, we show that the proposed method outperforms the state-of-art algorithms in different conditions.

We describe a real-time pedestrian detection system intended for use in automotive applications. Our system demonstrates superior detection performance when compared to many state-of-the-art detectors and is able to run at a speed of 14 fps on an Intel Core i7 computer when applied to 640×480 images. Our approach uses an analysis of geometric constraints to efficiently search feature pyramids and increases detection accuracy by using a multiresolution representation of a pedestrian model to detect small pixel-sized pedestrians normally missed by a single representation approach. We have evaluated our system on the Caltech Pedestrian benchmark which is currently the largest publicly available pedestrian dataset at the time of this publication. Our system shows a detection rate of 61% with 1 false positive per image (FPPI) whereas recent other state-of-the-art detectors show a detection rate of 50% ~ 61% under the `reasonable' test scenario (explained later). Furthermore, we also demonstrate the practicality of our system by conducting a series of use case experiments on selected videos of Caltech dataset.

In order to be deployed in real-world driving environments, autonomous vehicles must be able to recognize and respond to exceptional road conditions, such as highway workzones, because such unusual events can alter previously known traffic rules and road geometry. In this paper, we present a set of computer vision methods which recognize the bounds of a highway workzone and temporary changes in highway driving environments through recognition of workzone signs. Our approach filters out irrelevant image regions, localizes potential sign image regions using a learned color model, and recognizes signs through classification. Performance of individual unit tests is promising; still, it is unrealistic to expect perfect performance in sign recognition. Performance errors with individual modules in sign recognition will cause our system to misread temporary highway changes. To handle potential recognition errors, our method utilizes the temporal redundancy of sign occurrences and their corresponding classification decisions. Through testing, using video data recorded under various weather conditions, our approach was able to perfectly identify the boundaries of workzones and robustly detect a majority of driving condition changes.

This paper presents a monocular vision based 3D bicycle tracking framework for intelligent vehicles based on a detection method exploiting a deformable part model and a tracking method using an Interacting Multiple Model (IMM) algorithm. Bicycle tracking is important because bicycles share the road with vehicles and can move at comparable speeds in urban environments. From a computer vision standpoint, bicycle detection is challenging as bicycle's appearance can change dramatically between viewpoints and a person riding on the bicycle is a non-rigid object. To this end, we present a tracking-by-detection method to detect and track bicycles that takes into account these difficult issues. First, a mixture model of multiple viewpoints is defined and trained via a Latent Support Vector Machine (LSVM) to detect bicycles under a variety of circumstances. Each model uses a part based representation. This robust bicycle detector provides a series of measurements (i.e., bounding boxes) in the context of the Kalman filter. Second, to exploit the unique characteristics of bicycle tracking, two motion models based on bicycle's kinematics are fused using an IMM algorithm. For each motion model, an extended Kalman filter (EKF) is used to estimate the position and velocity of a bicycle in the vehicle coordinates. Finally, a single bicycle tracking method using an IMM algorithm is extended to that of multiple bicycle tracking by incorporating a Rao-Blackwellized Particle Filter which runs a particle filter for a data association and an IMM filter for each bicycle tracking. We demonstrate the effectiveness of this approach through a series of experiments run on a new bicycle dataset captured from a vehicle-mounted camera.

With the goal of avoiding obstacles on the road using only a single camera during autonomous driving, we propose an example-based clear path detection method that also considers the additional perspective cue from the estimated vanishing point. First, because of the benefit that the vanishing point provides knowledge for clear path detection, we apply an example-based method to estimate initial vanishing point candidates. Then, instead of building a pre-trained clear path model over the limited training set, we propose an example-based global image matching method to get an approximate idea of clear path candidate regions, and use a Gaussian Mixture Model (GMM) for local image patch modeling to further improve the clear path detection. Finally, we develop an iterative probabilistic refinement to improve performance of both components. Experimental results of real road scenes are presented to illustrate the effectiveness of the proposed method.

In using image analysis to assist a driver to avoid obstacles on the road, traditional approaches rely on various detectors designed to detect different types of objects. We propose a framework that is different from traditional approaches in that it focuses on finding a clear path ahead. We assume that the video camera is calibrated offline (with known intrinsic and extrinsic parameters) and vehicle information (vehicle speed and yaw angle) is known. We first generate perspective patches for feature extraction in the image. Then, after extracting and selecting features of each patch, we estimate an initial probability that the patch corresponds to clear path using a support vector machine (SVM) based probability estimator on the selected features. We finally perform probabilistic patch smoothing based on spatial and temporal constraints to improve the initial estimate, thereby enhancing detection performance. We show that the proposed framework performs well even in some challenging examples with shadows and illumination changes.

In this paper, we describe a prior-based vanishing point estimation method through global perspective structure matching (GPSM). In contrast to the traditional approaches which require an undistorted image with straight roads for vanishing point estimation, our method first infers vanishing point candidates of an input image from an image database with pre-labeled vanishing points. An image-based retrieval method is used to identify the best candidate images in the database by matching image's perspective structure. The initial estimation of input image's vanishing point is calculated from the pre-labeled vanishing points of the best candidates. Probabilistic refinement (PR) is then used to optimize the vanishing point estimate. Experimental results show that the proposed method works well in a variety of on-road driving environments (e.g., in urban, highway and country-side areas), especially with traffic captured by a fish-eye back-aid camera.

This paper presents a vision-based framework for intelligent vehicles to detect and track people riding bicycles in urban traffic environments. To deal with dramatic appearance changes of a bicycle according to different viewpoints as well as nonrigid nature of human appearance, a method is proposed which employs complementary detection and tracking algorithms. In the detection phase, we use multiple view-based detectors: frontal, rear, and right/left side view. For each view detector, a linear Support Vector Machine (SVM) is used for object classification in combination with Histograms of Oriented Gradients (HOG) which is one of the most discriminative features. Furthermore, a real-time enhancement for the detection process is implemented using the Integral Histogram method and a coarse-to-fine cascade approach. Tracking phase is performed by a multiple patch-based Lucas-Kanade tracker. We first run the Harris corner detector over the bounding box which is the result of our detector. Each of the corner points can be a good feature to track and, in consequence, becomes a template of each instance of multiple Lucas-Kanade trackers. To manage the set of patches efficiently, a novel method based on spectral clustering algorithm is proposed. Quantitative experiments have been conducted to show the effectiveness of each component of the proposed framework.

In this paper, a LIDAR-based road and road-edge detection method is proposed to identify road regions and road-edges, which is an essential component of autonomous vehicles. LIDAR range data is decomposed into signals in elevation and signals projected on the ground plane. First, the elevation-based signals are processed by filtering techniques to identify the road candidate region, and by pattern recognition techniques to determine whether the candidate region is a road segment. Then, the line representation of the projected signals on the ground plane is identified and compared to a simple road model in the top-down view to determine whether the candidate region is a road segment with its road-edges. The proposed method provides fast processing speed and reliable detection performance of road and road-edge detection. The proposed framework has been verified through the DARPA Urban Challenge to show its robustness and efficiency on the winning entry Boss vehicle.

In this paper, we propose a probabilistic framework for reconstructing scene geometry utilizing prior knowledge of a class of scenes, for example, scenes captured by a camera mounted on a vehicle driving through city streets. In this framework, we assume the video camera is calibrated, i.e., the intrinsic and extrinsic parameters are known all the time. While we assume a single camera moving during capturing, the framework can be generalized to multiple cameras as well. Traditional approaches try to match the points, lines or patches in multiple images to reconstruct scene geometry. The proposed framework also takes advantage of each patch's appearance and location to infer its orientation using prior information based on statistical learning from training data. The prior hence enhances the geometry reconstruction performance. We show that prior-based 3D reconstruction outperforms traditional 3D reconstruction with both synthetic data and real data, especially in the textureless areas.

We propose a novel video-based rendering algorithm with a single moving camera. We reconstruct a dynamic 3D model of the scene with a feature point set that "evolves" over time. As the scene's appearance changes due to camera and object motions, some existing feature points dynamically disappear while some new feature points dynamically appear relative to the camera. The newly generated feature points' 3D positions and motions are initialized using nearby existing feature points' positions and motions. Our feature evolution, when incorporated into standard tracking and 3D reconstruction algorithms, provides for robust and dense 3D meshes, and their corresponding motions. Consequently, the evolution-based, time-dependent 3D meshes, motions, and textures render good-quality images at a virtual viewpoint and at a desired time instance. We also extend the proposed video-based rendering algorithm from using one single moving camera with one reconstructed depth map to using multiple moving cameras with multiple reconstructed depth maps to avoid occlusion and improve the rendering quality.

In this paper, we study biometric key generation using face images. Given a face image, a set of biometric features are extracted. Each feature is compared with a threshold to generate a key-bit. By cascading the key-bits from all the features, we obtain one bio-key that can be used for security applications. The performance of a biometric key generation system, determined by the chosen thresholds, can be evaluated according to reliability and security. A generalized optimal thresholding method is proposed in this paper to improve the reliability by minimizing the authentication error rate and the security by maximizing the entropy of the generated key.

In this paper, we introduce a novel method for key generation based on biometrics. Given the biometrics, a set of reliable features are extracted. Each feature is compared with multiple thresholds to generate a multi-bit key. By cascading the multi-bit keys, we obtain one biokey that can be used for security applications. In order to generate a reliable bio-key, an optimal thresholding method is proposed to minimize the authentication error rate, in terms of the false accept rate (FAR) and the false reject rate (FRR). The experimental results show that the proposed approach of key generation is user-friendly and reliable.

In this paper, we analyze the security problem of user information associated key generation (UIAKG) systems. We consider three kinds of attacks from the hacker: the exhaustive search attack, the authentic key statistics attack and the device key statistics attack. Under each attack, we give the estimate of the number of guesses the hacker has to make in order to access the system. Such analysis provides useful guidelines in designing a UIAKG system. The analysis also suggests that a user-dependent UIAKG is more secure than a user-independent one. Two UIAKG systems using face images as inputs are designed and compared to support the above theoretical analysis.

Instead of using PINs and passwords as cryptographic keys that are either easy to forget or vulnerable to dictionary attacks, easy-to-carry and difficult-to-transfer keys can be generated based on user-specific biometric information. A framework is proposed to generate stable cryptographic keys from biometric data that is unstable in nature. The proposed framework differs from prior work in that user-dependent transforms are utilized to generate more compact and distinguishable features. Thereby, a longer and more stable bitstream can be generated as the cryptographic key. Experiments are performed on a face database to verify the feasibility of the proposed framework. The preliminary result is very encouraging.

We introduce a new classification algorithm based on the concept of symmetric maximized minimal distance in subspace (SMMS). Given the training data of authentic samples and imposter samples in the feature space, SMMS tries to identify a subspace in which all the authentic samples are close to each other and all the imposter samples are far away from the authentic samples. The optimality of the subspace is determined by maximizing the minimal distance between the authentic samples and the imposter samples in the subspace. We present a procedure to achieve such optimality and to identify the decision boundary. The verification procedure is simple since we only need to project the test sample to the subspace and compare it against the decision boundary. Using face authentication as an example, we show that the proposed algorithm outperforms several other algorithms based on support vector machines (SVM).

We introduce an improved classification algorithm based on the concept of symmetric maximized minimal distance in subspace (SMMS). Given the training data of authentic samples and imposter samples in the feature space, our previous approach, SMMS, tried to identify a subspace in which all the authentic samples were projected onto the origin and all the imposter samples were far away from the origin. The optimality of the subspace was determined by maximizing the minimal distance between the origin and the imposter samples in the subspace. The generalized SMMS relaxes the constraint of fitting all the authentic samples to the origin in the subspace to achieve the optimality and considers the optimal direction of the linear support-vector machines (SVM) as a feasible solution in our optimization procedure to guarantee that our result is no worse than the linear SVM. We present a procedure to achieve such optimality and to identify the subspace and the decision boundary. Once the subspace is trained, the verification procedure is simple since we only need to project the test sample onto the subspace and compare it against the decision boundary. Using face authentication as an example, we show that the proposed algorithm outperforms the linear classifier based on SMMS and SVM. The proposed algorithm also applies to multimodal feature spaces. The features can come from any modalities, such as face images, voices, fingerprints, etc.

Principle component analysis (PCA) has been widely used for analyzing the statistics of data. While applied to biometrics as a classification scheme, PCA faces certain challenges. We present a number of modifications to PCA in order to meet these challenges. Using face recognition as an example, we show how eigenflow, PCA applied to optimal flow, enables us to measure the difference between two images while allowing expression changes and registration error. We show how PCA can be updated to model time-varying statistics. We also show that PCA can be used to model the surface reflectance of human faces and reduce illumination variation that defeats most existing face recognition algorithms. Finally, we present distinguishing component analysis (DCA) and apply it to multimodal biometric authentication.