In order to be deployed in real-world driving environments, self-driving cars must be able to recognize and respond to exceptional road conditions, such as highway workzones, because such unusual events can alter previously known traffic rules and road geometry. In this paper, we present a set of computer vision methods that recognize, through identification of workzone signs, the bounds of a highway workzone and temporary changes in highway driving environments. Through testing using video data about highway workzones recorded under various weather conditions, our approach was able to perfectly identify the boundaries of workzones and robustly detect a majority of driving condition changes. In addition to these tests, we evaluated, using a mock workzone setup, the usefulness of our workzone recognition systems' outputs for safe-guarding a self-driving car.

A self-driving car, to be deployed in real-world driving environments, must be capable of reliably detecting and effectively tracking of nearby moving objects. This paper presents our new, moving object detection and tracking system that extends and improves our earlier system used for the 2007 DARPA Urban Challenge. We revised our earlier motion and observation models for active sensors (i.e., radars and LIDARs) and introduced a vision sensor. In the new system, the vision module detects pedestrians, bicyclists, and vehicles to generate corresponding vision targets. Our system utilizes this visual recognition information to improve a tracking model selection, data association, and movement classification of our earlier system. Through the test using the data log of actual driving, we demonstrate the improvement and performance gain of our new tracking system.

Our earlier lane-marking detection method identified lane-markings appearing on an input image based on the intensity contrast between lane-markings pixels and their neighboring pixels. This detection results in outputs with nearly-zero false negatives, but with many false positives. To filter out these false positives in a principled way, we utilize the driving direction of a roadway. We do this because longitudinal lane-markings delineate the driving direction of a road and the orientations of any true, longitudinal lane-markings appearing on input images should be aligned with this direction. To approximate the driving direction of a road, we detect the vanishing point on a horizon line and draw a line to link the image coordinates of the detected vanishing point to those of the center of the image bottom. We then filter out any lane-marking blobs if their orientations are not aligned with that of the approximated driving direction. Through testing with streets and inter-city highway images, the proposed method demonstrates its effectiveness.

This study aims at developing a method that predicts the CPU usage patterns of software tasks running on a self-driving car. To ensure safety of such dynamic systems, the worst-case-based CPU utilization analysis has been used; however, the nature of dynamically changing driving contexts requires more flexible approach for an efficient computing resource management. To better understand the dynamic CPU usage patterns, this paper presents an effort of designing a feature vector to represent the information of driving environments and of predicting, using regression methods, the selected tasks' CPU usage patterns given specific driving contexts. Experiments with real-world vehicle data show a promising result and validate the usefulness of the proposed method.

To be deployed in the real-world, automatic and semi-automatic systems should understand traffic rules by recognizing and comprehending contents of traffic signs, because traffic signs inform what driving behaviors should be. In this paper, we present the successful application of methods to improve the traffic sign localization performance. Given a potential sign region, our algorithm represents both the detected sign as a target and candidates in the subsequent frame as probability density functions. Then, our algorithm maximizes the similarity between a target and candidates to localize the sign. Finally, the maximum similarity among candidates is assigned as a tracked sign. The experimental results verify that our algorithm can robustly localize traffic signs in images under various weather conditions and driving scenarios.

Cartographic databases can be kept up to date through aerial image analysis. Such analysis is optimized when one knows what parts of an aerial image are roads and when one knows locations of complex road structures, such as overpasses and intersections. This paper proposes self-supervised computer vision algorithms that analyze a publicly available cartographic resource (i.e., screenshots of road vectors) to, without human intervention, identify road image-regions and detects overpasses.

This paper presents new aerial image analysis algorithms that, from highway ortho-images, produce lane-level detailed maps. We analyze screenshots of road vectors to obtain the relevant spatial and photometric cues of road image-regions. We then refine the obtained patterns to generate hypotheses about the true road-lanes. A road-lane hypothesis, since it explains only a part of the true road-lane, is then linked to other hypotheses to completely delineate boundaries of the true road-lanes. Finally, some of the refined image cues about the underlying road network are used to guide a linking process of road-lane hypotheses.

In order to be deployed in real-world driving environments, autonomous vehicles must be able to recognize and respond to exceptional road conditions, such as highway workzones, because such unusual events can alter previously known traffic rules and road geometry. In this paper, we present a set of computer vision methods which recognize the bounds of a highway workzone and temporary changes in highway driving environments through recognition of workzone signs. Our approach filters out irrelevant image regions, localizes potential sign image regions using a learned color model, and recognizes signs through classification. Performance of individual unit tests is promising; still, it is unrealistic to expect perfect performance in sign recognition. Performance errors with individual modules in sign recognition will cause our system to misread temporary highway changes. To handle potential recognition errors, our method utilizes the temporal redundancy of sign occurrences and their corresponding classification decisions. Through testing, using video data recorded under various weather conditions, our approach was able to perfectly identify the boundaries of workzones and robustly detect a majority of driving condition changes.

An autonomous robotic vehicle can drive through and park in a lot more reliably if it is guided by a parking lot map. This specialized map creates structure, including the centerlines of drivable regions and intersection locations in an often unstructured and unmarked environment and enables the vehicle to focus its attention on regions that require detailed analysis. Existing methods of building such maps require manu- ally driving vehicles for collecting sensor measurements. Instead of pursuing a labor-intensive approach, we analyze an aerial image of a parking lot to build a topological map. In particular, our algorithm produces a lane-graph of a parking lot's drivable regions by executing several image processing steps. First it estimates drivable regions by superimposing detection of a parking spots onto the parking lot boundary segmentation. Second, a distance transform is applied to drivable regions to reveal its skeleton. Lastly our algorithm searches the distance map to identify a set of the peak points and connects them to generate a lane-graph that concisely represents drivable regions. Experiments show promising results of real-world parking lot aerial-imagery analysis.

In this paper we present algorithms for automatically generating a road network description from aerial imagery. The road network inforamtion (RNI) produced by our algorithm includes a composite topoloigical and spatial representation of the roads visible in an aerial image. We generate this data for use by autonomous vehicles operating on-road in urban environments. This information is used by the vehicles to both route plan and determine appropriate tactical behaviors. RNI can provide important contextual cues that influence driving behaviors, such as the curvature of the road ahead, the location of traffic signals, or pedestrian dense areas. The value of RNI was demonstrated compellingly in the DARPA Urban Challenge, where the vehicles relied on this information to drive quickly, safely and efficiently.

Road network information (RNI) simplifies autonomous driving by providing strong priors about driving environments. Its usefulness has been demonstrated in the DARPA Urban Challenge. However, the need to manually generate RNI prevents us from fully exploiting its benefits. We envision an aerial image analysis system that automatically generates RNI for a route between two urban locations. As a step toward this goal, we present an algorithm that extracts the structure of a parking lot visible in an aerial image. We formulate this task as a problem of parking spot detection because extracting parking lot structures is closely related to detecting all of the parking spots. To minimize human intervention in use of aerial imagery, we devise a self-supervised learning algorithm that automatically obtains a set of canonical parking spot templates to learn the appearance of a parking lot and estimates the structure of the parking lot from the learned model. The data set extracted from a single image alone is too small to sufficiently learn an accurate parking spot model. To remedy this insufficient positive data problem, we utilize self-supervised parking spots obtained from other aerial images as prior information and a regularization technique to avoid an overfitting solution.

Knowledge of the driving environment is essential for robotic vehicles to comply with traffic rules while autonomously traversing intersections. However, due to limited sensing coverage and continuous changes in driving conditions, rigidly-mounted sensors may not guarantee coverage of all regions of interest, all the time. Unobserved regions around intersections increase uncertainty in driving conditions. This paper describes a dynamic sensor planning method that searches for the optimal angles of two pointable sensors to maximally cover relevant unobserved regions of interest. The obtained angles are used to adjust the orientations of the pointable sensors and hence reduce uncertainty around intersections. Simulation results show that the sensor planning method increases the percentage of covered area. In addition to the sensor pointing problem, we provide an initial discussion of how to reason about occlusions in an urban environment. Occlusions caused by structures and other environmental features can increase uncertainty in driving environments. An occlusion handling method is used to detect these occlusions and enable our vehicle to model the presence of occluded regions. An awareness of occlusions enables safer driving decisions.

This paper describes a method for learning user's interests in the Web-based personalized information filtering system called WAIR. The proposed method analyzes user's reactions to the presented documents and learns from them the profiles for the individual users. Reinforcement learning is used to adapt the term weights in the user profile so that user's preferences are best represented. In contrast to conventional relevance feedback methods which require explicit user feedbacks, our approach learns user preferences implicitly from direct observations of user behaviors during interaction. Field tests have been made which involved 7 users reading a total of 7,700 HTML documents during 4 weeks. The proposed method showed superior performance in personalized information filtering compared to the existing relevance feedback methods.