Autonomous vehicles face significant challenges in understanding pedestrian behavior, particularly in urban environments. In such settings, the system must recognize pedestrian intentions and anticipate their actions to achieve safe and intelligent driving. This paper focuses on predicting pedestrian crossings, enabling oncoming vehicles to react to pedestrians in a traffic scene in a timely manner. We investigate the effectiveness of various input features for pedestrian crossing prediction, including human poses, bounding boxes, and ego vehicle speed features. We propose a novel lightweight architecture based on LSTM and attention to accurately identify crossing pedestrians. Our methods are evaluated on two widely used public datasets for pedestrian behavior, PIE and JAAD datasets, and our algorithm achieves a state-of-the-art performance in both datasets by reaching a prediction accuracy of 91% and an F1-score of 84% on the PIE dataset and an accuracy of 67% and an F1-score of 77% on the JAAD Behavior (BEH) split. We make our code available at https://github.com/afnan29alofi/Pedestrian-Intent-Prediction.git

Advanced Driver Assistance Systems (ADAS) in intelligent vehicles rely on accurate driver perception within the vehicle cabin, often leveraging a combination of sensing modalities. However, these modalities operate at varying rates, posing challenges for real-time, comprehensive driver state monitoring. This paper addresses the issue of missing data due to sensor frame rate mismatches, introducing a generative model approach to create synthetic yet realistic thermal imagery. We propose using conditional generative adversarial networks (cGANs), specifically comparing the pix2pix and CycleGAN architectures. Experimental results demonstrate that pix2pix outperforms CycleGAN, and utilizing multi-view input styles, especially stacked views, enhances the accuracy of thermal image generation. Moreover, the study evaluates the model’s generalizability across different subjects, revealing the importance of individualized training for optimal performance. The findings suggest the potential of generative models in addressing missing frames, advancing driver state monitoring for intelligent vehicles, and underscoring the need for continued research in model generalization and customization.

The curation of large-scale datasets is still costly and requires much time and resources. Data is often manually labeled, and the challenge of creating high-quality datasets remains. In this work, we fill the research gap using active learning for multi-modal 3D object detection. We propose ActiveAnno3D, an active learning framework to select data samples for labeling that are of maximum informativeness for training. We explore various continuous training methods and integrate the most efficient method regarding computational demand and detection performance. Furthermore, we perform extensive experiments and ablation studies with BEVFusion and PV-RCNN on the nuScenes and TUM Traffic Intersection (TUMTraf-I) dataset. We show that we can achieve almost the same performance with PV-RCNN and the entropy-based query strategy when using only half of the training data (77.25 mAP compared to 83.50 mAP) of the TUMTraf-I dataset. BEVFusion achieved an mAP of 64.31 when using half of the training data and 52.88 mAP when using the complete nuScenes dataset. We integrate our active learning framework into the proAnno labeling tool to enable AI-assisted data selection and labeling and minimize the labeling costs. Finally, we provide code, weights, and visualization results on our website.

This paper explores the representation of vehicle lights in computer vision and its implications for various pattern recognition tasks in autonomous driving. Different representations for vehicle lights, including bounding boxes, center points, corner points, and segmentation masks, are discussed in terms of their strengths and weaknesses toward a variety of domain tasks, as well as associated data collection and annotation challenges. This leads to the introduction of the LISA Vehicle Lights Dataset, providing light annotations related to position, state, color, and signal, specifically designed for downstream applications in vehicle detection, intent and trajectory prediction, and safe path planning. A comparison of existing vehicle light datasets is provided, highlighting the unique features and limitations of each dataset. Because occlusions from vehicle pose and passing objects can limit camera observation, we introduce a group of Light Visibility neural networks, which take as input a detected vehicle image and return as output whether a corresponding vehicle light is present in the image. This is especially important for the evaluation of light localizations, states, and signals, as systems decisions should account for differences of lights being in unknown states due to occlusion versus model uncertainty. We show that our trained Light Visibility Models achieve over 90% accuracy on each of the four light classes. Our dataset and model are made available at https://cvrr.ucsd.edu/vehicle-lights-dataset.

It is critical for vehicles to prevent any collisions with pedestrians. Current methods for pedestrian collision prevention focus on integrating visual pedestrian detectors with Automatic Emergency Braking (AEB) systems which can trigger warnings and apply brakes as a pedestrian enters a vehicle’s path. Unfortunately, pedestrian-detection-based systems can be hindered in certain situations such as night-time or when pedestrians are occluded. Our system addresses such issues using an online, map-based pedestrian detection aggregation system where common pedestrian locations are learned after repeated passes of locations. Using a carefully collected and annotated dataset in La Jolla, CA, we demonstrate the system’s ability to learn pedestrian zones and generate advisory notices when a vehicle is approaching a pedestrian despite challenges like dark lighting or pedestrian occlusion. Using the number of correct advisories, false advisories, and missed advisories to define precision and recall performance metrics, we evaluate our system and discuss future positive effects with further data collection. We have made our code available at https://github.com/s7desai/ped-mapping, and a video demonstration of the CHAMP system at https://youtu.be/dxeCrS Gpkw.

One of the most important tasks for ensuring safe autonomous driving systems is accurately detecting road traffic lights and accurately determining how they impact the driver’s actions. In various real-world driving situations, a scene may have numerous traffic lights with varying levels of relevance to the driver, and thus, distinguishing and detecting the lights that are relevant to the driver and influence the driver’s actions is a critical safety task. This paper proposes a traffic light detection model which focuses on this task by first defining salient lights as the lights that affect the driver’s future decisions. We then use this salience property to construct the LAVA Salient Lights Dataset, the first US traffic light dataset with an annotated salience property. Subsequently, we train a Deformable DETR object detection transformer model using Salience-Sensitive Focal Loss to emphasize stronger performance on salient traffic lights, showing that a model trained with this loss function has stronger recall than one trained without.

Predicting the trajectories of surrounding agents is an essential ability for autonomous vehicles navigating through complex traffic scenes. The future trajectories of agents can be inferred using two important cues: the locations and past motion of agents, and the static scene structure. Due to the high variability in scene structure and agent configurations, prior work has employed the attention mechanism, applied separately to the scene and agent configuration to learn the most salient parts of both cues. However, the two cues are tightly linked. The agent configuration can inform what part of the scene is most relevant to prediction. The static scene in turn can help determine the relative influence of agents on each other's motion. Moreover, the distribution of future trajectories is multimodal, with modes corresponding to the agent's intent. The agent's intent also informs what part of the scene and agent configuration is relevant to prediction. We thus propose a novel approach applying multi-head attention by considering a joint representation of the static scene and surrounding agents. We use each attention head to generate a distinct future trajectory to address multimodality of future trajectories. Our model achieves state of the art results on the nuScenes prediction benchmark and generates diverse future trajectories compliant with scene structure and agent configuration.

A driver's gaze is critical for determining the driver's attention level, state, situational awareness, and readiness to take over control from partially and fully automated vehicles. Tracking both the head and eyes (pupils) can provide a reliable estimation of a driver's gaze using face images under ideal conditions. However, the vehicular environment introduces a variety of challenges that are usually unaccounted for - harsh illumination, nighttime conditions, and reflective/dark eyeglasses. Unfortunately, relying on head pose alone under such conditions can prove to be unreliable owing to significant eye movements. In this study, we offer solutions to address these problems encountered in the real world. To solve issues with lighting, we demonstrate that using an infrared camera with suitable equalization and normalization usually suffices. To handle eyeglasses and their corresponding artifacts, we adopt the idea of image-to-image translation using generative adversarial networks (GANs) to pre-process images prior to gaze estimation. To this end, we propose the Gaze Preserving CycleGAN (GPCycleGAN). This network preserves the driver's gaze while removing potential eyeglasses from infrared face images. Our approach exhibits improved performance and robustness on challenging real-world data spanning 13 subjects and a variety of driving conditions.

Pedestrians and vehicles often share the road in complex inner city traffic. This leads to interactions between the vehicle and pedestrians, with each affecting the other's motion. In order to create robust methods to reason about pedestrian behavior and to design interfaces of communication between self-driving cars and pedestrians we need to better understand such interactions. In this paper, we present a data-driven approach to implicitly model pedestrians' interactions with vehicles, to better predict pedestrian behavior. We propose a Long Short-Term Memory (LSTM) model that takes as input the past trajectories of the pedestrian and ego-vehicle, and pedestrian head orientation, and predicts the future positions of the pedestrian. Our experiments based on a real-world, inner city dataset captured with vehicle mounted cameras, show that the usage of such cues improve pedestrian prediction when compared to a baseline that purely uses the past trajectory of the pedestrian.

Assessing the driver's attention and detecting various hazardous and non-hazardous events during a drive are critical for driver's safety. Attention monitoring in driving scenarios has mostly been carried out using vision (camera-based) modality by tracking the driver's gaze and facial expressions. It is only recently that bio-sensing modalities such as Electroencephalogram (EEG) are being explored. But, there is another open problem which has not been explored sufficiently yet in this paradigm. This is the detection of specific events, hazardous and non-hazardous, during driving that affects the driver's mental and physiological states. The other challenge in evaluating multi-modal sensory applications is the absence of very large scale EEG data because of the various limitations of using EEG in the real world. In this paper, we use both of the above sensor modalities and compare them against the two tasks of assessing the driver's attention and detecting hazardous vs. non-hazardous driving events. We collect user data on twelve subjects and show how in the absence of very large-scale datasets, we can still use pre-trained deep learning convolution networks to extract meaningful features from both of the above modalities. We used the publicly available KITTI dataset for evaluating our platform and to compare it with previous studies. Finally, we show that the results presented in this paper surpass the previous benchmark set up in the above driver awareness-related applications.

In this paper, we focus on obtaining 2D and 3D labels, as well as track IDs for objects on the road with the help of a novel 3D Bounding Box Annotation Toolbox (3D BAT). Our open source, web-based 3D BAT incorporates several smart features to improve usability and efficiency. For instance, this annotation toolbox supports semi-automatic labeling of tracks using interpolation, which is vital for downstream tasks like tracking, motion planning and motion prediction. Moreover, annotations for all camera images are automatically obtained by projecting annotations from 3D space into the image domain. In addition to the raw image and point cloud feeds, a Masterview consisting of the top view (bird's-eye-view), side view and front views is made available to observe objects of interest from different perspectives. Comparisons of our method with other publicly available annotation tools reveal that 3D annotations can be obtained faster and more efficiently by using our toolbox.

Existing nighttime vehicle detection methods use color as the primary cue for detecting vehicles. However, complex road and ambient lighting conditions, and camera configurations can influence the effectiveness of such explicit rule and threshold-based methods. In this paper, there are three main contributions. First, we present a novel method to detect vehicles during nighttime involving both learned classifiers and explicit rules, which can operate in the presence of varying ambient lighting conditions. The proposed method that is titled as Vehicle Detection using Active-learning during Nighttime (VeDANt) employs a modified form of active learning for training Adaboost classifiers with Haar-like features using gray-level input images. The hypothesis windows are then further verified using the proposed techniques involving perspective geometries and color information of the taillights. Second, VeDANt is extended to analyze the dynamics of the vehicles during nighttime by detecting three taillight activities-braking, turning left, and turning right. Third, we release three new and fully annotated Laboratory for Intelligent and Safe Automobiles-Night data sets with over 5000 frames for evaluation and benchmarking, which capture a variety of complex traffic and lighting conditions. Such comprehensively annotated and complex public data sets are a first in the area of nighttime vehicle detection. We show that VeDANt is able to detect vehicles during nighttime with over 98% accuracy and less than 1% false detections.

To safely and efficiently navigate through complex traffic scenarios, autonomous vehicles need to have the ability to predict the future motion of surrounding vehicles. Multiple interacting agents, the multi-modal nature of driver behavior, and the inherent uncertainty involved in the task make motion prediction of surrounding vehicles a challenging problem. In this paper, we present an LSTM model for interaction aware motion prediction of surrounding vehicles on freeways. Our model assigns confidence values to maneuvers being performed by vehicles and outputs a multi-modal distribution over future motion based on them. We compare our approach with the prior art for vehicle motion prediction on the publicly available NGSIM US-101 and I-80 datasets. Our results show an improvement in terms of RMS values of prediction error. We also present an ablative analysis of the components of our proposed model and analyze the predictions made by the model in complex traffic scenarios.

Semantic Segmentation methods play a key role in today’s Autonomous Driving research, since they provide a global understanding of the traffic scene for upper-level tasks like navigation. However, main research efforts are being put on enlarging deep architectures to achieve marginal accuracy boosts in existing datasets, forgetting that these algorithms must be deployed in a real vehicle with images that were not seen during training. On the other hand, achieving robustness in any domain is not an easy task, since deep networks are prone to overfitting even with thousands of training images. In this paper, we study in a systematic way what is the gap between the concepts of “accuracy” and “robustness”. A comprehensive set of experiments demonstrates the relevance of using data augmentation to yield models that can produce robust semantic segmentation outputs in any domain. Our results suggest that the existing domain gap can be significantly reduced when appropriate augmentation techniques regarding geometry (position and shape) and texture (color and illumination) are applied. In addition, the proposed training process results in better calibrated models, which is of special relevance to assess the robustness of current systems.

From driver assistance in manual mode to takeover requests in highly automated mode, knowing the state of driver (e.g. sleeping, distracted, attentive) is critical for safe, comfortable and stress-free driving. Since driving is a visually demanding task, driver's gaze is especially important in estimating the state of driver; it has the potential to derive what the driver has attended to or is attending to and predict future actions. We developed a machine vision based framework to model driver's behavior by representing the gave dynamics over a time period using gaze fixations and transition frequencies. As a use case, we explore the driver's gaze patterns during maneuvers executed in freeway driving, namely, left lane change maneuver, right lane change maneuver and lane keep. It is shown that mapping gaze dynamics to gaze fixations and transition frequencies leads to recurring patterns based on driver activities. Furthermore, using data from on-road driving, we show that modeling these patterns show predictive powers in on-road driving maneuver detection around a few hundred milliseconds a priori.

The knowledge of driver distraction will be important for self driving cars in the near future to determine the handoff time to the driver. Driver's gaze direction has been previously shown as an important cue in understanding distraction. While there has been a significant improvement in personalized driver gaze zone estimation systems, a generalized gaze zone estimation system which is invariant to different subjects, perspective and scale is still lagging behind. We take a step towards the generalized system using a Convolutional Neural Network (CNN). For evaluating our system, we collect large naturalistic driving data of 11 drives, driven by 10 subjects in two different cars and label gaze zones for 47515 frames. We train our CNN on 7 subjects and test on the other 3 subjects. Our best performing model achieves an accuracy of 93.36% showing good generalization capability.

Drivers use some combination of head, eye and hand movements to perform varying number of tasks from driving related to non-driving secondary tasks. Furthermore, the combinations may vary depending on the task performed. It is important to model and understand these variations in order to build predictive systems, explore driving styles, detect activities, etc. This study, therefore, introduces a framework to model the spatio-temporal movements of head, eyes and hands given naturalistic driving data of looking-in at the driver for any events or tasks performed of interest. As a use case, we explore the temporal coordination of the modalities on data of drivers executing maneuvers at stop-controlled intersections; the maneuvers executed are go straight, turn left and turn right. In sequentially increasing time windows, by training classifiers which have the ability to provide discriminative quality of its input variable, the experimental study at intersections shows which type of, when and how long distinguishable preparatory movements occur in the range of a few milliseconds to a few seconds.

Object detection and localization in images involve a multi-scale reasoning process. First, responses of object detectors are known to vary with image scale. Second, contextual relationships on a part-level, object-level, and scene-level appear at different scales of the image. This paper studies efficient modeling of these two components by training multi-scale template models. The input to the proposed algorithm involves image features computed at varying image scales, hence operating on volumes in the feature pyramid. The approach generalizes single-scale, local-region detection approaches (e.g. sliding window or region proposals), jointly learning detection and localization cues. Extending the common single-scale detection to a multi-scale volume allows learning scale-specific models as well as analyzing the importance of contextual information at different scales. Experimental analysis on the PASCAL VOC dataset shows the method to considerably improve both detection and localization performance for different type of features, histogram of oriented gradients and deep convolutional neural network features.

We aim to study the modeling limitations of the commonly employed boosted decision trees classifier. Inspired by the success of large, data-hungry visual recognition models (e.g. deep convolutional neural networks), this paper focuses on the relationship between modeling capacity of the weak learners, dataset size, and dataset properties. A set of novel experiments on the Caltech Pedestrian Detection benchmark results in the best known performance among non-CNN techniques while operating at fast run-time speed. Furthermore, the performance is on par with deep architectures (9.71% log-average miss rate), while using only HOG+LUV channels as features. The conclusions from this study are shown to generalize over different object detection domains as demonstrated on the FDDB face detection benchmark (93.37% accuracy). Despite the impressive performance, this study reveals the limited modeling capacity of the common boosted trees model, motivating a need for architectural changes in order to compete with multi-level and very deep architectures.

Human drivers continuously attend to important scene elements in order to safely and smoothly navigate in intricate environments and under uncertainty. This paper develops a human-centric framework for object recognition by analyzing a notion of object importance, as measured in a spatio-temporal context of driving a vehicle. Given a video, a main research question in this paper is - which of the surrounding agents are most important? The answer inherently requires complex reasoning over the current driving task, object properties, scene context, intent, and possible future actions. Therefore, we find that various spatio-temporal cues are relevant for the importance classification task. Furthermore, we demonstrate the usefulness of the importance annotations in evaluating vision algorithms (specifically, for the task of object detection) in an application where trust in automation is imperative and errors are costly. Finally, we show that importance-guided training of object detection models results in improved detection performance of surrounding objects of higher importance. Hence, such models may be better suited for use in representing safety-critical situations, predicting surrounding agents' intentions, and in human-robot interactivity. The dataset and code will be made publicly available.

Face detection is a vital step in the process of extracting semantic information about the driver's state, such as distraction and fatigue, from pixel values in images looking at the driver. Therefore, in the context of time and safety critical situation like driving, efficient use of time and reliable detection of faces is essential. While challenges like lighting and occlusion are prevalent in the vehicle cockpit and disruptive for time and reliabilities sake, the automobile cabin has a unique and advantageous environment for face detection. In this study we introduce a deep CNN based face detection method with discrete head pose estimation which address key challenges such as lighting conditions, occlusions, varying view points. One of the vital points in training the CNN based system is the compilation of positive samples via real-world dataset and synthetic data augmentation useful for in-vehicular settings. Performance evaluation on publicly available naturalistic driving data set, called VIVA-Face Dataset, shows promising results compared to baseline methods.

In this paper, we study the complex coordination of head, eyes and hands as the driver approaches a stop-controlled intersection. The proposed framework is made up of three major parts. The first part is the naturalistic driving dataset collection: synchronized capture of sensors looking-in and looking-out, multiple drivers driving in urban environment, and segmenting events at stop-controlled intersections. The second part is extracting reliable features from purely vision sensors looking in at the driver: eye movements, head pose and hand location respective to the wheel. The third part is in the design of appropriate temporal features for capturing coordination. A random forest algorithm is employed for studying relevance and understanding the temporal evolution of head, eye, and hand cues. Using 24 different events (from 5 drivers resulting in ~ 12200 frames analyzed) of three different maneuvers at stop-controlled intersections, we found that preparatory motions range in the order of a few seconds to a few milliseconds, depending on the modality (i.e. eyes, head, hand), before the event occurs.

Intelligent vehicles of the future are that which, having a holistic (i.e. inside and outside the vehicle) perception and understanding of the driving environment, make it possible for passengers to go from point A to point B safely and in a timely manner. This may happen by way of providing active assistance for drivers, giving full control to automated cars or some combination of the two. No matter how, a holistic perception and understanding of inside and outside the vehicle is absolutely necessary, and vision based techniques are expected to play an increasing role in this holistic view. The question is, how well do these vision techniques work in order to be used in time and safety critical driving situations? We introduce one part of the Vision for Intelligent Vehicles and Applications (VIVA), the face challenge. VIVA is a platform designed to share naturalistic driving data with the community in order to: present issues and challenges in vision from real-world driving conditions, benchmark existing vision approaches using proper metrics and progress the development of future vision algorithms. With a special focus on challenges from looking inside at the driver's face, this articles provides information on how the data is acquired and annotated, and how methods are compared.

Naturalistic driving studies (NDSs) capture drive data from multiple sensor modalities over long periods of time and under varying road conditions. NDS data reduction dictionaries list a range of events that are directly related to the conflicts and threat posed by the dynamics of the surrounding vehicles on the ego-vehicle. Manual reduction of such large scale data for events/conflicts related to dynamics of multiple vehicles is inefficient and prone to errors. In this paper, we present drive analysis techniques for automated NDS data reduction that can be deployed to identify, quantify and visualize threats posed to the ego-vehicle. In this regard, we propose safe maneuver zones (SMZs) that are derived based on the dynamics of surrounding vehicles with respect to the ego-vehicle. A set of metrics are formulated using the SMZs to quantify the threat posed by surrounding vehicles on the ego-vehicle. A detailed drive analysis of naturalistic driving data comprising more than 500,000 frames of data from over 5 hours of highway driving is presented. The resulting drive analysis reports characterize 7 different drives using the different metrics from the SMZs.

This paper presents the challenges that researchers must overcome in traffic light recognition (TLR) research and provides an overview of ongoing work. The aim is to elucidate which areas have been thoroughly researched and which have not, thereby uncovering opportunities for further improvement. An overview of the applied methods and noteworthy contributions from a wide range of recent papers is presented, along with the corresponding evaluation results. The evaluation of TLR systems is studied and discussed in depth, and we propose a common evaluation procedure, which will strengthen evaluation and ease comparison. To provide a shared basis for comparing TLR systems, we publish an extensive public data set based on footage from U.S. roads. The data set contains annotated video sequences, captured under varying light and weather conditions using a stereo camera. The data set, with its variety, size, and continuous sequences, should challenge current and future TLR systems.

Typically, in a detector framework, the model size is fixed at the size of the smallest object to be detected, and larger objects are detected by scaling the input image. The information lost due to scaling could be vital for accurately detecting large objects, which is an essential task for vision-based driver-assistance systems. To this end, we evaluate a multiresolution detector framework by training models at different sizes and demonstrate its effectiveness on a state-of-the-art pedestrian detector. Our comprehensive evaluation demonstrates meaningful improvement in detector performance. On the KITTI dataset under moderate difficulty settings, we achieve a 6% increase in the detector's average precision over the baseline single-resolution result on the KITTI benchmark. Further insights into the detector's improvements are provided using a fine-grained analysis of the detector's performance at various threshold settings.

Hands are a very important cue for understanding and analyzing driver activity and human activity, in general. Vision-based hand detection and tracking involve major challenges, such as attaining robustness to inconsistencies in lighting and scale, background clutter, object occlusion/disappearance and the large variability in hand shape, size, color, and structure. In this paper, we introduce a novel framework suitable for tracking multiple hands online. Assigning tracks to these detections is modeled as a bipartite matching problem with an objective of minimizing the total cost. Both motion and appearance cues are integrated in order to gain robustness to occlusion, fast movement, and interacting hands. Additionally, we study the utility of a left versus right hand classifier to disambiguate hand tracks and reduce ID switches. The proposed tracker shows promise on an extensive, naturalistic, and publicly available driving (VIVA Challenge) data set, by tracking both hands of the driver and the passenger effectively.

On-road vehicle detection is a critical operation in automotive active safety systems such as collision avoidance, merge assist, lane change assistance, etc. In this paper, we present VeDAS-Vehicle Detection using Active learning and Symmetry. VeDAS is a multipart-based vehicle detection algorithm that employs Haar-like features and Adaboost classifiers for the detection of fully and partially visible rear views of vehicles. In order to train the classifiers, a modified active learning framework is proposed that selects positive and negative samples of multiple parts in an automated manner. Furthermore, the detected parts from the classifiers are associated by using a novel iterative window search algorithm and a symmetry-based regression model to extract fully visible vehicles. The proposed method is evaluated on seven different datasets that capture varying road, traffic, and weather conditions. Detailed evaluations show that the proposed method gives high true positive rates of over 95% and performs better than existing state-of-the-art rear-view-based vehicle detection methods. Additionally, VeDAS also detects partially visible rear views of vehicles using the residues left behind after detecting the fully visible vehicles. VeDAS is able to detect partial rear views with a detection rate of 87% on a new partially visible rear-view vehicle dataset that we release as part of this paper.

Research in traffic light recognition (TLR) has stagnated compared to related computer vision areas, such as pedestrian detection and and traffic sign recognition. We focus on the detection sub-problem, since this is the most challenging problem and solving this is the key to a successful TLR system. This is done by looking at four detectors from different author groups and their reported results. From surveying existing work it is clear that currently evaluation is limited primarily to small local datasets. In order to provide a common basis for future comparison of TLR research an extensive public database is collected based on footage from US roads. The database consists of continuous test and training video sequences, totaling 46,418 frames and 112,971 annotated traffic lights. The sequences are captured by a stereo camera mounted on the roof of a vehicle driving under both night and day conditions with varying light and weather.

This paper presents a monocular and purely vision based pedestrian trajectory tracking and prediction framework with integrated map-based hazard inference. In Advanced Driver Assistance systems research, a lot of effort has been put into pedestrian detection over the last decade, and several pedestrian detection systems are indeed showing impressive results. Considerably less effort has been put into processing the detections further. We present a tracking system for pedestrians, which based on detection bounding boxes tracks pedestrians and is able to predict their positions in the near future. The tracking system is combined with a module which, based on the car's GPS position acquires a map and uses the road information in the map to know where the car can drive. Then the system warns the driver about pedestrians at risk, by combining the information about hazardous areas for pedestrians with a probabilistic position prediction for all observed pedestrians.

This paper studies the usefulness of appearance patterns for the challenging task of pedestrian detection. Despite appearance specific models being common in rigid object detection, the technique is still little understood for pedestrians. Three main approaches for reasoning over orientation, occlusion, and visual cues in obtaining the appearance patterns are compared. This work demonstrates that large gains in detection performance (up to 17 AP points on the challenging KITTI dataset) can be made using a state-of-the-art pedestrian detector.

We are concerned with investigating efficient video representations for the purpose of hand gesture recognition in settings of naturalistic driving. In order to provide a common experimental setup for previously proposed space-time features, we study a color and depth naturalistic hand gesture benchmark. The dataset allows for evaluation of descriptors under settings of common self-occlusion and large illumination variation. A collection of simple and quick to extract spatio-temporal cues requiring no codebook encoding are proposed. Their effectiveness is validated on our dataset, as well as on the Cambridge hand gesture dataset, improving state-of-the-art. Finally, fusion of the modalities and various cues is studied.

In order to understand dangerous situations in the driving environment, naturalistic driving studies (NDS) are conducted by collecting and analyzing data from sensors looking inside and outside of the car. Manually processing the overwhelming amounts of data that are generated in such studies is very comprehensive. We propose a method for automatic data reduction for NDS based on stereo vision vehicle detection and tracking during day- and nighttime. The developed system can automatically register five NDS events, mainly related to intersections, from an existing NDS dictionary. We propose a new drive event which takes advantage of the extra dimension provided by stereo vision. In total, six drive events are selected on the basis of them being problematic to detect automatically using conventional monocular computer vision approaches. The proposed system is evaluated on day-and nighttime data, resulting in drive analysis report. The proposed system reach an overall precision of 0.78 and an overall recall of 0.72.

Analysis of naturalistic driving data provides a rich set of semantics which can be used to determine the driving characteristics that could lead to crashes and near-crashes. In this paper, we introduce “drive quality” analysis as part of the drive analysis process of naturalistic driving studies (NDSs) that we have previously introduced in [1]. In this first work on drive quality analysis for NDS data reduction, lane change maneuvers that are reduced from naturalistic driving data are further analyzed in a detailed manner in order to characterize them. Visual data from multiple perspectives and the data from in-vehicle sensors are used to characterize lane changes based on both the ego-vehicle kinematics and ego-vehicle surround dynamics. According to available literature on NDS and data reduction, this is the first work that presents an analysis of visual data from multiple perspectives to characterize and extract semantics related to ego-vehicle maneuvers in NDSs such as SHRP2.

In the recent years, mobile computing platforms are becoming increasingly cheaper and yet more powerful in terms of computational resources. Automobiles provide a suitable environment to deploy such mobile platforms in order to provide low cost driver assistance systems. In this paper, we propose Snap-DAS which is a vision-based driver assistance system that is implemented on a SnapdragonTMembedded platform. A forward facing camera combined with the SnapdragonTMplatform constitute Snap-DAS. The compute efficient implementation of the LASeR lane estimation algorithm in [1] is exploited to implement a set of lane related functions on Snap-DAS, which include lane drift warning and lane change event detection. A detailed evaluation is performed on live data and Snap-DAS is also field tested on freeways. Furthermore, we explore the possibility of using Snap-DAS for analyzing drives for online naturalistic driving studies.

Vision-based lane detection is a critical component of modern automotive active safety systems. Although a number of robust and accurate lane estimation (LE) algorithms have been proposed, computationally efficient systems that can be realized on embedded platforms have been less explored and addressed. This paper presents a framework that incorporates contextual cues for LE to further enhance the performance in terms of both computational efficiency and accuracy. The proposed context-aware LE framework considers the state of the ego vehicle, its surroundings, and the system-level requirements to adapt and scale the LE process resulting in substantial computational savings. This is accomplished by synergistically fusing data from multiple sensors along with the visual data to define the context around the ego vehicle. The context is then incorporated as an input to the LE process to scale it depending on the contextual requirements. A detailed evaluation of the proposed framework on real-world driving conditions shows that the dynamic and static configuration of the lane detection process results in computation savings as high as 90%, without compromising on the accuracy of LE.

This paper presents a comprehensive research study of the detection of U.S. traffic signs. Until now, the research in Traffic Sign Recognition systems has been centered on European traffic signs, but signs can look very different across different parts of the world, and a system that works well in Europe may indeed not work in the U.S. We go over the recent advances in traffic sign detection and discuss the differences in signs across the world. Then we present a comprehensive extension to the publicly available LISA-TS traffic sign data set, almost doubling its size, now with high-definition-quality footage. The extension is made with testing of tracking sign detection systems in mind, providing videos of traffic sign passes. We apply the Integral Channel Features and Aggregate Channel Features detection methods to U.S. traffic signs and show performance numbers outperforming all previous research on U.S. signs (while also performing similarly to the state of the art on European signs). Integral Channel Features have previously been used successfully for European signs, whereas Aggregate Channel Features have never been applied to the field of traffic signs. We take a look at the performance differences between the two methods and analyze how they perform on very distinctive signs, as well as white, rectangular signs, which tend to blend into their environment.

This paper studies efficient means in dealing with intracategory diversity in object detection. Strategies for occlusion and orientation handling are explored by learning an ensemble of detection models from visual and geometrical clusters of object instances. An AdaBoost detection scheme is employed with pixel lookup features for fast detection. The analysis provides insight into the design of a robust vehicle detection system, showing promise in terms of detection performance and orientation estimation accuracy.

Naturalistic driving studies (NDSs) capture large volumes of drive data from multiple sensor modalities, which are analyzed for critical information about driver behavior and driving characteristics that lead to crashes and near crashes. One of the key steps in such studies is data reduction, which is defined as a process by which “trained employees” review segments of driving video and record a taxonomy of variables that provides information regarding the sequence of events leading to crashes. Given the volume of sensor data in NDSs, such manual analysis of the drive data can be time-consuming. In this paper, we introduce “drive analysis” as one of the first steps toward automating the process of extracting midlevel semantic information from raw sensor data. Techniques are proposed to analyze the sensor data from multiple modalities and to extract a set of 23 semantics about lane positions, vehicle localization within lanes, vehicle speed, traffic density, and road curvature. The proposed techniques are demonstrated using real-world test drives comprising over 150 000 frames of visual data, which are also accompanied by vehicle dynamics that are captured from an in-vehicle controller-area-network bus, an inertial motion unit, and a Global Positioning System.

Automotive electronics is a rapidly expanding area with an increasing number of safety, driver assistance, and infotainment devices becoming standard in new vehicles. Current vehicles generally employ a number of different networking protocols to integrate these systems into the vehicle. The introduction of large numbers of sensors to provide driver assistance applications and the associated high-bandwidth requirements of these sensors have accelerated the demand for faster and more flexible network communication technologies within the vehicle. This paper presents a comprehensive overview of current research on advanced intra-vehicle networks and identifies outstanding research questions for the future.

Naturalistic driving dataset is at the heart of automotive user interface research, detecting/measuring driver distraction, and many other driver safety related studies. Recent advances in the collection of large scale naturalistic driving data include the second Strategic Highway Research Program (SHRP2) consisting of more than 3000 subjects and the 100-Car study. Public access to such data, however, is made difficult due to personal identifiable information and protection of privacy. We propose de-identification filters for protecting the privacy of drivers while preserving sufficient details to infer driver behavior, such as the gaze direction, in naturalistic driving videos. Driver's gaze estimation is of particular interest because it is a good indicator of driver's visual attention and a good predictor of driver's intent. We implement and compare de-identification filters, which are made up of a combination of preserving eye regions, superimposing head pose encoded face mask and replacing background with black pixels, and show promising results.

In this paper, a multiview, multimodal vision framework is proposed in order to characterize driver activity based on head, eye, and hand cues. Leveraging the three types of cues allows for a richer description of the driver's state and for improved activity detection performance. First, regions of interest are extracted from two videos, one observing the driver's hands and one the driver's head. Next, hand location hypotheses are generated and integrated with a head pose and facial landmark module in order to classify driver activity into three states: wheel region interaction with two hands on the wheel, gear region activity, or instrument cluster region activity. The method is evaluated on a video dataset captured in on-road settings.

As vehicles travel through a scene, changes in aspect ratio and appearance as observed from a camera (or an array of cameras) make vehicle detection a difficult computer vision problem. Rather than relying solely on appearance cues, we propose a framework for detecting vehicles and eliminating false positives by utilizing the motion cues in the scene in addition to the appearance cues. As a case study, we focus on overtaking vehicle detection in a freeway setting from forward and rear views of the ego-vehicle. The proposed integration occurs in two steps. First, motion-based vehicle detection is performed using optical flow. Taking advantage of epipolar constraints, salient motion vectors are extracted and clustered using spectral clustering to form bounding boxes of vehicle candidates. Post-processing and outlier removal further refine the detections. Second, the motion-based detections are then combined with the output of an appearance-based vehicle detector to reduce false positives and produce the final vehicle detections.

Accurate and efficient lane estimation is a critical component of active safety systems in automobiles such as lane departure warning systems. A substantial number of lane estimation methods have been proposed and evaluated in literature. However, a common set of evaluation metrics that assess different components of lane estimation process has not been addressed. This paper proposes a set of performance evaluation metrics for lane estimation process, that can be deployed to evaluate different kinds of lane estimation algorithms. Evaluation by applying the proposed metrics is demonstrated using a recent lane estimation method.

In this work, we propose a vision-based analysis framework for recognizing in-vehicle activities such as interactions with the steering wheel, the instrument cluster and the gear. The framework leverages two views for activity analysis, a camera looking at the driver's hand and another looking at the driver's head. The techniques proposed can be used by researchers in order to extract ‘mid-level’ information from video, which is information that represents some semantic understanding of the scene but may still require an expert in order to distinguish difficult cases or leverage the cues to perform drive analysis. Unlike such information, ‘low-level’ video is large in quantity and can't be used unless processed entirely by an expert. This work can apply to minimizing manual labor so that researchers may better benefit from the accessibility of the data and provide them with the ability to perform larger-scaled studies.

In this work, we propose a framework for the recognition and prediction of driver maneuvers by considering holistic cues. With an array of sensors, driver's head, hand, and foot gestures are being captured in a synchronized manner together with lane, surrounding agents, and vehicle parameters. An emphasis is put on real-time algorithms. The cues are processed and fused using a latent-dynamic discriminative framework. As a case study, driver activity recognition and prediction in overtaking situations is performed using a naturalistic, on-road dataset. A consequence of this work would be in development of more effective driver analysis and assistance systems.

The dynamic appearance of vehicles as they enter and exit a scene makes vehicle detection a difficult and complicated problem. Appearance based detectors generally provide good results when vehicles are in clear view, but have trouble in the scenes edges due to changes in the vehicles aspect ratio and partial occlusions. To compensate for some of these deficiencies, we propose incorporating motion cues from the scene. In this work, we focus on a overtaking vehicle detection in a freeway setting with front and rear facing monocular cameras. Motion cues are extracted from the scene, and leveraging the epipolar geometry of the monocular setup, motion compensation is performed. Spectral clustering is used to group similar motion vectors together, and after post-processing, vehicle detections candidates are produced. Finally, these candidates are combined with an appearance detector to remove any false positives, outputting the detections as a vehicle travels through the scene.

Naturalistic driving studies (NDS) provide critical information about driving behaviors and characteristics that could lead to crashes and near-crashes. Such studies involve analysis of large volumes of data from multiple sensors and detection and extraction of critical events is an important step in NDS. This paper introduces techniques that analyze the visual data complemented with other sensors in the vehicle to determine critical events related to lane drifts, road departures and road delineations. To the best knowledge of the authors, this is the first work that detects and extract events listed in visual reference dictionary of NDS studies like Strategic Highway Research Program 2 (SHRP2). Detailed evaluations with real-world NDS data is presented.

This paper details the research, development, and demonstrations of real-world systems intended to assist the driver in urban environments, as part of the Urban Intelligent Assist (UIA) research initiative. A 3-year collaboration between Audi AG, Volkswagen Group of America Electronics Research Laboratory, and UC San Diego, the driver assistance portion of the UIA project focuses on two main use cases of vital importance in urban driving. The first, Driver Attention Guard, applies novel computer vision and machine learning research for accurately tracking the driver's head position and rotation using an array of cameras. The system then infers the driver's focus of attention, alerting the driver and engaging safety systems in case of extended driver inattention. The second application, Merge and Lane Change Assist, applies a novel probabilistic compact representation of the on-road environment, fusing data from a variety of sensor modalities. The system then computes safe and low-cost merge and lane-change maneuver recommendations. It communicates desired speeds to the driver via Head-up Display, when the driver touches the blinker, indicating his desired lane. The fully-implemented systems, complete with HMI, were demonstrated to the public and press in San Francisco in January of 2014.

Analysis of driver's head behavior is an integral part of driver monitoring system. Driver's coarse gaze direction or gaze zone is a very important cue in understanding driver-state. Many existing gaze zone estimators are, however, limited to single camera perspectives, which are vulnerable to occlusions of facial features from spatially large head movements away from the frontal pose. Non-frontal glances away from the driving direction, though, are of special interest as interesting events, critical to driver safety, occur during those times. In this paper, we present a distributed camera framework for gaze zone estimation using head pose dynamics to operate robustly and continuously even during large head movements. For experimental evaluations, we collected a dataset from naturalistic on-road driving in urban streets and freeways. A human expert provided the gaze zone ground truth using all vision information including eyes and surround context. Our emphasis is to understand the efficacy of the head pose dynamic information in predicting eye-gaze-based zone ground truth. We conducted several experiments in designing the dynamic features and compared the performance against static head pose based approach. Analyses show that dynamic information significantly improves the results.

A common pool of naturalistic driving data is necessary to develop and compare algorithms that infer driver behavior, in order to improve driving safety. Naturalistic driving data, such as video sequences of looking at a driver, however, cause concern for the privacy of individual drivers. In an ideal situation, a deidentification filter applied to a raw image of looking at a driver would, semantically, protect the identity and preserve the behavior (e.g, eye gaze, head pose, and hand activity) of the driver. Driver gaze estimation is of particular interest because it is a good indicator of a driver's visual attention and a good predictor of a driver's intent. Interestingly, the same facial features that are explicitly or implicitly used for gaze estimation play a key role in recognizing a person's identity. In this paper, we implement a specific deidentification filter on video sequences of looking at a driver from naturalistic driving and present novel findings on its effect on face recognition and driver gaze-zone estimation.

In this paper, we develop a vision-based system that employs a combined RGB and depth descriptor to classify hand gestures. The method is studied for a human-machine interface application in the car. Two interconnected modules are employed: one that detects a hand in the region of interaction and performs user classification, and another that performs gesture recognition. The feasibility of the system is demonstrated using a challenging RGBD hand gesture data set collected under settings of common illumination variation and occlusion.

In this paper, we present a novel probabilistic compact representation of the on-road environment, i.e., the dynamic probabilistic drivability map (DPDM), and demonstrate its utility for predictive lane change and merge (LCM) driver assistance during highway and urban driving. The DPDM is a flexible representation and readily accepts data from a variety of sensor modalities to represent the on-road environment as a spatially coded data structure, encapsulating spatial, dynamic, and legal information. Using the DPDM, we develop a general predictive system for LCMs. We formulate the LCM assistance system to solve for the minimum-cost solution to merge or change lanes, which is solved efficiently using dynamic programming over the DPDM. Based on the DPDM, the LCM system recommends the required acceleration and timing to safely merge or change lanes with minimum cost. System performance has been extensively validated using real-world on-road data, including urban driving, on-ramp merges, and both dense and free-flow highway conditions.

Analysis of a driver's head behavior is an integral part of a driver monitoring system. In particular, the head pose and dynamics are strong indicators of a driver's focus of attention. Many existing state-of-the-art head dynamic analyzers are, however, limited to single-camera perspectives, which are susceptible to occlusion of facial features from spatially large head movements away from the frontal pose. Nonfrontal glances away from the road ahead, however, are of special interest since interesting events, which are critical to driver safety, occur during those times. In this paper, we present a distributed camera framework for head movement analysis, with emphasis on the ability to robustly and continuously operate even during large head movements. The proposed system tracks facial features and analyzes their geometric configuration to estimate the head pose using a 3-D model. We present two such solutions that additionally exploit the constraints that are present in a driving context and video data to improve tracking accuracy and computation time. Furthermore, we conduct a thorough comparative study with different camera configurations. For experimental evaluations, we collected a novel head pose data set from naturalistic on-road driving in urban streets and freeways, with particular emphasis on events inducing spatially large head movements (e.g., merge and lane change). Our analyses show promising results.

Currently there are many research focused on using smartphone as a data collection device. Many have shown its sensors ability to replace a lab test bed. These inertial sensors can be used to segment and classify driving events fairly accurately. In this research we explore the possibility of using the vehicle's inertial sensors from the CAN bus to build a profile of the driver to ultimately provide proper feedback to reduce the number of dangerous car maneuver. Braking and turning events are better at characterizing an individual compared to acceleration events. Histogramming the time-series values of the sensor data does not help performance. Furthermore, combining turning and braking events helps better differentiate between two similar drivers when using supervised learning techniques compared to separate events alone, albeit with anemic performance.

Abstract — Vehicle detection is a key problem in computer vision, with applications in driver assistance and active safety. A challenging aspect of the problem is the common occlusion of vehicles in the scene. In this paper, we present a visionbased system for vehicle localization and tracking for detecting partially visible vehicles. Consequently, vehicles are localized more reliably and tracked for longer periods of time. The proposed system detects vehicles using an active-learning based monocular vision approach and motion (optical flow) cues. A calibrated stereo rig is utilized to acquire a depth map, and consequently the real-world coordinates of each detected vehicle. Tracking is performed using a Kalman filter. The tracking is formulated to integrate stereo-monocular information. We demonstrate the effectiveness of the proposed system on a multilane highway dataset containing instances of vehicles with relative motion to the ego-vehicle. I.

In this paper, we focus on the analysis of naturalistic driver behavior using hand activity. To that end, a dataset of color and depth images under varying operating modes and illumination settings was collected. The proposed framework provides a robust solution for localizing the hands by partitioning visible and depth images into disjoint sub-regions which may be of interest for studying the state of the driver: wheel, lap, hand rest, gear, and infotainment region. Different feature extraction methods are proposed and thoroughly studied in terms of speed and performance for each of the five regions. A model for hand presence is learned for each region separately, and these are integrated using a second-stage classifier. As the appearance of hands varies among regions and the hands can only be found in a subset of the regions chosen, the technique leverages information and confidence from multiple regions to produce hand activity classification.

Research on pedestrian detection system still presents a lot of space for improvements, both on speed and detection accuracy. This paper presents a full implementation of a pedestrian detection system, using a part-based classification for the candidates identification and a feature based tracking for increasing the result robustness. The novelty of this approach relies on the use of part-based approach with a combination of Haar-cascade and HOG-SVM. Tests have been conducted using standard datasets showing results aligned with those of the other state-of-the-art systems available in literature. Real world tests also show high speed performance.

This document provides a review of the past decade's literature in on-road vision-based vehicle detection. Over the past decade, vision-based surround perception has matured significantly from its infancy. We detail advances in vehicle detection, discussing representative works from the monocular and stereo-vision domains. We provide discussion on the state-of-the-art, and provide perspective on future research directions in the field.

In this study, we present novel work focused on assisting the driver during merge maneuvers. We use an automotive testbed instrumented with sensors for monitoring critical regions in the vehicle's surround. Fusing information from multiple sensor modalities, we integrate measurements into a contextually relevant, intuitive, general representation, which we term the Dynamic Probabilistic Drivability Map [DPDM]. We formulate the DPDM for driver assistance as a compact representation of the surround environment, integrating vehicle tracking information, lane information, road geometry, obstacle detection, and ego-vehicle dynamics. Given a robust understanding of the ego-vehicle's dynamics, other vehicles, and the on-road environment, our system recommends merge maneuvers to the driver, formulating the maneuver as a dynamic programming problem over the DPDM, searching for the minimum cost solution for merging. Based on the configuration of the road, lanes, and other vehicles on the road, the system recommends the appropriate acceleration or deceleration for merging into the adjacent lane, specifying when and how to merge.

Automotive electronics is a rapidly expanding area with an increasing number of driver assistance and infotainment devices becoming standard in new vehicles. A review of current networking standards within vehicles reveals a fragmented and proprietary situation with several standards such as MOST, CAN and LVDS dominating, all of which are currently being used by various vehicle manufacturers. Due to the cost of employing a range of networking standards, there is a general desire within the automotive industry to converge on the use of the 802.3 Ethernet for all in-vehicle communication between devices. The introduction of in-vehicle cameras to provide driver assistance applications and the associated high bandwidth requirements of multi camera systems has accelerated the demand for a unifying automotive network architecture. This paper presents an overview of current research present in the literature and identifies trends in the field for the future.

Detecting pedestrians is still a challenging task for automotive vision systems due to the extreme variability of targets, lighting conditions, occlusion, and high-speed vehicle motion. Much research has been focused on this problem in the last ten years and detectors based on classifiers have gained a special place among the different approaches presented. This paper presents a state-of-the-art pedestrian detection system based on a two-stage classifier. Candidates are extracted with a Haar cascade classifier trained with the Daimler Detection Benchmark data set and then validated through a part-based histogram-of-oriented-gradient (HOG) classifier with the aim of lowering the number of false positives. The surviving candidates are then filtered with feature-based tracking to enhance the recognition robustness and improve the results' stability. The system has been implemented on a prototype vehicle and offers high performance in terms of several metrics, such as detection rate, false positives per hour, and frame rate. The novelty of this system relies on the combination of a HOG part-based approach, tracking based on a specific optimized feature, and porting on a real prototype.

In this paper, we introduce a synergistic approach to integrated lane and vehicle tracking for driver assistance. The approach presented in this paper results in a final system that improves on the performance of both lane tracking and vehicle tracking modules. Further, the presented approach introduces a novel approach to localizing and tracking other vehicles on the road with respect to lane position, which provides information on higher contextual relevance that neither the lane tracker nor vehicle tracker can provide by itself. Improvements in lane tracking and vehicle tracking have been extensively quantified. Integrated system performance has been validated on real-world highway data. Without specific hardware and software optimizations, the fully implemented system runs at near-real-time speeds of 11 frames per second.

In this paper, we introduce vehicle detection by independent parts (VDIP) for urban driver assistance. In urban environments, vehicles appear in a variety of orientations, i.e., oncoming, preceding, and sideview. Additionally, partial vehicle occlusions are common at intersections, during entry and exit from the camera's field of view, or due to scene clutter. VDIP provides a lightweight robust framework for detecting oncoming, preceding, sideview, and partially occluded vehicles in urban driving. In this paper, we use active learning to train independent-part detectors. A semisupervised approach is used for training part-matching classification, which forms sideview vehicles from independently detected parts. The hierarchical learning process yields VDIP, featuring efficient evaluation and robust performance. Parts and vehicles are tracked using Kalman filtering. The fully implemented system is lightweight and runs in real time. Extensive quantitative analysis on real-world on-road data sets is provided.

This paper provides a review of the literature in on-road vision-based vehicle detection, tracking, and behavior understanding. Over the past decade, vision-based surround perception has progressed from its infancy into maturity. We provide a survey of recent works in the literature, placing vision-based vehicle detection in the context of sensor-based on-road surround analysis. We detail advances in vehicle detection, discussing monocular, stereo vision, and active sensor-vision fusion for on-road vehicle detection. We discuss vision-based vehicle tracking in the monocular and stereo-vision domains, analyzing filtering, estimation, and dynamical models. We discuss the nascent branch of intelligent vehicles research concerned with utilizing spatiotemporal measurements, trajectories, and various features to characterize on-road behavior. We provide a discussion on the state of the art, detail common performance metrics and benchmarks, and provide perspective on future research directions in the field.

We present a novel facial expression recognition framework using audio-visual information analysis. We propose to model the cross-modality data correlation while allowing them to be treated as asynchronous streams. We also show that our framework can improve the recognition performance while significantly reducing the computational cost by avoiding redundant or insignificant frame processing by incorporating auditory information. In particular, we design a single good image representation of image sequence by weighted sums of registered face images where the weights are derived using auditory features. We use a still image based technique for the expression recognition task. Our framework, however, can be generalized to work with dynamic features as well. We performed experiments using eNTERFACE'05 audio-visual emotional database containing six archetypal emotion classes: Happy, Sad, Surprise, Fear, Anger and Disgust. We present one-to-one binary classification as well as multi-class classification performances evaluated using both subject dependent and independent strategies. Furthermore, we compare multi-class classification accuracies with those of previously published literature which use the same database. Our analyses show promising results.

An important goal in automotive user interface research is to predict a user's reactions and behaviors in a driving environment. The behavior of both drivers and passengers can be studied by analyzing eye gaze, head, hand, and foot movement, upper body posture, etc. In this paper, we focus on estimating head pose, which has been shown to be a good predictor of driver intent and a good proxy for gaze estimation, and provide a valuable head pose database for future comparative studies. Most existing head pose estimation algorithms are still struggling under large spatial head turns. Our method, however, relies on using facial features that are visible even during large spatial head turns to estimate head pose. The method is evaluated on the LISA-P Head Pose database, which has head pose data from on-road daytime and nighttime drivers of varying age, race, and gender; ground truth for head pose is provided using a motion capture system. In special regards to eye gaze estimation for automotive user interface study, the automatic head pose estimation technique presented in this paper can replace previous eye gaze estimation methods that rely on manual data annotation or be used in conjunction with them when necessary.

We present a real-time vision-based system that discriminates hand gestures performed by in-vehicle front-row seat occupants for accessing the infotainment system. The hand gesture-based visual user interface may be more natural and intuitive to the user than the current tactile interaction interface. Consequently, it may encourage a gaze-free interaction, which can alleviate driver distraction without limiting the user's infotainment experience. The system uses visible and depth images of the dashboard and center-console area in the vehicle. The first step in the algorithm uses the representation of the image area given by a modified histogram-of-oriented-gradients descriptor and a support vector machine (SVM) to classify whether the driver, passenger, or no one is interacting with the region of interest. The second step extracts gesture characteristics from temporal dynamics of the features derived in the initial step, which are then inputted to a SVM in order to perform gesture classification from a set of six classes of hand gestures. The rate of correct user classification into one of the three classes is 97.9% on average. Average hand gesture classification rates for the driver and passenger using color and depth input are above 94%. These rates were achieved on in-vehicle collected data over varying illumination conditions and human subjects. This approach demonstrates the feasibility of the hand gesture-based in-vehicle visual user interface.

Automatic perception of human posture and gesture from vision input has an important role in developing intelligent video systems. In this paper, we present a novel gesture recognition approach for human computer interactivity based on marker-less upper body pose tracking in 3-D with multiple cameras. To achieve the robustness and real-time performance required for practical applications, the idea is to break the exponentially large search problem of upper body pose into two steps: first, the 3-D movements of upper body extremities (i.e., head and hands) are tracked. Then using knowledge of upper body model constraints, these extremities movements are used to infer the whole 3-D upper body motion as an inverse kinematics problem. Since the head and hand regions are typically well defined and undergo less occlusion, tracking is more reliable and could enable more robust upper body pose determination. Moreover, by breaking the problem of upper body pose tracking into two steps, the complexity is reduced considerably. Using pose tracking output, the gesture recognition is then done based on longest common subsequence similarity measurement of upper body joint angles dynamics. In our experiment, we provide an extensive validation of the proposed upper body pose tracking from 3-D extremity movement which showed good results with various subjects in different environments. Regarding the gesture recognition based on joint angles dynamics, our experimental evaluation of five subjects doing six upper body gestures with average classification accuracies over 90% indicates the promise and feasibility of the proposed system.

In this paper, we provide a survey of the traffic sign detection literature, detailing detection systems for traffic sign recognition (TSR) for driver assistance. We separately describe the contributions of recent works to the various stages inherent in traffic sign detection: segmentation, feature extraction, and final sign detection. While TSR is a well-established research area, we highlight open research issues in the literature, including a dearth of use of publicly available image databases and the over-representation of European traffic signs. Furthermore, we discuss future directions of TSR research, including the integration of context and localization. We also introduce a new public database containing U.S. traffic signs.

The ability to monitor the state of a given roadway in order to better manage traffic congestion has become increasingly important. Sophisticated traffic management systems able to process both the static and mobile sensor data and provide traffic information for the roadway network are under development. In addition to typical traffic data such as flow, density, and average traffic speed, there is now strong interest in environmental factors such as greenhouse gases, pollutant emissions, and fuel consumption. It is now possible to combine high-resolution real-time traffic data with instantaneous emission models to estimate these environmental measures in real time. In this paper, a system that estimates average traffic fuel economy, CO2, CO, HC, and NOxemissions using a computer-vision-based methodology in combination with vehicle-specific power-based energy and emission models is presented. The CalSentry system provides not only typical traffic measures but also gives individual vehicle trajectories (instantaneous dynamics) and recognizes vehicle categories, which are used in the emission models to predict environmental parameters. This estimation process provides far more dynamic and accurate environmental information compared with static emission inventory estimation models.

In this paper we present a new method for automatic object detection in images and video sequences. As a classifier the popular Ad aBoost algorithm is used, that combines several weak classifiers into one strong classifier. To create a detector based on this classifier, the weak classifiers are set into relation during boosting by using a geometric model. All votes of the weak detectors are evaluated in a voting space. The voting space allows a detection with combinations of different object features. We trained and tested the proposed method with SIFT and kAS features and combinations of these. The learned detector is then used to localize objects in images and video sequences. The performance of the algorithm is examined based on selected image data.

Computer vision, speech and machine learning technologies play an important role and are increasingly used in today's vehicles to improve the safety as well as comfort in the car. Driving in particular presents a context in which a user's emotional state plays a significant role. Emotions have been found to affect cognitive style and performance. Even mildly positive feeling can have a profound effect on the flexibility and efficiency of thinking and problem solving. In this paper, we review some of the existing approaches for analyzing in-vehicle driver affect using audio and visual cues. We will discuss challenges in developing robust system and hopefully provide some insight in practical realization of such system. In particular, we present our ongoing efforts in collecting driving data using simulator as well as real world driving testbeds, and propose to utilize a multilevel audio-visual fusion scheme to utilize contextual information often available in co-existing tasks in an intelligent system.

Automobiles are quickly becoming more complex as new sensors and support systems are being added to improve safety and comfort. The next generation of intelligent driver assistance systems will need to utilize this wide array of sensors to fully understand the driving context and situation. Effective interaction requires these systems to examine the intentions, desires, and needs of the driver for preemptive actions which can help prepare for or avoid dangerous situations. This manuscript develops a real-time on-road prediction system able to detect a driver's intention to change lanes seconds before it occurs. In-depth analysis highlights the challenges when moving intent prediction from the laboratory to the road and provides detailed characterization of on-road performance.

Pedal errors have been reported as a cause of fatal traffic accidents. However it is not well understood why and when these pedal errors happen as well as how to mitigate them. In this paper, we study pedal error events in a real-world stop-and-go driving experiment, in which we quantitatively analyze foot behavior with measurements from embedded vehicle sensors (e.g. brake or acceleration pedal state) as well as a video input looking at the driver's foot. Our analysis shows some initial insights in factors influencing pedal errors (beside other possible causes like driver age, gender, and driver workload), including Sequential Effects and Cue Modality. We also develop a new vision-based approach for driver foot behavior analysis and use it to predict brake and acceleration pedal presses. Experimental results over twelve different subjects show that the proposed approach correctly detects pedal misapplications approximately 200ms before the actual press. This indicates the potential of applying this approach to predict and mitigate pedal errors in real-world driving.

Society is rapidly accepting the use of video cameras in many new and varied locations, but effective methods to utilize and manage the massive resulting amounts of visual data are only slowly developing. This paper presents a framework for live video analysis in which the behaviors of surveillance subjects are described using a vocabulary learned from recurrent motion patterns, for real-time characterization and prediction of future activities, as well as the detection of abnormalities. The repetitive nature of object trajectories is utilized to automatically build activity models in a 3-stage hierarchical learning process. Interesting nodes are learned through Gaussian mixture modeling, connecting routes formed through trajectory clustering, and spatio-temporal dynamics of activities probabilistically encoded using hidden Markov models. Activity models are adapted to small temporal variations in an online fashion using maximum likelihood regression and new behaviors are discovered from a periodic retraining for long-term monitoring. Extensive evaluation on various data sets, typically missing from other work, demonstrates the efficacy and generality of the proposed framework for surveillance-based activity analysis.

Automatic recognition of emotional states via speech signal has attracted increasing attention in recent years. A number of techniques have been proposed which are capable of providing reasonably high accuracy for controlled studio settings. However, their performance is considerably degraded when the speech signal is contaminated by noise. In this paper, we present a framework with adaptive noise cancellation as front end to speech emotion recognizer. We also introduce a new feature set based on cepstral analysis of pitch and energy contours. Experimental analysis shows promising results.

Confronted with the broad range of personalities of drivers on the roads today, in-vehicle technology must be able to function either in spite of or in harmony with each driver's style. Individual drivers act and respond in different manners under various conditions, and many Driver Assistance Systems would benefit from some measure of the driver's likely behaviors in these conditions. In this study we present several measures of driving style and show how they correlate with the predictability and responsiveness of the driver in several experimental conditions. In the first two experiments, one simulator and one real-world, we find that “aggressive” drivers are more consistent in behaviors and significantly more predictable than “non-aggressive” drivers. In the third experiment, however, we find that though “non-aggressive” drivers are less predictable, they are also significantly more receptive of feedback from Driver Assistance Systems. These results could affect the design, effectiveness, and feedback mechanisms of future Driver Assistance Systems.

This paper presents a method for improving the disparity values obtained with a stereo camera by applying an iconic Kalman filter and known ego-motion. The improvements are demonstrated in an application of determining the free space in a scene as viewed by a vehicle-mounted camera. Using disparity maps from a stereo camera and known camera motion, the disparity maps are first filtered by an iconic Kalman filter, operating on each pixel individually, thereby reducing variance and increasing the density of the filtered disparity map. Then, a stochastic occupancy grid is calculated from the filtered disparity map, providing a top-down view of the scene where the uncertainty of disparity measurements are taken into account. These occupancy grids are segmented to indicate a maximum depth free of obstacles, enabling the marking of free space in the accompanying intensity image. Even without motion of the camera, the quality of the disparity map is increased significantly. Applications of the intermediate results are discussed, enabling features such as motion detection and quantifying the certainty of the measurements. The evaluation shows significant improvement in disparity variance and disparity map density, and consequently an improvement in the application of marking free space.

With the introduction of intelligent driver support systems, vehicles have become more comfortable and safer. But, these systems require new sensors and the information they contain must be efficiently presented to the driver. The cognitive demands for interpreting these signals may prove to be a distraction with negative impact on driving performance. This work describes a unified visualization scheme, the Vehicle Iconic Surround Observer, capable of introducing new surround sensors into a common display environment which quickly conveys critical surround context with minimal driver interpretation.

In this paper, we present improved lane tracking using vehicle localization. Lane markers are detected using a bank of steerable filters, and lanes are tracked using Kalman filtering. On-road vehicle detection has been achieved using an active learning approach, and vehicles are tracked using a Condensation particle filter. While most state-of-the art lane tracking systems are not capable of performing in high-density traffic scenes, the proposed framework exploits robust vehicle tracking to allow for improved lane tracking in high density traffic. Experimental results demonstrate that lane tracking performance, robustness, and temporal response are significantly improved in the proposed framework, while also tracking vehicles, with minimal additional hardware requirements.

Automated analysis of human affective behavior has attracted increasing attention in recent years. Driver's emotion often influences driving performance which can be improved if the car actively responds to the emotional state of the driver. It is important for an intelligent driver support system to accurately monitor the driver's state in an unobtrusive and robust manner. Ever changing environment while driving poses a serious challenge to existing techniques for speech emotion recognition. In this paper, we utilize contextual information of the outside environment as well as inside car user to improve the emotion recognition accuracy. In particular, a noise cancellation technique is used to suppress the noise adaptively based on the driving context and a gender based context information is analyzed for developing the classifier. Experimental analyses show promising results.

Driver's body posture in 3D contains information potentially related to driver intent, driver affective state, and driver distraction. In this paper, we discuss issues and possibilities in developing a vision-based, markerless system to systematically explore the role of 3D driver posture dynamics for driver assistance. At high level, two main emphases in the proposed system are: (i) The coordination between real world driving testbed and simulation environment and (ii) The usefulness of driver posture dynamics is studied not only as an individual cue but also in relation with other contextual information (e.g. head dynamics, facial features, and vehicle dynamics). Some initial results in our experiment following these guidelines show the feasibility and promise of extracting and using 3D driver posture dynamics for driver assistance.

Microphones and cameras have been extensively used to observe and detect human activity and to facilitate natural modes of interaction between humans and intelligent systems. Human brain processes the audio and video modalities, extracting complementary and robust information from them. Intelligent systems with audiovisual sensors should be capable of achieving similar goals. The audiovisual information fusion strategy is a key component in designing such systems. In this paper, we exclusively survey the fusion techniques used in various audiovisual information fusion tasks. The fusion strategy used tends to depend mainly on the model, probabilistic or otherwise, used in the particular task to process sensory information to obtain higher level semantic information. The models themselves are task oriented. In this paper, we describe the fusion strategies and the corresponding models used in audiovisual tasks such as speech recognition, tracking, biometrics, affective state recognition, and meeting scene analysis. We also review the challenges and existing solutions and also unresolved or partially resolved issues in these fields. Specifically, we discuss established and upcoming work in hierarchical fusion strategies and cross-modal learning techniques, identifying these as critical areas of research in the future development of intelligent systems.

We present a novel real-time computer-vision system that robustly discriminates which of the front-row seat occupants is accessing the infotainment controls. The knowledge of who is the user-that is, driver, passenger, or no one-can alleviate driver distraction and maximize the passenger infotainment experience. The system captures visible and near-infrared images of the front-row seat area in the vehicle. The algorithm uses a modified histogram-of-oriented-gradients feature descriptor to represent the image area over the infotainment controls and a support vector machine (SVM) and median filtering over time to classify each image to one of the three classes with 97.9% average correct classification rate. This rate was achieved over a wide range of illumination conditions, human subjects, and times of day. With an offset of 5 pixels in any direction, the rate could still be maintained at better than 85%. This approach represents an alternative to detecting and tracking the hand movements and then classifying the hands into the respective classes. This approach demonstrates the ability to achieve good classification rates, despite the presence of vast illumination changes of the vehicle environment.

Driver distraction and inattention are prominent causes of automotive collisions. To enable driver-assistance systems to address these problems, we require new sensing approaches to infer a driver's focus of attention. In this paper, we present a new procedure for static head-pose estimation and a new algorithm for visual 3-D tracking. They are integrated into the novel real-time (30 fps) system for measuring the position and orientation of a driver's head. This system consists of three interconnected modules that detect the driver's head, provide initial estimates of the head's pose, and continuously track its position and orientation in six degrees of freedom. The head-detection module consists of an array of Haar-wavelet Adaboost cascades. The initial pose estimation module employs localized gradient orientation (LGO) histograms as input to support vector regressors (SVRs). The tracking module provides a fine estimate of the 3-D motion of the head using a new appearance-based particle filter for 3-D model tracking in an augmented reality environment. We describe our implementation that utilizes OpenGL-optimized graphics hardware to efficiently compute particle samples in real time. To demonstrate the suitability of this system for real driving situations, we provide a comprehensive evaluation with drivers of varying ages, race, and sex spanning daytime and nighttime conditions. To quantitatively measure the accuracy of system, we compare its estimation results to a marker-based cinematic motion-capture system installed in the automotive testbed.

This paper introduces a general active-learning framework for robust on-road vehicle recognition and tracking. This framework takes a novel active-learning approach to building vehicle-recognition and tracking systems. A passively trained recognition system is built using conventional supervised learning. Using the query and archiving interface for active learning (QUAIL), the passively trained vehicle-recognition system is evaluated on an independent real-world data set, and informative samples are queried and archived to perform selective sampling. A second round of learning is then performed to build an active-learning-based vehicle recognizer. Particle filter tracking is integrated to build a complete multiple-vehicle tracking system. The active-learning-based vehicle-recognition and tracking (ALVeRT) system has been thoroughly evaluated on static images and roadway video data captured in a variety of traffic, illumination, and weather conditions. Experimental results show that this framework yields a robust efficient on-board vehicle recognition and tracking system with high precision, high recall, and good localization.

Automated analysis of human affective behavior has attracted increasing attention in recent years. With the research shift toward spontaneous behavior, many challenges have come to surface ranging from database collection strategies to the use of new feature sets (e.g., lexical cues apart from prosodic features). Use of contextual information, however, is rarely addressed in the field of affect expression recognition, yet it is evident that affect recognition by human is largely influenced by the context information. Our contribution in this paper is threefold. First, we introduce a novel set of features based on cepstrum analysis of pitch and intensity contours. We evaluate the usefulness of these features on two different databases: Berlin Database of emotional speech (EMO-DB) and locally collected audiovisual database in car settings (CVRRCar-AVDB). The overall recognition accuracy achieved for seven emotions in the EMO-DB database is over 84% and over 87% for three emotion classes in CVRRCar-AVDB. This is based on tenfold stratified cross validation. Second, we introduce the collection of a new audiovisual database in an automobile setting (CVRRCar-AVDB). In this current study, we only use the audio channel of the database. Third, we systematically analyze the effects of different contexts on two different databases. We present context analysis of subject and text based on speaker/text-dependent/-independent analysis on EMO-DB. Furthermore, we perform context analysis based on gender information on EMO-DB and CVRRCar-AVDB. The results based on these analyses are promising.

This study is to investigate the fundamental problems of, (1) facial feature detection and localization, especially eye features; and (2) eye dynamics, including tracking and blink detection. We first describe our contribution to eye localization. Following that, we discuss a simultaneous eye tracking and blink detection system. Facial feature detection is solved in a general object detection framework and its performance for eye localization is presented. A binary tree representation based on feature dependency partitions the object feature space in a coarse to fine manner. In each compact feature subspace, independent component analysis (ICA) is used to get the independent sources, whose probability density functions (PDFs) are modeled by Gaussian mixtures. When applying this representation for the task of eye detection, a subwindow is used to scan the entire image and each obtained image patch is examined using Bayesian criteria to determine the presence of an eye subject. After the eyes are automatically located with binary tree-based probability learning, interactive particle filters are used for simultaneously tracking the eyes and detecting the blinks. The particle filters use classification-based observation models, in which the posterior probabilities are evaluated by logistic regressions in tensor subspaces. Extensive experiments are used to evaluate the performance from two aspects, (1) blink detection rate and the accuracy of blink duration in terms of the frame numbers; (2) eye tracking accuracy. We also present an experimental setup for obtaining the benchmark data in tracking accuracy evaluation. The experimental evaluation demonstrates the capability of this approach.

Recently a large amount of research has been devoted to automatic activity analysis. Typically, activities have been defined by their motion characteristics and represented by trajectories. These trajectories are collected and clustered to determine typical behaviors. This paper evaluates different similarity measures and clustering methodologies to catalog their strengths and weaknesses when utilized for the trajectory learning problem. The clustering performance is measured by evaluating the correct clustering rate on different datasets with varying characteristics.

Reverberant environments pose a challenge to speech acquisition from distant microphones. Approaches using microphone arrays have met with limited success. Recent research using audio-visual sensors for tasks such as speaker localization has shown improvement over traditional audio-only approaches. Using computer vision techniques we can estimate the orientation of the speaker's head in addition to the location of the speaker. In this paper we study the utility of using the head pose information for effective beamforming and clean speech acquisition from distant microphones. The improvements in speech recognition accuracy relative to that of a close talking microphone are presented and the results provide sufficient motivation for incorporating head pose information in beamforming techniques.

We introduce a vision based, marker less upper body pose tracking approach that first tracks the 3D movements of extremities, including head and hands. Then based on the knowledge of upper body model, these extremity movements are used to predict the whole upper body motion as an inverse kinematics problem. The experimental validation showed the promise of applying this approach in several smart environments and HCI situations, e.g. user activity observation in driving scene, meeting room, teleconference scene.

A multiview, multimodal fusion algorithm for classifying marine plankton is described and its performance is evaluated on laboratory data from live animals. The algorithm uses support vector machines with softmax outputs to classify either acoustical or optical features. Outputs from these single-view classifiers are then combined together using a feedback network with confidence weighting. For each view or modality, the initial classification and classifications from all other views and modalities are confidence-weighted and combined to render a final, improved classification. Simple features are computed from acoustic and video data with an aim at noise robustness. The algorithm is tested on acoustic and video data collected in the laboratory from live, untethered copepods and mysids (two dominant crustacean zooplankton). It is shown that the algorithm is able to yield significant (> 50%) reductions in error by combining views together. In addition, it is shown that the algorithm is able boost performance by giving more weight to views or modalities that are more discriminant than others, without any a priori knowledge of which views are more discriminant.

The capacity to estimate the head pose of another person is a common human ability that presents a unique challenge for computer vision systems. Compared to face detection and recognition, which have been the primary foci of face-related vision research, identity-invariant head pose estimation has fewer rigorously evaluated systems or generic solutions. In this paper, we discuss the inherent difficulties in head pose estimation and present an organized survey describing the evolution of the field. Our discussion focuses on the advantages and disadvantages of each approach and spans 90 of the most innovative and characteristic papers that have been published on this topic. We compare these systems by focusing on their ability to estimate coarse and fine head pose, highlighting approaches that are well suited for unconstrained environments.

The main focus of this special issue is to bring together solutions from research and engineering for the automatic understanding of a complex scene, via a multimodal array of sensors with automation of adaptation as the theme connecting various research areas. Four papers comprise this special issue.

Driver behavioral cues may present a rich source of information and feedback for future intelligent advanced driver-assistance systems (ADASs). With the design of a simple and robust ADAS in mind, we are interested in determining the most important driver cues for distinguishing driver intent. Eye gaze may provide a more accurate proxy than head movement for determining driver attention, whereas the measurement of head motion is less cumbersome and more reliable in harsh driving conditions. We use a lane-change intent-prediction system (McCall et al., 2007) to determine the relative usefulness of each cue for determining intent. Various combinations of input data are presented to a discriminative classifier, which is trained to output a prediction of probable lane-change maneuver at a particular point in the future. Quantitative results from a naturalistic driving study are presented and show that head motion, when combined with lane position and vehicle dynamics, is a reliable cue for lane-change intent prediction. The addition of eye gaze does not improve performance as much as simpler head dynamics cues. The advantage of head data over eye data is shown to be statistically significant (p < 0.01) 3 s ahead of lane-change situations, indicating that there may be a biological basis for head motion to begin earlier than eye motion during "lane-change"-related gaze shifts.

The nine papers in this special section examine perception, planning, and navigation for intelligent vehicles.

In this paper, we introduce a novel laser-based wide-area heads-up windshield display which is capable of actively interfacing with a human as part of a driver assistance system. The dynamic active display (DAD) is a unique prototype interface that presents safety-critical visual icons to the driver in a manner that minimizes the deviation of his or her gaze direction without adding to unnecessary visual clutter. As part of an automotive safety system, the DAD presents alerts in the field of view of the driver only if necessary, which is based upon the state and pose of the driver, vehicle, and environment. This paper examines the effectiveness of DAD through a comprehensive comparative experimental evaluation of a speed compliance driver assistance system, which is implemented on a vehicular test bed. Three different types of display protocols for assisting a driver to comply with speed limits are tested on actual roadways, and these are compared with a conventional dashboard display. Given the inclination, drivers who are given an overspeed warning alert reduced the time required to slow down to the speed limit by 38% (p < 0.01) as compared with the drivers not given the alert. Additionally, certain alerts decreased distraction levels by reducing the time spent looking away from the road by 63% (p < 0.01). Ultimately, these alerts demonstrate the utility and promise of the DAD system.

This paper presents a general framework for live video analysis. The activities of surveillance subjects are described using a spatio-temporal vocabulary learned from recurrent motion patterns. The repetitive nature of object trajectories is used to build a topographical scene description where nodes are points of interest (POT) and the edges correspond to activity paths (AP). The POI are learned through as a mixture of Gaussians and AP by clustering trajectories. The paths are probabilistically represented by hidden Markov models and adapt to temporal variations using online maximum likelihood regression (MLLR) and through a periodic batch update. Using the scene graph, new trajectories can be analyzed in online fashion to categorize past and present activity, predict future behavior, and detect abnormalities.

Tracking humans in an indoor environment is an essential part of surveillance systems. Vision based and microphone array based trackers have been extensively researched in the past. Audio-visual tracking frameworks have also been developed. In this paper we consider human tracking to be a specific instance of a more general problem of information fusion in multimodal systems. Dynamic Bayesian networks have been the modeling technique of choice to build such information fusion schemes. The complexity and non-Gaussianity of distributions of the dynamic Bayesian networks for such multimodal systems have led to the use of particle filters as an approximate inference technique. In this paper we present an alternative approach to the information fusion problem. The iterative decoding algorithm is based on the theory of turbo codes and factor graphs used in communication systems. We modify and adapt the iterative decoding algorithm to do probabilistic inference for the problem of tracking humans in an indoor space, using multiple cameras and microphone arrays.

The fusion of information from heterogenous sensors is crucial to the effectiveness of a multimodal system. Noise affect the sensors of different modalities independently. A good fusion scheme should be able to use local estimates of the reliability of each modality to weight the decisions. This paper presents an iterative decoding based information fusion scheme motivated by the theory of turbo codes. This fusion framework is developed in the context of hidden Markov models. We present the mathematical framework of the fusion scheme. We then apply this algorithm to an audio-visual speech recognition task on the GRID audio-visual speech corpus and present the results.

This paper presents an adaptive framework for live video analysis. The activities of surveillance subjects are described using a spatio-temporal vocabulary learned from recurrent motion patterns. The repetitive nature of object trajectories are used to build a topographical map, where nodes are points of interest and the edges correspond to activities, to describe a scene. The graph is learned in an unsupervised manner but is flexible and able to adjust to changes in the environment or other scene variations.

We propose an integrated framework for automated hand model initialization and tracking using voxel data. Starting with an initial specific hand pose, the Laplacian Eigenspace (LE) based segmentation method is applied to segment hand voxel into different parts. This segmentation result is then used to extend the kinematically constrained gaussian mixture model (KC-GMM) method for articulated body pose inference with an automated hand model initialization. Our experiment with both synthesized hand voxel and real hand voxel captured from multi-perspective thermal cameras show that by combining the two methods, we have a more powerful system than using each one solitarily.

This paper presents a methodology for analyzing multimodal and multiperspective systems for person surveillance. Using an experimental testbed consisting of two color and two infrared cameras, we can accurately register the color and infrared imagery for any general scene configuration, expanding the scope of multispectral analysis beyond the specialized long-range surveillance experiments of previous approaches to more general scene configurations common to unimodal approaches. We design an algorithmic framework for detecting people in a scene that can be generalized to include color, infrared, and/or disparity features. Using a combination of a histogram of oriented gradient (HOG) feature-based support vector machine and size/depth-based constraints, we create a probabilistic score for evaluating the presence of people. Using this framework, we train person detectors using color stereo and infrared stereo features as well as tetravision-based detectors that combine the detector outputs from separately trained color stereo and infrared stereo-based detectors. Additionally, we incorporate the trifocal tensor in order to combine the color and infrared features in a unified detection framework and use these trained detectors for an experimental evaluation of video sequences captured with our designed testbed. Our evaluation definitively demonstrates the performance gains achievable when using the trifocal framework to combine color and infrared features in a unified framework. Both of the trifocal setups outperform their unimodal equivalents, as well as the tetravision-based analysis. Our experiments also demonstrate how the trained detector generalizes well to different scenes and can provide robust input to an additional tracking framework.

This paper presents a survey of trajectory-based activity analysis for visual surveillance. It describes techniques that use trajectory data to define a general set of activities that are applicable to a wide range of scenes and environments. Events of interest are detected by building a generic topographical scene description from underlying motion structure as observed over time. The scene topology is automatically learned and is distinguished by points of interest and motion characterized by activity paths. The methods we review are intended for real-time surveillance through definition of a diverse set of events for further analysis triggering, including virtual fencing, speed profiling, behavior classification, anomaly detection, and object interaction.

This paper presents two different types of visual activity analysis modules based on vehicle tracking. The highway monitoring module accurately classifies vehicles into eight different types and collects traffic flow statistics by leveraging tracking information. These statistics are continuously accumulated to maintain daily highway models that are used to categorize traffic flow in real time. The path modeling block is a more general analysis tool that learns the normal motions encountered in a scene in an unsupervised fashion. The spatiotemporal motion characteristics of these motion paths are encoded by a hidden Markov model. With the path definitions, abnormal trajectories are detected and future intent is predicted. These modules add realtime situational awareness to highway monitoring for high-level activity and behavior analysis.

This paper presents an new framework for homography-based analysis of pedestrian-vehicle activity in crowded scenes. Planar homography constraint is exploited to extract view-invariant object features including footage area and velocity of objects on the ground plane. Spatio-temporal relationships between people- and vehicle- tracks are represented by a semantic event. Context awareness of the situation is achieved by the estimated density distribution of objects and the anticipation of possible directions of near-future tracks using piecewise velocity history. Single-view and multi-view based homography mapping options are compared. Our framework can be used to enhance situational awareness for disaster prevention, human interactions in structured environments, and crowd movement analysis at wide regions

This paper deals with the development of Human-Centric Intelligent Driver Assistance Systems. Rear-end collisions account for a large portion of traffic accidents. To help mitigate this problem, predictive braking systems and adaptive cruise control systems have been developed. However, these types of systems usually rely solely on the vehicle and vehicle surround sensors, either ignoring the human component of driving or learning the driver's control behavior using only these sensors. As with all human-computer interfaces, this has the potential to work against the driver, distract the driver further, or even annoy the driver so that the driver ignores or disables the system. It is, therefore, important to directly take the driver's intended actions into account when designing a driver assistance system. By using a probabilistic model for the system, warnings and preventative measures can be constructed based on varying levels of situational severity and driver attentiveness and intent. The research is based upon carefully conducted experimental trials involving a human subjects driving in natural manner and on typical freeways in the USA. The experiments, designed by inputs from cognitive scientist, were conducted in a specially designed instrumented vehicle to record important cues associated with driver's behavior, vehicle state, and vehicle surround in a synchronized manner. Quantitative results and analysis of the experimental trials are presented to show the feasibility and promise of this framework to predict the driver's intent to brake, the need for braking given the current situation, and at what level the driver should be warned

This paper presents the design and application of novel multisensory testbeds for collection, synchronization, archival, and analysis of multimodal data for health monitoring of transportation infrastructures. The framework for data capture from vision and seismic sensors is described, and the important issue of synchronization between these modalities is addressed. Computer-vision algorithms are used to detect and track vehicles and extract their properties. It is noted that the video and seismic sensors in the testbed supply complementary information about passing vehicles. Data fusion between features obtained from these modalities is used to perform vehicle classification. Experimental results of vehicle detection, tracking, and classification obtained with these testbeds are described

This paper describes the recent research on the enhancement of pedestrian safety to help develop a better understanding of the nature, issues, approaches, and challenges surrounding the problem. It presents a comprehensive review of research efforts underway dealing with pedestrian safety and collision avoidance. The importance of pedestrian protection is emphasized in a global context, discussing the research programs and efforts in various countries. Pedestrian safety measures, including infrastructure enhancements and passive safety features in vehicles, are described, followed by a systematic description of active safety systems based on pedestrian detection using sensors in vehicle and infrastructure. The pedestrian detection approaches are classified according to various criteria such as the type and configuration of sensors, as well as the video cues and classifiers used in detection algorithms. It is noted that collision avoidance not only requires detection of pedestrians but also requires collision prediction using pedestrian dynamics and behavior analysis. Hence, this paper includes research dealing with probabilistic modeling of pedestrian behavior for predicting collisions between pedestrians and vehicles.

This paper presents an analysis of color-, infrared-, and multimodal-stereo approaches to pedestrian detection. We design a four-camera experimental testbed consisting of two color and two infrared cameras for capturing and analyzing various configuration permutations for pedestrian detection. We incorporate this four-camera system in a test vehicle and conduct comparative experiments of stereo-based approaches to obstacle detection using unimodal color and infrared imageries. A detailed analysis of the color and infrared features used to classify detected obstacles into pedestrian regions is used to motivate the development of a multimodal solution to pedestrian detection. We propose a multimodal trifocal framework consisting of a stereo pair of color cameras coupled with an infrared camera. We use this framework to combine multimodal-image features for pedestrian detection and to demonstrate that the detection performance is significantly higher when color, disparity, and infrared features are used together. This result motivates experiments and discussion toward achieving multimodal-feature combination using a single color and a single infrared camera arranged in a cross-spectral stereo pair. We demonstrate an approach to registering multiple objects across modalities and provide an experimental analysis that highlights issues and challenges of pursuing the cross-spectral approach to multimodal and multiperspective pedestrian analysis.

In this paper, we demonstrate a driver intent inference system that is based on lane positional information, vehicle parameters, and driver head motion. We present robust computer vision methods for identifying and tracking freeway lanes and driver head motion. These algorithms are then applied and evaluated on real-world data that are collected in a modular intelligent vehicle test bed. Analysis of the data for lane change intent is performed using a sparse Bayesian learning methodology. Finally, the system as a whole is evaluated using a novel metric and real-world data of vehicle parameters, lane position, and driver head motion.

This paper presents investigations into the role of computer-vision technology in developing safer automobiles. We consider vision systems, which cannot only look out of the vehicle to detect and track roads and avoid hitting obstacles or pedestrians but simultaneously look inside the vehicle to monitor the attentiveness of the driver and even predict her intentions. In this paper, a systems-oriented framework for developing computer-vision technology for safer automobiles is presented. We will consider three main components of the system: environment, vehicle, and driver. We will discuss various issues and ideas for developing models for these main components as well as activities associated with the complex task of safe driving. This paper includes a discussion of novel sensory systems and algorithms for capturing not only the dynamic surround information of the vehicle but also the state, intent, and activity patterns of drivers

In the interest of 24-7 long-term surveillance, a truly robust, adaptive, and fast background-foreground segmentation technique is required. This paper deals with the especially difficult but extremely common problems of moving backgrounds, shadows, highlights, and illumination changes. To produce reliable foreground extraction in the face of these problems, the best practical aspects of two algorithms, Codebook Segmentation[6] and HSV Shadow Suppression[2] are combined. The main contribution of this paper is the introduction of the "Hybrid Cone-Cylinder" Codebook (HC3) model. Results show superior speed and quantitatively better performance in many different conditions and environments. Applications include people-tracking with Omni-directional cameras and vehicle-counting with rectilinear cameras.

This paper proposes a concept of panoramic appearance map to perform reidentification of a people who leave the scene and reappear after some time. The map is a compact signature of appearance information of a person extracted from multiple cameras. The person is detected and tracked in multiple cameras and triangulation is used to accurately localize the person in 3-D. A virtual cylinder is formed around the person's location and mapped onto an image with the horizontal axis representing the azimuth angle and vertical axis representing the height. Each bin in the map image gets the appearance information from all the cameras which can observe it. The maps between different tracks are matched using a weighted metric. Experimental results showing person matching and reidentification show the effectiveness of the approach.

This paper presents a method for registering multimodal imagery in short range surveillance situations when the differences in object depths preclude any global registration techniques. An analysis of multimodal registration approaches gives insight into the limitations of global assumptions and motivates the developed algorithm. Using calibrated stereo imagery, we use maximization of mutual information in sliding correspondence windows that inform a disparity voting scheme to demonstrate successful registration of color and thermal images. Extensive testing of scenes with multiple people at different depths and levels of occlusion shows high rates of successful registration and gives a reliable framework for further processing and analysis of the multimodal imagery.

Visual surveillance systems intend to extract meaning from a scene. Two initial steps for this extraction are the detection and tracking of objects followed by the classification of these objects. Often times these are viewed as separate problems where each is solved by an individual module. These tasks should not be done individually because they can help one another. This paper demonstrates the benefit gained both in tracking and classification through the communication between these individual modules. This is shown on a real-time system monitoring highway traffic. The system retreives online video at 10 frames/sec and conducts tracking and classification simultaneously. Results show an improvement from 74% to 88% accuracy in classification results.

In this paper we introduce the NDS-Forest data structure, which can be used for the calculation and representation of Maximally Stable Extremal Regions in real-time video. In contrast to the standard MSER algorithm, the NDS-Forest stores information about the extremal regions as they are formed, mak-ing it unnecessary to regrow the regions from seed pixels. Using the NDS-Forest structure, we describe a system that uses MSERs in an automobile for face registration, segmentation, and pose estimation of the driver. 1

We propose a new approach to the hand pose estimation problem using only volume information. We describe a thermal and color image-based approach to generate silhouettes of the hand with which voxel images are produced using shape-from-silhouette. We assume a 16-component, 27 degree-of-freedom kinematically constrained Gaussian mixture model, and fit it over the voxel images. We constrain the otherwise freely-arranged components of this model by a system of equations describing joint characteristics parameterized by component centroids and orientations. We use the EM algorithm and steepest descent to estimate the parameters of the model such that they yield the maximum likelihood and kinematically correct pose estimates. We demonstrate the effectiveness of the proposed system on synthesized as well as captured voxel images of the hand, and show that given appropriate initial conditions, the iterative model parameter estimation procedure effectively converges to and tracks the "voxelized" articulated hand.

Awareness of what surrounds a vehicle directly affects the safe driving and maneuvering of an automobile. This paper focuses on the capture of vehicle surroundings using video inputs. Surround information or maps can help in studies of driver behavior as well as provide critical input in the development of effective driver assistance systems. A survey of literature related to surround analysis is presented, emphasizing detecting objects such as vehicles, pedestrians, and other obstacles. Omni cameras, which give a panoramic view of the surroundings, can be useful for visualizing and analyzing the nearby surroundings of the vehicle. The concept of Dynamic Panoramic Surround (DPS) map that shows the nearby surroundings of the vehicle and detects the objects of importance on the road is introduced. A novel approach for synthesizing the DPS using stereo and motion analysis of video images from a pair of omni cameras on the vehicle is developed. Successful generation of the DPS in experimental runs on an instrumented vehicle test bed is demonstrated. These experiments prove the basic feasibility and show promise of omni-camera-based DPS capture algorithm to provide useful semantic descriptors of the state of moving vehicles and obstacles in the vicinity of a vehicle

Driver-assistance systems that monitor driver intent, warn drivers of lane departures, or assist in vehicle guidance are all being actively considered. It is therefore important to take a critical look at key aspects of these systems, one of which is lane-position tracking. It is for these driver-assistance objectives that motivate the development of the novel "video-based lane estimation and tracking" (VioLET) system. The system is designed using steerable filters for robust and accurate lane-marking detection. Steerable filters provide an efficient method for detecting circular-reflector markings, solid-line markings, and segmented-line markings under varying lighting and road conditions. They help in providing robustness to complex shadowing, lighting changes from overpasses and tunnels, and road-surface variations. They are efficient for lane-marking extraction because by computing only three separable convolutions, we can extract a wide variety of lane markings. Curvature detection is made more robust by incorporating both visual cues (lane markings and lane texture) and vehicle-state information. The experiment design and evaluation of the VioLET system is shown using multiple quantitative metrics over a wide variety of test conditions on a large test path using a unique instrumented vehicle. A justification for the choice of metrics based on a previous study with human-factors applications as well as extensive ground-truth testing from different times of day, road conditions, weather, and driving scenarios is also presented. In order to design the VioLET system, an up-to-date and comprehensive analysis of the current state of the art in lane-detection research was first performed. In doing so, a comparison of a wide variety of methods, pointing out the similarities and differences between methods as well as when and where various methods are most useful, is presented

This paper presents a track-based system for human movement analysis and privacy protection. Our system is adaptive to environmental contexts such as illumination variations, complex moving cast shadows, different camera perspectives, and diverse sue scenarios. Most of outdoor surveillance systems have been targeting at specific environmental situation: i.e., specific time, place, and activity scenarios. We address that more general human movement analysis systems should be able to handle multiple heterogeneous situations in an adaptive manner. We introduce the concept of "spatio-temporal personal boundary" to represent different grouping patterns of human tracks, and we incorporate the concept with various site models. Experimental evaluations with extensive outdoor data show our system's robustness to environmental changes and effectiveness to properly handle various environmental contexts.

In a matter of only about three decades, Computer vision as a discipline has made a significant impact on a number of diverse application domains. Beginning from the seventies through the nineties, computer vision started proving its practical value in a wide range of diverse application domains including medical diagnostics, manufacturing, environmental monitoring, space exploration, and military systems such as automatic target recognition, precision weapons, and reconnaissance.

This paper describes an approach for detecting objects in front of an automobile using wide field of view stereo with a pair of omni cameras. Several configurations are suggested for effective detection of vehicles and pedestrians. The omni cameras are calibrated using sets of parallel lines on a parking lot. The calibration is used to rectify the omni images. Stereo matching is performed on the rectified images to detect other vehicles and pedestrians. Experimental results show promise of detecting these objects on the road.

Pixelwise Gaussian mixture based background modeling algorithm proposed in S. Stauffer and W.E.L. Grimson (1999 and 2000) has been proved to be robust for many motion detection applications. However, the algorithm is not sensitive to fast motion. One possible solution is to introduce local correlations. Starting from this, this paper proposes to use the Gaussian mixture globally for modeling the distribution of the difference image between the new frame and the estimated background. Experimental evaluation validates the algorithm. Motivated by the demands of selecting the more appropriate algorithm for a specific application, qualitative and quantitative performance comparisons of these two approaches are presented. We proposes three metrics. One is to characterize the pixel level accuracy and the other two are to evaluate the errors in the object level. Pros and cons of both algorithms are summarized.

Intelligent environments can be viewed as systems where humans and machines (rooms) collaborate. Intelligent (or smart) environments need to extract and maintain an awareness of a wide range of events and human activities occurring in these spaces. This requirement is crucial for supporting efficient and effective interactions among humans as well as humans and intelligent spaces. Visual information plays an important role for developing accurate and useful representation of the static and dynamic states of an intelligent environment. Accurate and efficient capture, analysis, and summarization of the dynamic context requires the vision system to work at multiple levels of semantic abstractions in a robust manner. In this paper, we present details of a long-term and ongoing research project, where indoor intelligent spaces endowed with a range of useful functionalities are designed, built, and systematically evaluated. Some of the key functionalities include: intruder detection; multiple person tracking; body pose and posture analysis; person identification; human body modeling and movement analysis; and for integrated systems for intelligent meeting rooms, teleconferencing, or performance spaces. The paper includes an overall system architecture to support design and development of intelligent environments. Details of panoramic (omnidirectional) video camera arrays, calibration, video stream synchronization, and real-time capture/processing are discussed. Modules for multicamera-based multiperson tracking, event detection and event based servoing for selective attention, voxelization, streaming face recognition, are also discussed. The paper includes experimental studies to systematically evaluate performance of individual video analysis modules as well as to evaluate basic feasibility of an integrated system for dynamic context capture and event based servoing, and semantic information summarization.

In this paper we discuss the intelligent or smart camera based systems for capturing visual contextual information at multiple levels of information abstraction. A distributed video array, with multiple omnidirectional and rectilinear cameras, is used to acquire visual information. System architecture as well as 3D tracking and human identification modules are described. Examples of the use of distributed video arrays in indoor, outdoor, and mobile environments are presented.

In this paper, we present a fast algorithm for automatically generating a 3D model of a human face and synthesizing affects on it. This work is a step towards low bandwidth virtual tele-presence using avatars. The idea is to generate a 3D model of a person's face and animate different emotions on the face at a remote observer's end, thus eliminating the need to send raw video. This research is part of a larger project, which deals with development of novel sensory systems and interfaces to enhance safety of a driver. The proposed algorithm uses a stereo pair of images of a driver's face and creates a 3D mesh model by deforming and texture mapping a predefined 3D mesh.

Moving shadows need careful consideration in the development of robust dynamic scene analysis systems. Moving shadow detection is critical for accurate object detection in video streams since shadow points are often misclassified as object points, causing errors in segmentation and tracking. Many algorithms have been proposed in the literature that deal with shadows. However, a comparative evaluation of the existing approaches is still lacking. In this paper, we present a comprehensive survey of moving shadow detection approaches. We organize contributions reported in the literature in four classes two of them are statistical and two are deterministic. We also present a comparative empirical evaluation of representative algorithms selected from these four classes. Novel quantitative (detection and discrimination rate) and qualitative metrics (scene and object independence, flexibility to shadow situations, and robustness to noise) are proposed to evaluate these classes of algorithms on a benchmark suite of indoor and outdoor video sequences. These video sequences and associated "ground-truth" data are made available at http://cvrr.ucsd.edu/aton/shadow to allow for others in the community to experiment with new algorithms and metrics.

Room reverberation is typically the main obstacle for designing robust microphone-based source localization systems. The purpose of the paper is to analyze the achievable performance of acoustical source localization methods when room reverberation is present. To facilitate the analysis, we apply well known results from room acoustics to develop a simple but useful statistical model for the room transfer function. The properties of the statistical model are found to correlate well with results from real data measurements. The room transfer function model is further applied to analyze the statistical properties of some existing methods for source localization. In this respect we consider especially the asymptotic error variance and the probability of an anomalous estimate. A noteworthy outcome of the analysis is that the so-called PHAT time-delay estimator is shown to be optimal among a class of cross-correlation based time-delay estimators. To verify our results on the error variance and the outlier probability we apply the image method for simulation of the room transfer function.

The main obstacle for designing robust microphone-based source localization systems is the effects of room reverberation. Utilizing results from statistical room acoustics, we analyze the performance of GCC-based methods for time-delay estimation. An interesting outcome of the analysis is that the so-called PHAT time-delay estimator is shown to be optimal among a class of cross-correlation based time-delay estimators.

We present a framework for articulated body model acquisition and tracking from voxel data. A 3D voxel reconstruction of the person's body is computed from silhouettes extracted from four cameras. The model acquisition process is fully automated. In the first frame, body parts are located sequentially. The head is located first, since its shape and size are unique and stable. Other parts are found by sequential template growing and fitting. This initial estimate of body part locations, sizes and orientations is then used as a measurement for the extended Kalman filter which ensures a valid articulated body model. The same filter, with a slightly modified state and state transition matrix, is then used for tracking. The performance of the system has been evaluated on several video sequences with promising results.

Robustness to changes in illumination conditions as well as viewing perspectives is an important requirement for many computer vision applications. One of the key factors in enhancing the robustness of dynamic scene analysis is that of accurate and reliable means for shadow detection. Shadow detection is critical for correct object detection in image sequences. Many algorithms have been proposed in the literature that deal with shadows. However, a comparative evaluation of the existing approaches is still lacking. In this paper, the full range of problems underlying the shadow detection is identified and discussed. We classify the proposed solutions to this problem using a taxonomy of four main classes, deterministic model and non-model based, and statistical parametric and nonparametric. Novel quantitative (detection and discrimination accuracy) and qualitative metrics (scene and object independence, flexibility to shadow situations and robustness to noise) are proposed to evaluate these classes of algorithms on a benchmark suite of indoor and outdoor video sequences.

We present the real-time flow imaging and classification system (REFLICS), a real-time machine vision system for detecting and classifying pelagic fish eggs (e.g. sardine, anchovy) flowing through a shipboard pumping and filtering system. REFLICS images the dynamic flow with a progressive-scan area camera and a synchronized backlight strobe. Digitization and processing occur on a dual processor Pentium II PC and a pipeline image processing board. REFLICS first uses a segmentation algorithm to locate fish egg-like objects in the image and then a classifier to determine fish eggs, species and development stages (ages). We present the integrated system design of REFLICS and performance results. REFLICS can perform in 60 Hz real-time, classify fish eggs with low false negative rates on data collected from real cruises, and work in harsh conditions aboard ships at sea. REFLICS allows more accurate and timely survey of fish stock in a cost-effective manner.

We present an algorithm for segmentation of traffic scenes that distinguishes moving objects from their moving cast shadows. A fading memory estimator calculates mean and variance of all three color components for each background pixel. Given the statistics for a background pixel, simple rules for calculating its statistics when covered by a shadow are used. Then, MAP classification decisions are made for each pixel. In addition to the color features, we examine the use of neighborhood information to produce smoother classification. We also propose the use of temporal information by modifying class a priori probabilities based on predictions from the previous frame.

A collection of distributed databases forms an important architectural component of the ATON project for networked incidence management of highway traffic. The database sub-architecture supports the architectural integration of many thematic areas of the ATON, and provides many high level abstractions that semantically correspond to traffic incidents. These databases are queried for the detection of local or distributed traffic incidents by many distributed control and analysis algorithms. The abstract database sub-architecture leads to many databases of different types having different functions. The semantic event/activity database is one of them. The distributed multi-sensory sub-architecture detects atomic semantic events that occur in the environment. The event/activity database stores them and their temporal structures. An activity is a temporal composition of atomic events. A specialized query language is used to flexibly define and detect activities of interest. The query language embodies high level semantic pattern matching abstractions. Preliminary results of using the event/activity database are also presented.

Intelligent environments provide challenging research problems for natural and efficient interfaces between humans and computers as well as between humans. We present a multimodal sensory intelligent system testbed based on some general requirements for developing intelligent environments. We also present rigorous experimental investigations on the processing and control modules for the active camera networks and the microphone array which are embedded in the intelligent room. An integrated intelligent system is developed utilizing four basic modules for visual and audio processing. The integrated system has the functionality of human tracking, active camera control, face recognition and speaker recognition. This system is demonstrated to be suitable for teleconferencing type of applications.

Describes an infrastructural framework to support interactivity among entities that are not residing in the same physical environment. In such an environment, mobile robots supported by a perceptual information infrastructure gather information and communicate with people in a distant place instead of the users. This is one of the possible and promising applications of robots based on the current technologies of robotics, multimedia and computer networks. The paper describes the overall architecture and main system components. We discuss experimental studies where underlying concepts and algorithms are evaluated.

This paper presents a summary of a comprehensive research project involving development of a completely autonomous multirobot system for railway track maintenance applications. The specifications of this "real-world" application required us to consider a multirobot system conceptualization, rather than a single robot system.

A Neural integrated Fuzzy conTroller (NiF-T) which integrates the fuzzy logic representation of human knowledge with the learning capability of neural networks is developed for nonlinear dynamic control problems. NiF-T architecture comprises of three distinct parts: (1) Fuzzy logic Membership Functions (FMF), (2) a Rule Neural Network (RNN), and (3) an Output-Refinement Neural Network (ORNN). FMF are utilized to fuzzify sensory inputs. RNN interpolates the fuzzy rule set; after defuzzification, the output is used to train ORNN. The weights of the ORNN can be adjusted on-line to fine-tune the controller. In this paper, real-time implementations of autonomous mobile robot navigation and multirobot convoying behavior utilizing the NiF-T are presented. Only five rules were used to train the wall following behavior, while nine were used for the hall centering. Also, a robot convoying behavior was realized with only nine rules. For all of the described behaviors-wall following, hall centering, and convoying, their RNN's are trained only for a few hundred iterations and so are their ORNN's trained for only less than one hundred iterations to learn their parent rule sets.

Segmentation is a fundamental process affecting the overall quality and utility of a machine vision system. Range profile tracking (RPT) is a systematic approach for stable, accurate and high speed segmentation of range images that is based on Kalman filtering. Tests of RPT have produced stable decompositions of second order surfaces bounded by jump and crease discontinuities, having a volumetric error of a few percent, in under 6 sec. for a wide variety of conditions. Results from over 900 tests on synthetic scenes and 150 real range images are presented.

This paper presents a wall following algorithm for a mobile robot. The robot uses range information provided by a single rotating ultrasonic transducer. Experimental evaluations have shown that the algorithm is capable of navigating and dealing with a variety of wall structures typically encountered in an indoor environment.

In this paper real-time implementation of multirobot convoying behavior utilizing neuro-fuzzy control is presented. With only nine rules, the follower convoys the leader closely and smoothly by varying its speed and changing its direction. When the leader stops, the follower stops very close to the specified safe distance (within an inch). The follower backs up quickly when the leader moves backward toward it (herding behavior).

This paper describes the development of an integrated autonomous multirobot system, where two heterogeneous robots exhibit coordinated convoying behavior. Coordination is accomplished without any communication but by visual observation. An active perception system uses visual servoing to track the leading vehicle to determine it's heading and relative distance. A novel object tracking algorithm based on minimum noise and correlation energy (MINACE) filter is used for determining the heading of the leading robot. A fuzzy logic based real-time motion controller uses the heading to control the steering and relative distance and speed information to adjust the speed to let the trailing robot smoothly follow the leader. The performance and robustness of the convoying system has been verified in a series of extensive laboratory trials.

A controller with blend of neural networks and fuzzy logic is proposed for a nonlinear control problem. Fuzzy logic membership functions are utilized to fuzzify input parameters; neural network interpolates the fuzzy rule set; after defuzzification, the output is used to train a smaller size of neural network; the weights of the later neural network can be adjusted to fine tune the controller. This controller successfully balances balls with one third of the required 27 rules. With learning capability, it approaches its goal more frequently in general.

We present an integrated methodology that can be used to extract the depth from a sequence of images. The method combines the ability of stereo processing to acquire highly accurate depth measurements and the efficiency of spatial and temporal gradient analysis. As a result of this integration, depth measurements of a high quality are obtained at a speed approximately ten-times higher than that of stereo processing. Without any a priori information of the locations of the points in the scene, the correspondence problem in stereo processing is computationally expensive. In our approach, we use the spatial and temporal gradient analysis, which has been shown to provide the depth with a high efficiency but limited accuracy, to guide the stereo matching process. Experiments for lateral and axial camera translations have acquired depth results with a mean error of less than 3 percent.

A real-time correspondence based tracking algorithm is detailed. The system uses a pipeline processor, a general purpose processor, a camera and a display. The minimum noise and correlation energy (MINACE) filter is used in the tracking algorithm as it provides a good combination of speed, accuracy and flexibility for the targeted hardware system. The system designed is fast and tracking is accomplished at a rate of 15 Hz. The system is adaptive and does not rely on a previous model of the object; the training image for filter synthesis is acquired from previous image frames and the filter is synthesized online to accommodate 3D variations of the target being tracked. The system tracks an object consistently as is demonstrated by the low deviation of the results in the evaluation.

The segmentation of scenes into perceptually meaningful partitions has been a basic problem in image understanding, especially when unsupervised methodology has been desired. A novel unsupervised segmentation approach based on texture is developed. The texture model is based on sets of gray level cooccurence (GLC) matrices rather than measures extracted from them. The algorithmic constituents for the segmentation scheme: choice of seed regions, normalized match distances between texture models, region homogeneity, and aggregation criteria are systematically developed. The unsupervised algorithm works so that “seed” regions are discovered by an image search process. Initial estimates of the texture model prototypes are automatically computed for each “seed” region, and classification thresholds are based on the variance of the model over the “seed” region. An aggregation process then results in regions being successively classified and segmented “out” of the image. This recursive process of segmentation is continued until all pixels are classified. The segmentation strategy was tested successfully on natural texture mosaics. The results are analytically presented. These experiments demonstrate that the unsupervised process can correctly identify the perceptual constituents of the image based on texture.

Presents a Radon transform-based approach to the detection of ship wakes in synthetic aperture radar (SAR) images. The key element of this technique is a localization of the Radon transform, whereby the intensity integration is performed over short line segments rather than across the entire image. A linear feature detection algorithm, which utilizes this localized Radon transform, is then developed. In this algorithm, referred to as the feature space line detector algorithm, the transform space is subjected to processing which serves to isolate and locate the response of linear features and suppresses the response of false alarms. This algorithm is tested on both synthetic images corrupted by various levels of Weibull multiplicative noise and on actual SAR images of ship wakes. The results of this testing demonstrate the algorithm's robustness in the presence of noise, as well as its ability to detect and localize linear features that are significantly shorter than the image dimensions.>

The detection of objects in high-resolution aerial imagery has proven to be a difficult task. In the authors' application, the amount of image clutter is extremely high. Under these conditions, detection based on low-level image cues tends to perform poorly. Neural network techniques have been proposed in object detection applications due to proven robust performance characteristics. A neural network filter was designed and trained to detect targets in thermal infrared images. The feature extraction stage was eliminated and raw gray levels were utilized as input to the network. Two fundamentally different approaches were used to design the training sets. In the first approach, actual image data were utilized for training. In the second case, a model-based approach was adopted to design the training set vectors. The training set consisted of object and background data. The neuron transfer function was modified to improve network convergence and speed and the backpropagation training algorithm was used to train the network. The neural network filter was tested extensively on real image data. Receiver operating characteristic (ROC) curves were determined in each case. The detection and false alarm rates were excellent for the neural network filters. Their overall performance was much superior to that of the size-matched contrast-box filter, especially in the images with higher amounts of visual clutter.

Intelligent robots interact with the real world by employing their advanced sensory mechanisms to perceive their environment, and using their effectors and tools to change the state of their environment. In this paper we introduce a System Architecture for Sensor-based Intelligent Robots (SASIR). The system architecture consists of perception, motor, task planner, knowledge-base, user interface and supervisor modules. SASIR is constructed using a frame data structure, which provides a suitable and flexible scheme for representation and manipulation of the world model, the sensor derived information, as well as for describing the actions required for the execution of a specific task. The experimental results show the basic validity of the general architecture as well as the robust and successful performance of two working systems: 1) the autonomous spin cleaning robotic system, and 2) ROBOSIGHT, which is capable of a range of autonomous inspection and manipulation tasks. Simulation and animation techniques were employed, in addition to the real-world testing, during the system development.>

This paper presents a Radon transform-based approach to the detection of linear features in images characterized by high noise levels. This approach is based on the localized Radon transform where the intensity integration is performed over short line segments rather than across the entire image. The algorithm, referred to as the feature space line detector (FSLD) algorithm, is tested on synthetic images of linear features with very high noise levels. The results of this testing demonstrate the algorithm's robustness in the presence of noise, as well as its ability to detect and localize linear features that are significantly shorter than the image dimensions or that display some curvature.>

Motion information is extracted by identifying the temporal signature associated with the textured objects in the scene. We present a new computational framework for motion perception. Our methodology considers spatiotemporal frequency (STF) domain analysis to extract the optical flow information. First, we show that a sequence of image frames can be used to extract the motion parameters for the different regions in a dynamic scene using the basic Fourier transform properties in the STF analysis approach. A detailed analytical description of this model to interchangably extract motion and depth parameters and results to highlight their salient properties are presented.>

Texture is an important preattentive cue in region-based segmentation of images. In this paper, we discuss the use of circular-Mellin features for segmenting an image into homogenous regions. The circular-Mellin operators represent the spectral decomposition of the image scene in the polar-log coordinate system and are invariant to both scale and orientation of the target. Coupled with the unique shift invariance property of the correlator architecture, these circular-Mellin operators can be used for rotation- and scale-invariant feature extraction. We note that while both these feature extractors have similar functional form to the Gabor functions, the distortion-invariant characteristics of the circular-Mellin operators make them preferable for texture segmentation. Segmentation results to demonstrate their salient properties are presented.>

This paper describes one aspect of a project whose goal is to move a robot in an unknown environment and find pipes to decommission. While moving through the environment a low level map, in the form of an occupancy grid, along with detailed location of pipes in the environment are obtained. This paper deals only with the low level "reflex" of obstacle avoidance that is performed while the robot moves along its path as well as some of the higher level tasks involved in path planning and robot motion in order to negotiate through a hazardous environment. The overall high level interaction with the system is made as simple and user friendly as possible using a graphical interface to control high level tasks. All low level communications and processing are transparent. The robot used for experimentation is a wheeled mobile platform equipped with many different sensors including ultrasonic range sensors and cameras. A quick and efficient obstacle avoidance algorithm has been developed using sixteen ultrasonic range sensor and one infrared proximity sensor.>

In this paper we present a computational approach for extracting three-dimensional structure of controllable resolution, depth of field, and accuracy, all made available at real-time speeds. This approach utilizes the spatial and the temporal gradients of the streams of images acquired using an actively controlled camera. Depending on the requirements of a particular task, appropriate parameters such as disparity value sought, the inter-frame camera displacement, and number of frames in a stream, are chosen to control the resolution, depth of field, and accuracy. The acquisition and processing of the image stream are done in real-time on a pipeline architecture based processor. Extensive experiments are presented to demonstrate the accuracy, controllability of depth of field and resolution, and ability to perform successfully in a variety of scenes. The system operated with no latency between the image acquisition and processing. The total acquisition and processing time in these experiments is in the range of 0.27 to 1.56 sec. The depth results have an accuracy of 85% to 92%.>

A real-time correspondence based tracking algorithm is detailed. The system uses a pipeline processor, a general purpose processor, a camera and a display. The Minimum Noise and Correlation Energy (MINACE) filter is used in the tracking algorithm as it provides a good combination of speed, accuracy and flexibility for the targeted hardware system. The system designed is fast and tracking is accomplished at a rate of 15 hz. The system is adaptive and does not rely on a previous model of the object; the training image for filter synthesis is acquired from previous image frames and the filter is synthesized online to accommodate 3-D variations of the target being tracked. The system tracks an object consistently as is demonstrated by the low deviation of the results in the evaluation. The correlation filter-based tracking algorithm has proved to be useful in our research in cooperative mobile robots. A visual servoing system has been implemented using this tracking algorithm for convoying of multiple mobile robots.>

This paper develops and demonstrates a new computational framework for an accurate, robust, and efficient stereo approach. In multi-primitive hierarchical (MPH) computational model, stereo analysis is performed in multiple stages, incorporating multiple primitives, utilizing a hierarchical control strategy. The MPH stereo system consists of three integrated subsystems: region-based analysis module; linear edge segment-based analysis module; and edgel-based stereo analysis module. Results of stereo analysis at higher levels of the hierarchy are used for guidance at the lower levels. The MPH stereo system does not overly rely on one type of primitive and therefore will reliably work on a wide range of scenes. The MPH stereo analysis results in the generation of several disparity maps of multiple abstraction. Disparity maps generated at each level can be fused to obtain an accurate and fine resolution disparity map. The MPH approach also provides the capability to selectively analyze image regions with varying detail. This provides the means for adaptively extracting range information of only sufficient resolution. Thus, a stereo system that utilizes primitives of different abstraction and a multilevel hierarchical computational strategy will be superior to a single-level, single-primitive system. Extensive experimentation is carried out on a wide array of scenes of varying complexity from two application domains to systematically evaluate the validity and performance of the MPH framework. The MPH stereo system is able to analyze images in most cases with 85%/spl sim/100% matching accuracy in under a minute of processing time and yield depth values typically within /spl plusmn/2% of the actual depth.>

Most simulation and animation systems utilized in robotics are concerned with simulation of the robot and its environment without simulation of sensors. These systems have difficulty in handling robots that utilize sensory feedback in their operation. In this paper, a new design of an environment for simulation, animation, and visualization of sensor-driven robots is presented. As sensor technology advances, increasing numbers of robots are equipped with various types of sophisticated sensors. The main goal of creating the visualization environment is to aid the automatic robot programming and off-line programming capabilities of sensor-driven robots. The software system will help the users visualize the motion and reaction of the sensor-driven robot under their control program. Therefore, the efficiency of the software development is increased, the reliability of the software and the operation safety of the robot are ensured, and the cost of new software development is reduced. Conventional computer-graphics-based robot simulation and animation software packages lack of capabilities for robot sensing simulation. This paper describes a system designed to overcome this deficiency.

This paper describes a reactive obstacle negotiating algorithm for articulated-tracked robots and demonstrates a working system by presenting the simulation, animation, and control methodologies for an articulated-tracked mobile robot with a manipulator and sensory modules. Developing the motion planning strategies for robots with articulated-tracked locomotin is much different from path planning techniques developed for wheeled robots traveling on a planar surface, as the problems are not limited to solving the obstacle avoidance only, but involve finding the obstacle negotiating strategy as well. The authors have been developing tracked mobile-manipulator systems for applications in hazardous environments. A simulation, animation, visualization and interactive control (SAVIC) environment has been developed for the design and operation of a integrated robotic manipulator system.

Describes the status of ongoing work in developing a sensory-motor robotic system, R2H, with 10 degrees of freedoms for research in active vision. To complement the R2H system, a Graphical Simulation and Animation (GSA) environment has also been developed. The objective of building the GSA system is to create a comprehensive design tool to design and study the behavior of active systems and their interactions with the environment. The GSA system will aid the researchers to develop high performance and reliable software and hardware in a most effective manner. The GSA environment integrates sensing and motor actions and features complete kinematic simulation of the R2H system, sensors and its workspace. With the aid of the GSA environment, depth-from-focus, depth-from-vergence, and depth-from-stereo modules have been implemented and tested.

The basic operational scenario for many application domains is to deploy nonhuman system components in the hazardous workspace and keep the human operator away from the hazardous but real work environment. The operator is an essential element of the integrated system and is provided with various powerful displays (for both visual and nonvisual information) for telepresence and interactive controls for teleoperation. Thus, the operator stationed in the safe and virtual work environment, is able to cooperate with the robotic system in accomplishing an assigned task. The team has been investigating various problems associated with the use of virtual-reality based concepts in robot control and system development for the past several years. Some of the main ideas which underlie the investigations are, multiple sensory based operation, transparency between real and virtual mode operation, integrating sensing and task planning in simulation, and reliance on extensive and systematic experimental evaluation. The systems developed are useful for both off-online and online operation.

A Simulation, Animation, Visualization and Interactive Control (SAVIC) environment has been developed for the design and operation of an integrated robotic manipulator system. This unique system possesses the abilities for (1) multi-sensor simulation, (2) kinematics and locomotion animation, (3) dynamic motion and manipulation animation, (4) transformation between real and virtual modes within the same graphics system, (5) ease in exchanging software modules and hardware devices between real and virtual world operations, and (6) interfacing with a real robotic system. This research is focused on enhancing the overall productivity of an integrated human-robot system. This paper describes a working system and illustrates the concepts by presenting the simulation, animation and control methodologies for a unique mobile robot with articulated tracks, a manipulator, and sensory modules.

A computational framework for an accurate, robust, and efficient stereo approach is developed. Most of the deficiencies prevailing in current computational models of stereo can be attributed to their use of a single, typically edge-element-based, primitive for stereo analysis and to their use of a single-level control strategy. The multi-primitive hierarchical (MPH) framework for stereo analysis presented is directed toward overcoming these deficiencies. In the MPH model, stereo analysis is performed in multiple stages, incorporating multiple primitives and utilizing a hierarchical control strategy. The higher levels of the hierarchical system are based on primitives containing more semantic information, and the results of stereo analysis at higher levels are used for guidance at the lower levels. It is shown that such a stereo system is superior to a single-level, single-primitive system.>

The authors describe a simulation environment which can be of utility in the design, development, and operation of sensor-driven robotic systems. This environment incorporates the ability to simulate sensory inputs and their effects on the planning and control functions. A detailed design of the simulator is presented. The simulator is integrated within an entire robotic system, and is utilized for automatic robot programming. Alternatively, the simulated system operation flow can be controlled through either a teach mode or a program mode. Five sensing modalities can be simulated: proximity, point laser range, ultrasonic range, laser range depth imagery, and edge-based intensity imagery. The simulation and animation of a particular robot is described. The unique features of the simulator are that it simulates and utilizes sensory information feedback, it integrates planning and simulation, and it makes the switching between real and simulation mode in the robotic system transparent to the user.>

Intelligent robots interact with the real world by employ- ing their advanced sensory mechanisms to perceive their environment, and their effectors and tools to change the state of their environment. Such intelligent robots should possess three important capabilities: (1) planning, i.e., given a goal, the ability to generate a set of task plans which will lead to achieving the goal, (2) coordination and execution of the perceptual actions, i.e., the abilities to coordinate sensors, acquire sensory data, and process, in- terpret and transform the sensory information, and (3) coordination and execution of the motor actions, i.e., the ability to navigate in their environments and the ability to coordinate effectors and tools and manipulate objects to accomplish assigned tasks. In this paper we introduce a System Architecture for Sensor-based Intelligent Robots, (SASIR), constructed using a frame data structure, which provides a suitable and flexible scheme for representation and manipulation of the world model, the sensor derived information, as well as for describing the actions reqnired for the execution of a specific task.

In this paper we nzake an attempt to bring out the rationale that, for complex robotic applications, contact sensors can provide certain vital information quite easily as compared to non-contact sensors. We begin with describing and discussing our approach towards the development of a multi-sen,sory closed-loop active robotic exploration system for workspace characterization. The exploration system involves acquisition, processing, and integration of sensory inputs acquired from uision, poin,t laser range, tactile, and force/torque sensors. The exploration process results in building three-dimensional Half-space models of various polyhedral (convex and concave) objects encountered in the workspace. Subsequent to the process of building object models using active exploration mechanisms, the object models allow extraction of essential geometric information required for performing tasks such as object identification, grasping, and man ipiila t io n. Th e expe rim en tal resuh sh ow tli at vu rious geometrical descriptors that are required for model building, can be extracted quite reliably using the sensor specific exploratory mechanisms. 1. Introductioii to Multi-sensory Active Exploration Exploration is the process of deriving an environ

Finding the transformation between different rotation representations in the development of high-level software systems that must be transported to different robots utilizing different rotation representation schemes is addressed. Euler angle representations, which are widely used in commercial robots and research laboratory robots, are considered. Typically, there are three different Euler angle representation systems, and each system describes a particular orientation of a rigid body in a reference coordinate frame by specifying three angles. Mathematical derivations of the transformation relationships underlying the parameters from two most commonly utilized representation schemes are presented. These relationships are derived in two different ways. One is based on the solution of the inverse transform for Euler angles and the other is based on the Napier's rules associated with spherical trigonometry.>

A general approach is presented for the integration of vision, range, proximity, and touch sensory data to derive a better estimate of the position and orientation (pose) of an object appearing in the work space. Efficient and robust methods for analyzing vision and range data to derive an interpretation of input images are discussed. Vision information analysis includes a model-based object recognition module and an image-to-world coordinate transformation module to identify the three-dimensional (3-D) coordinates of the recognized objects. The range information processing includes modules for reprocessing, segmentation, and 3-D primitive extraction. The multisensory information integration approach represents sensory information in a sensor-independent form and formulates an optimization problem to find a minimum-error solution to the problem. The capabilities of a multisensor robotic system are demonstrated by performing a number of experiments using an industrial robot equipped with several sensors of differing types.>

The authors describe research efforts directed towards the development of sensor-driven robotic systems for performing various inspection and manipulation tasks. Emphasis is on modules designed for analyzing vision and range sensor data for object recognition and pose estimation. The object recognition system uses a model-based approach. Pose estimation by vision sensor includes derivation of an image-to-world coordinate transformation module. Range data analysis includes modules for preprocessing, segmentation, and 3-D primitive extraction. Experiments are presented to demonstrate the applicability of the analysis modules using a laboratory-based robotic testbed.>

The authors present a curve-fitting method for the recovery of elongated regions in aerial images by analyzing the local image features that constitute these regions. Elongation is defined in terms of an axial model that describes a broad class of objects found in aerial scenes. The authors fit the model using a perceptual strip clustering that maintains constraints on feature connectivity and curvilinear alignment. Connectivity is obtained from internal representations of closest-point properties, specifically, a thresholded version of the Gabriel graph. This is shown to offer the advantages of both the minimum spanning tree and the Delauney triangulation, while being easily computable from finite neighborhood operations. An iterative curve fitting is used for the clustering of maximally elongated partitions, incorporating the perceptual constraints. This has the ability to interpolate across missing elements with a smooth curve, while ignoring nearby complicating structures and systematic gross noise. The general parametric strip model is described, with specific algorithms and examples given for lines and circles. A complete analysis of linear shape in low-level region segmentations is presented, with results for a suburban road network.>

The authors consider the development of a practical binocular stereo approach for extracting depth information. This approach is particularly well suited for robot vision systems designed for inspection and manipulation tasks. The authors emphasize the importance of semantic content and stability of the primitive used in stereo matching and introduce the use of homogeneous regions as features in stereo matching. Considering the lower number of features and the high discrimination power of the primitive, the region-based matcher is more efficient and accurate than those using edge-based primitives. A region-based matching technique can utilize both local and global information and thus yield a more globally consistent solution. However, region-based matching processes typically yield coarse disparity maps. It is noted that it is critical that an efficient and robust stereo system utilize the most appropriate set of primitives at each state of the process. A hierarchical stereo approach that does so is proposed. Several experiments to evaluate the performance of a region-based stereo matcher and a straightforward disparity and depth generation module are described.>

A low-level segmentation methodology based upon fuzzy clustering principles is developed. The approach utilizes region growing concepts and a pyramid data structure for the hierarchical analysis of aerial images. It is assumed that measurement vectors corresponding to perceptually homogeneous regions cluster together in the measurement space. The fuzzy c-means (FCM) clustering algorithm is used in the formulation. Utilization of the fuzzy partitioning allows one to derive a correspondence between the cluster membership function values and (the proportions of) the classes constituting a region. Thus cluster membership values can be used to split mixture regions into smaller regions at a higher resolution level. The feasibility of the methodology is evaluated using a three-channel Landsat image. The results show that the FCM clustering can be used in the single-level segmentation; and that cluster membership function values derived using this algorithm can be utilized effectively as indicators of region homogeneity.