Explainability is essential for autonomous vehicles and other robotics systems interacting with humans and other objects during operation. Humans need to understand and anticipate the actions taken by machines for trustful and safe cooperation. In this work, we aim to develop an explainable model that generates explanations consistent with both human domain knowledge and the model’s inherent causal relation. In particular, we focus on an essential building block of autonomous driving—multi-agent interaction modeling. We propose Grounded Relational Inference (GRI). It models an interactive system’s underlying dynamics by inferring an interaction graph representing the agents’ relations. We ensure a semantically meaningful interaction graph by grounding the relational latent space into semantic interactive behaviors defined with expert domain knowledge. We demonstrate that it can model interactive traffic scenarios under both simulation and real-world settings and generate semantic graphs explaining the vehicle’s behavior through their interactions.

Digitization of newspapers is of interest for many reasons including preservation of history, accessibility and search ability, etc. While digitization of documents such as scientific articles and magazines is prevalent in literature, one of the main challenges for digitization of newspaper lies in its complex layout (e.g. articles spanning multiple columns, text interrupted by images) analysis, which is necessary to preserve human read-order. This work provides a major breakthrough in the digitization of newspapers on three fronts: first, releasing a dataset of 3000 fully-annotated, real-world newspaper images from 21 different U.S. states representing an extensive variety of complex layouts for document layout analysis; second, proposing layout segmentation as a precursor to existing optical character recognition (OCR) engines, where multiple state-of-the-art image segmentation models and several post-processing methods are explored for document layout segmentation; third, providing a thorough and structured evaluation protocol for isolated layout segmentation and end-to-end OCR.

Situation awareness (SA) is generally considered as the perception, understanding, and projection of objects’ properties and positions. We believe if the system can sense drivers’ SA, it can appropriately provide warnings for objects that drivers are not aware of. To investigate drivers’ awareness, in this study, a human-subject experiment of driving simulation was conducted for data collection. While a previous predictive model for drivers’ situation awareness utilized drivers’ gaze movement only, this work utilizes object properties, characteristics of human visual sensory and memory mechanism. As a result, the proposed driver SA prediction model achieves over 70% accuracy and outperforms the baselines.

A vehicle driving along the road is surrounded by many objects, but only a small subset of them influence the driver's decisions and actions. Learning to estimate the importance of each object on the driver's real-time decision-making may help better understand human driving behavior and lead to more reliable autonomous driving systems. Solving this problem requires models that understand the interactions between the ego-vehicle and the surrounding objects. However, interactions among other objects in the scene can potentially also be very helpful, e.g., a pedestrian beginning to cross the road between the ego-vehicle and the car in front will make the car in front less important. We propose a novel framework for object importance estimation using an interaction graph, in which the features of each object node are updated by interacting with others through graph convolution. Experiments show that our model outperforms state-of-the-art baselines with much less input and pre-processing.

Eye-tracking techniques have the potential for estimating driver awareness of road hazards. However, traditional eye-movement measures based on static areas of interest may not capture the unique characteristics of driver eyeglance behavior and challenge the real-time application of the technology on the road. This article proposes a novel method to operationalize driver eye-movement data analysis based on moving objects of interest. A human-subject experiment conducted in a driving simulator demonstrated the potential of the proposed method. Correlation and regression analyses between indirect (i.e., eye-tracking) and direct measures of driver awareness identified some promising variables that feature both spatial and temporal aspects of driver eye-glance behavior relative to objects of interest. Results also suggest that eye-glance behavior might be a promising but insufficient predictor of driver awareness. This work is a preliminary step toward real-time, on-road estimation of driver awareness of road hazards. The proposed method could be further combined with computer-vision techniques such as object recognition to fully automate eye-movement data processing as well as machine learning approaches to improve the accuracy of driver awareness estimation.

We formulate a new problem as Object Importance Estimation (OIE) in on-road driving videos, where the road users are considered as important objects if they have influence on the control decision of the ego-vehicle's driver. The importance of a road user depends on both its visual dynamics, e.g., appearance, motion and location, in the driving scene and the driving goal, e.g., the planned path, of the ego vehicle. We propose a novel framework that incorporates both visual model and goal representation to conduct OIE. To evaluate our framework, we collect an on-road driving dataset at traffic intersections in the real world and conduct human-labeled annotation of the important objects. Experimental results show that our goal-oriented method outperforms baselines and has much more improvement on the left-turn and right-turn scenarios. Furthermore, we explore the possibility of using object importance for driving control prediction and demonstrate that binary brake prediction can be improved with the information of object importance.

Road-users are a critical part of decision-making for both self-driving cars and driver assistance systems. Some road-users, however, are more important for decision-making than others because of their respective intentions, ego-vehicle's intention and their effects on each other. In this paper, we propose a novel architecture for road-user importance estimation which takes advantage of the local and global context of the scene. For local context, the model exploits the appearance of the road users (which captures orientation, intention, etc.) and their location relative to ego-vehicle. The global context in our model is defined based on the feature map of the convolutional layer of the module which predicts the future path of the ego-vehicle and contains rich global information of the scene (e.g., infrastructure, road lanes, etc.), as well as the ego-vehicle's intention information. Moreover, this paper introduces a new data set of real-world driving, concentrated around intersections and includes annotations of important road users. Systematic evaluations of our proposed method against several baselines show promising results.

Determining object of fixation is an important factor in many application of intelligent vehicles including driver's situational awareness estimation. The objective of this work is to infer object of fixation given fixation is occurring. We propose a system architecture that identifies object tracks in the scene, derives object characteristics independent of and jointly with gaze behavior, and utilizes a spatio-temporal sensitive machine learning framework to estimate the likelihood of an object being the object of fixation. Performance evaluation is conducted on a dataset of on-road driving, centered around urban intersections, with manual annotations of object of fixation. Our proposed system can achieve up to 83% average precision accuracy when compared to baseline of 78%.Furthermore, comparing the effects of different combinations of object characteristics on precision and recall accuracy show promising insights on factors affecting reliable estimation of object of fixation.

From driver assistance in manual mode to takeover requests in highly automated mode, knowing the state of driver (e.g. sleeping, distracted, attentive) is critical for safe, comfortable and stress-free driving. Since driving is a visually demanding task, driver's gaze is especially important in estimating the state of driver; it has the potential to derive what the driver has attended to or is attending to and predict future actions. We developed a machine vision based framework to model driver's behavior by representing the gave dynamics over a time period using gaze fixations and transition frequencies. As a use case, we explore the driver's gaze patterns during maneuvers executed in freeway driving, namely, left lane change maneuver, right lane change maneuver and lane keep. It is shown that mapping gaze dynamics to gaze fixations and transition frequencies leads to recurring patterns based on driver activities. Furthermore, using data from on-road driving, we show that modeling these patterns show predictive powers in on-road driving maneuver detection around a few hundred milliseconds a priori.

Drivers use some combination of head, eye and hand movements to perform varying number of tasks from driving related to non-driving secondary tasks. Furthermore, the combinations may vary depending on the task performed. It is important to model and understand these variations in order to build predictive systems, explore driving styles, detect activities, etc. This study, therefore, introduces a framework to model the spatio-temporal movements of head, eyes and hands given naturalistic driving data of looking-in at the driver for any events or tasks performed of interest. As a use case, we explore the temporal coordination of the modalities on data of drivers executing maneuvers at stop-controlled intersections; the maneuvers executed are go straight, turn left and turn right. In sequentially increasing time windows, by training classifiers which have the ability to provide discriminative quality of its input variable, the experimental study at intersections shows which type of, when and how long distinguishable preparatory movements occur in the range of a few milliseconds to a few seconds.

Face detection is a vital step in the process of extracting semantic information about the driver's state, such as distraction and fatigue, from pixel values in images looking at the driver. Therefore, in the context of time and safety critical situation like driving, efficient use of time and reliable detection of faces is essential. While challenges like lighting and occlusion are prevalent in the vehicle cockpit and disruptive for time and reliabilities sake, the automobile cabin has a unique and advantageous environment for face detection. In this study we introduce a deep CNN based face detection method with discrete head pose estimation which address key challenges such as lighting conditions, occlusions, varying view points. One of the vital points in training the CNN based system is the compilation of positive samples via real-world dataset and synthetic data augmentation useful for in-vehicular settings. Performance evaluation on publicly available naturalistic driving data set, called VIVA-Face Dataset, shows promising results compared to baseline methods.

In this paper, we study the complex coordination of head, eyes and hands as the driver approaches a stop-controlled intersection. The proposed framework is made up of three major parts. The first part is the naturalistic driving dataset collection: synchronized capture of sensors looking-in and looking-out, multiple drivers driving in urban environment, and segmenting events at stop-controlled intersections. The second part is extracting reliable features from purely vision sensors looking in at the driver: eye movements, head pose and hand location respective to the wheel. The third part is in the design of appropriate temporal features for capturing coordination. A random forest algorithm is employed for studying relevance and understanding the temporal evolution of head, eye, and hand cues. Using 24 different events (from 5 drivers resulting in ~ 12200 frames analyzed) of three different maneuvers at stop-controlled intersections, we found that preparatory motions range in the order of a few seconds to a few milliseconds, depending on the modality (i.e. eyes, head, hand), before the event occurs.

Intelligent vehicles of the future are that which, having a holistic (i.e. inside and outside the vehicle) perception and understanding of the driving environment, make it possible for passengers to go from point A to point B safely and in a timely manner. This may happen by way of providing active assistance for drivers, giving full control to automated cars or some combination of the two. No matter how, a holistic perception and understanding of inside and outside the vehicle is absolutely necessary, and vision based techniques are expected to play an increasing role in this holistic view. The question is, how well do these vision techniques work in order to be used in time and safety critical driving situations? We introduce one part of the Vision for Intelligent Vehicles and Applications (VIVA), the face challenge. VIVA is a platform designed to share naturalistic driving data with the community in order to: present issues and challenges in vision from real-world driving conditions, benchmark existing vision approaches using proper metrics and progress the development of future vision algorithms. With a special focus on challenges from looking inside at the driver's face, this articles provides information on how the data is acquired and annotated, and how methods are compared.

Naturalistic driving dataset is at the heart of automotive user interface research, detecting/measuring driver distraction, and many other driver safety related studies. Recent advances in the collection of large scale naturalistic driving data include the second Strategic Highway Research Program (SHRP2) consisting of more than 3000 subjects and the 100-Car study. Public access to such data, however, is made difficult due to personal identifiable information and protection of privacy. We propose de-identification filters for protecting the privacy of drivers while preserving sufficient details to infer driver behavior, such as the gaze direction, in naturalistic driving videos. Driver's gaze estimation is of particular interest because it is a good indicator of driver's visual attention and a good predictor of driver's intent. We implement and compare de-identification filters, which are made up of a combination of preserving eye regions, superimposing head pose encoded face mask and replacing background with black pixels, and show promising results.

In this paper, a multiview, multimodal vision framework is proposed in order to characterize driver activity based on head, eye, and hand cues. Leveraging the three types of cues allows for a richer description of the driver's state and for improved activity detection performance. First, regions of interest are extracted from two videos, one observing the driver's hands and one the driver's head. Next, hand location hypotheses are generated and integrated with a head pose and facial landmark module in order to classify driver activity into three states: wheel region interaction with two hands on the wheel, gear region activity, or instrument cluster region activity. The method is evaluated on a video dataset captured in on-road settings.

In this work, we propose a vision-based analysis framework for recognizing in-vehicle activities such as interactions with the steering wheel, the instrument cluster and the gear. The framework leverages two views for activity analysis, a camera looking at the driver's hand and another looking at the driver's head. The techniques proposed can be used by researchers in order to extract ‘mid-level’ information from video, which is information that represents some semantic understanding of the scene but may still require an expert in order to distinguish difficult cases or leverage the cues to perform drive analysis. Unlike such information, ‘low-level’ video is large in quantity and can't be used unless processed entirely by an expert. This work can apply to minimizing manual labor so that researchers may better benefit from the accessibility of the data and provide them with the ability to perform larger-scaled studies.

In this work, we propose a framework for the recognition and prediction of driver maneuvers by considering holistic cues. With an array of sensors, driver's head, hand, and foot gestures are being captured in a synchronized manner together with lane, surrounding agents, and vehicle parameters. An emphasis is put on real-time algorithms. The cues are processed and fused using a latent-dynamic discriminative framework. As a case study, driver activity recognition and prediction in overtaking situations is performed using a naturalistic, on-road dataset. A consequence of this work would be in development of more effective driver analysis and assistance systems.

A common pool of naturalistic driving data is necessary to develop and compare algorithms that infer driver behavior, in order to improve driving safety. Naturalistic driving data, such as video sequences of looking at a driver, however, cause concern for the privacy of individual drivers. In an ideal situation, a deidentification filter applied to a raw image of looking at a driver would, semantically, protect the identity and preserve the behavior (e.g, eye gaze, head pose, and hand activity) of the driver. Driver gaze estimation is of particular interest because it is a good indicator of a driver's visual attention and a good predictor of a driver's intent. Interestingly, the same facial features that are explicitly or implicitly used for gaze estimation play a key role in recognizing a person's identity. In this paper, we implement a specific deidentification filter on video sequences of looking at a driver from naturalistic driving and present novel findings on its effect on face recognition and driver gaze-zone estimation.

Analysis of a driver's head behavior is an integral part of a driver monitoring system. In particular, the head pose and dynamics are strong indicators of a driver's focus of attention. Many existing state-of-the-art head dynamic analyzers are, however, limited to single-camera perspectives, which are susceptible to occlusion of facial features from spatially large head movements away from the frontal pose. Nonfrontal glances away from the road ahead, however, are of special interest since interesting events, which are critical to driver safety, occur during those times. In this paper, we present a distributed camera framework for head movement analysis, with emphasis on the ability to robustly and continuously operate even during large head movements. The proposed system tracks facial features and analyzes their geometric configuration to estimate the head pose using a 3-D model. We present two such solutions that additionally exploit the constraints that are present in a driving context and video data to improve tracking accuracy and computation time. Furthermore, we conduct a thorough comparative study with different camera configurations. For experimental evaluations, we collected a novel head pose data set from naturalistic on-road driving in urban streets and freeways, with particular emphasis on events inducing spatially large head movements (e.g., merge and lane change). Our analyses show promising results.

Currently there are many research focused on using smartphone as a data collection device. Many have shown its sensors ability to replace a lab test bed. These inertial sensors can be used to segment and classify driving events fairly accurately. In this research we explore the possibility of using the vehicle's inertial sensors from the CAN bus to build a profile of the driver to ultimately provide proper feedback to reduce the number of dangerous car maneuver. Braking and turning events are better at characterizing an individual compared to acceleration events. Histogramming the time-series values of the sensor data does not help performance. Furthermore, combining turning and braking events helps better differentiate between two similar drivers when using supervised learning techniques compared to separate events alone, albeit with anemic performance.

An important goal in automotive user interface research is to predict a user's reactions and behaviors in a driving environment. The behavior of both drivers and passengers can be studied by analyzing eye gaze, head, hand, and foot movement, upper body posture, etc. In this paper, we focus on estimating head pose, which has been shown to be a good predictor of driver intent and a good proxy for gaze estimation, and provide a valuable head pose database for future comparative studies. Most existing head pose estimation algorithms are still struggling under large spatial head turns. Our method, however, relies on using facial features that are visible even during large spatial head turns to estimate head pose. The method is evaluated on the LISA-P Head Pose database, which has head pose data from on-road daytime and nighttime drivers of varying age, race, and gender; ground truth for head pose is provided using a motion capture system. In special regards to eye gaze estimation for automotive user interface study, the automatic head pose estimation technique presented in this paper can replace previous eye gaze estimation methods that rely on manual data annotation or be used in conjunction with them when necessary.