With the advancement of AI-generated content technologies, AI-generated images (AGIs) have become increasingly influential in artistic creation and visual communication. However, the aesthetic quality of AGIs varies significantly due to technical limitations and the influence of user input, underscoring the urgent need for systematic aesthetic evaluation of AGIs. In addition, it is difficult to ensure the consistency of text-to-image, which compresses the application space of AGIs. To address these issues, a fine-grained dataset for Aesthetic and Alignment evaluation of AGIs (AGIAA-2K) is presented. This dataset contains 2,064 images generated using 172 well-designed prompts across six different AGI models, with each image annotated based on the subjective experiment. Then, the rationality of the dataset is verified by data analysis. Finally, the performances of the existing algorithms are evaluated in terms of image aesthetic assessment and text-to-image alignment of AGIs. The results demonstrate that these algorithms cannot effectively evaluate these two aspects. The AGIAA-2K is available at https://github.com/BoHu90/AGIAA-2K.

Machine Learning (ML) has replaced handcrafted methods for perception and prediction in autonomous vehicles. Yet for the equally important planning task, the adoption of ML-based techniques is slow. We present nuPlan, the world’s first real-world autonomous driving dataset and benchmark. The benchmark is designed to test the ability of ML-based planners to handle diverse driving situations and to make safe and efficient decisions. We introduce a new large-scale dataset that consists of 1282 hours of diverse driving scenarios from 4 cities (Las Vegas, Boston, Pittsburgh, and Singapore) and includes high-quality auto-labeled object tracks and traffic light data. We mine and taxonomize common & rare driving scenarios which are used during evaluation to get fine-grained insights into the performance and characteristics of a planner. Beyond the dataset, we provide a simulation and evaluation framework that enables a planner’s actions to be simulated in closed-loop to account for interactions with other traffic participants. We present a detailed analysis of numerous baselines and investigate gaps between ML-based and traditional methods. Find the nuPlan dataset and code at nuplan.org.

Despite recent advancements in depth estimation and face alignment, it remains difficult to predict the distance to a human face in arbitrary videos due to the lack of camera calibration. A typical pipeline is to perform calibration with a checkerboard before the video capture, but this is inconvenient to users or impossible for unknown cameras. This work proposes to use the human face as the calibration object to estimate metric depth information and camera intrinsics. Our novel approach alternates between optimizing the 3D face and the camera intrinsics parameterized by a neural network. Compared to prior work, our method performs camera calibration on a larger variety of videos captured by unknown cameras. Further, due to the face prior, our method is more robust to noise in 2D observations compared to previous self-calibration methods. We show that our method improves calibration and depth prediction accuracy over prior works on both synthetic and real data. Code will be available at https://github.com/yhu9/FaceCalibration.

In this paper, we introduce the first published planner to drive a car in dense, urban traffic using Inverse Reinforcement Learning (IRL). Our planner, DriveIRL, generates a diverse set of trajectory proposals and scores them with a learned model. The best trajectory is tracked by our self-driving vehicle's low-level controller. We train our trajectory scoring model on a 500+ hour real-world dataset of expert driving demonstrations in Las Vegas within the maximum entropy IRL framework. DriveIRL's benefits include: a simple design due to only learning the trajectory scoring function, a flexible and relatively interpretable feature engineering approach, and strong real-world performance. We validated DriveIRL on the Las Vegas Strip and demonstrated fully autonomous driving in heavy traffic, including scenarios involving cut-ins, abrupt braking by the lead vehicle, and hotel pickup/dropoff zones. Our dataset, a part of nuPlan, has been released to the public to help further research in this area.

Traffic light detection is crucial for environment perception and decision-making in autonomous driving. State-of-the-art detectors are built upon deep Convolutional Neural Networks (CNNs) and have exhibited promising performance. However, one looming concern with CNN based detectors is how to thoroughly evaluate the performance of accuracy and robustness before they can be deployed to autonomous vehicles. In this work, we propose a visual analytics system, VATLD, equipped with a disentangled representation learning and semantic adversarial learning, to assess, understand, and improve the accuracy and robustness of traffic light detectors in autonomous driving applications. The disentangled representation learning extracts data semantics to augment human cognition with human-friendly visual summarization, and the semantic adversarial learning efficiently exposes interpretable robustness risks and enables minimal human interaction for actionable insights. We also demonstrate the effectiveness of various performance improvement strategies derived from actionable insights with our visual analytics system, VATLD, and illustrate some practical implications for safety-critical applications in autonomous driving.

Traffic light detection by camera is a challenging task for autonomous driving mainly due to the small size of traffic lights in the road scene especially for early detection. The limited resolution in the corresponding area of traffic lights reduces their contrast to the background, as well as the effectiveness of the visual cues from the traffic light itself. We believe understanding the scene semantics between traffic lights and their surroundings can play a vital role in tackling this challenge. Towards this goal, we build a generative adversarial network (GAN) model to predict the existence of traffic lights from the road scene image where existing traffic lights are removed with image inpainting. Using Cityscape dataset [2], we verify that the proposed GAN model indeed captures the desired semantics by showing effective predictions of existence of traffic lights that are consistent with real images. Moreover, we leverage this model to augment the training data where traffic lights are inserted to the road scene images based on the prediction of the GAN model. While the augmented images may not be realistic looking, results show that such data augmentation can improve the traffic light detector performance that is comparable to using additional real data collection, and better than other data augmentation with various randomization schemes. These results verify the importance of semantic consistency for data augmentation to improve the traffic light detection.

Driving is a highly dynamic activity where drivers' awareness to the traffic environment plays the essential role for successful performance. Easy, smooth driving depends on drivers' awareness to develop situation-specific expectations, where infrequent or unexpected situations are not taken into account. Understanding these unexpected situations provides important insight on driver situation awareness and accident prevention. This study takes advantage of recent development of wearable devices and uses driver physiological signals to identify such unexpected situations during driver hard brake. Based on a naturalistic driving dataset, we define two types of hard brake behavior: reactive and intended hard brake. The reactive hard brake relates to drivers reacting to unexpected situations that usually leads to deviated physiological signals due to stress. The intended hard brake relates to planned maneuver implementation that consists of stable physiological signals. By using the human evaluation, we identified the different situations in which these two types of hard brake occurs. Clear difference is observed between these two groups of situations. Our goal is to identify features that are representative of these two type of road environment, especially the situations where unexpected reactive hard brake happens. Following this direction, we extracted features from Lidar depth scanner to represent the road scene, and applied both lasso regression and logistic regression classifier for feature analysis. The regression model achieves high correlation of 0.77 between the prediction and the ground truth while the classification achieves F-score of 0.76. The selected Lidar features can serve as high level road scene representation that facilitate next generation advanced driver assistance systems (ADAS) to prevent accident in unexpected traffic scenarios.

This study presents the preliminary attempt to investigate the usage of driver physiology signals, including electrocardiography (ECG) and respiration wave signals, to predict driving maneuvers. While most studies on driving maneuver prediction uses direct measurements from vehicle or road scene, we believe the mental state changes from the driver when making plans for maneuver can be reflected from the physiological signals. We extract both time and frequency domain features from the physiological signals, and use them as the features to predict the drivers' future maneuver. We formulate the prediction of driver maneuver as a multi-class classification problem by using the features extracted from signal before the driving maneuvers. The multi classes correspond to various types of driving maneuvers including Start, Stop, Lane Switch and Turn. We use the support vector machine (SVM) as the classifier, and compare the performance of using both physiological and car signals (CAN bus) with the baseline classifier that is trained with only car signal. An improved performance is observed when using the physiological features with 0.04 in F-score on average. This improvement is more obvious as the prediction is made earlier.

Current pedestrian collision warning systems use either auditory alarms or visual symbols to inform drivers. These traditional approaches cannot tell the driver where the detected pedestrians are located, which is critical for the driver to respond appropriately. To address this problem, we introduce a new driver interface taking advantage of a volumetric head-up display (HUD). In our experimental user study, sixteen participants drove a test vehicle in a parking lot while braking for crossing pedestrians using different interface designs on the HUD. Our results showed that spatial information provided by conformal graphics on the HUD resulted in not only better driver performance but also smoother braking behavior as compared to the baseline.

This study explores the feasibility of detecting drivers' mirror-checking actions using noninvasive sensors. Checking the mirrors is an important primary driving action that allows drivers to maintain their situational awareness, particularly when they are planning to turn or change lanes. Recognizing when drivers are checking the mirrors can facilitate the detection of hazard scenarios by considering contextual information (e.g., turning without checking mirrors, lack of mirror-checking actions signaling cognitive distractions, or distinction between gazes due to primary or secondary tasks). This study analyzes drivers' mirror-checking actions under various real driving conditions. We analyze the drivers' mirror-checking actions under normal conditions, as well as when the drivers are engaged in secondary tasks such as tuning the radio or operating a cell phone. We also compare mirror-checking behaviors observed during different maneuver actions: driving straight, turning, and switching lanes. This study reveals statistically significant differences in mirror-checking actions among most of the comparisons. The results suggest that mirror-checking actions can be useful indicators in recognizing drivers engaged in secondary tasks, as well as in detecting driving maneuvers. We propose to detect mirror-checking actions using features extracted from multiple noninvasive sensors (CAN-Bus and cameras facing the driver and the road). We consider three machine learning algorithms for unbalanced data sets, achieving an F-score of 91%. The recognized mirror-checking actions are used as additional features to improve the performance of secondary task detection and maneuver recognition. These promising results suggest that it is possible to detect mirror-checking actions, providing contextual information to improve new driver monitoring systems.

A driver's behaviors can be affected by visual, cognitive, auditory, and manual distractions. While it is important to identify the patterns associated with particular secondary tasks, it is more general and useful to define distraction modes that capture the general behaviors induced by various sources of distractions. By explicitly modeling the distinction between types of distractions, we can assess the detrimental effects induced by new in-vehicle technology. This study investigates drivers' behaviors associated with visual and cognitive distractions, both separately and jointly. External observers assessed the perceived cognitive and visual distractions from real-world driving recordings, showing high interevaluator agreement in both dimensions. The scores from the perceptual evaluation are used to define regression models with elastic net regularization and binary classifiers to separately estimate the cognitive and visual distraction levels. The analysis reveals multimodal features that are discriminative of cognitive and visual distractions. Furthermore, the study proposes a novel joint visual-cognitive distraction space to characterize driver behaviors. A data-driven clustering approach identifies four distraction modes that provide insights to better understand the deviation in driving behaviors induced by secondary tasks. Binary and multiclass recognition problems demonstrate the effectiveness of the proposed multimodal features to infer these distraction modes defined in the visual-cognitive space.

An ideal gaze user interface should be able to accurately estimates the user's gaze direction in a non-intrusive setting. Most studies on gaze estimation focus on the accuracy of the estimation results, imposing important constraints on the user such as no head movement, intrusive head mount setting and repetitive calibration process. Due to these limitations, most graphic user interfaces (GUIs) are reluctant to include gaze as an input modality. We envision user-independent gaze detectors for user computer interaction that do not impose any constraints on the users. We believe the appearance of the eye pairs, which implicitly reveals head pose, provides conclusive information on the gaze direction. More importantly, the relative appearance changes in the eye pairs due to the different gaze direction should be consistent among different human subjects. We collected a multimodal corpus (MSP-GAZE) to study and evaluate user independent, appearance based gaze estimation approaches. This corpus considers important factors that affect the appearance based gaze estimation: the individual difference, the head movement, and the distance between the user and the interface's screen. Using this database, our initial study focused on the eye pair appearance eigenspace approach, where the projections into the eye appearance eigenspace basis are used to build regression models to estimate the gaze position. We compare the results between user dependent (training and testing on the same subject) and user independent (testing subject is not included in the training data) models. As expected, due to the individual differences between subjects, the performance decreases when the models are trained without data from the target user. The study aims to reduce the gap between user dependent and user independent conditions.

The design of gaze-based computer interfaces has been an active research area for over 40 years. One challenge of using gaze detectors is the repetitive calibration process required to adjust the parameters of the systems, and the constrained conditions imposed on the user for robust gaze estimation. We envision user-independent gaze detectors that do not require calibration, or any cooperation from the user. Toward this goal, we investigate an appearance-based approach, where we estimate the eigenspace for the gaze using principal component analysis (PCA). The projections are used as features of regression models that estimate the screen's coordinates. As expected, the performance of the approach decreases when the models are trained without data from the target user (i.e., user-independent condition). This study proposes an appealing training approach to bridge the gap in performance between user-dependent and user-independent conditions. Using the projections onto the eigenspace, the scheme identifies samples in training set that are similar to the testing images. We build the sample covariance matrix and the regression models only with these samples. We consider either similar frames or data from subjects with similar eye appearance. The promising results suggest that the proposed training approach is a feasible and convenient scheme for gaze-based multimodal interfaces.

Drivers are exposed to a growing risk of being distracted with the recent development of in-vehicle systems for navigation, communication and infotainment. As a result, there is a need for tracking systems that can monitor the drivers' attention. This study investigates driver distractions using a multimodal corpus collected from real world driving scenarios. The paper focuses on facial cues automatically extracted from a frontal camera facing the driver. We conducted subjective evaluations by external observers to assess the perceived visual and cognitive distraction of drivers performing secondary tasks. The data is divided into two classes - distracted and normal. This partition is separately created for visual and cognitive scores. Binary classifiers are built with features describing action units (AU) and gaze (e.g., head poses). The classifiers achieve 80.8% F-score for visual distractions, and 73.8% F-score for cognitive distractions. The study identifies features that are relevant for detecting both types of distractions. Furthermore, the paper presents a logistic regression analysis to identify facial features that are useful for detecting samples in which cognitive distraction scores are not related to visual distraction scores. The analysis reveals the benefits of using AU in cognitive related distraction detection.

Given the crucial role of eye movements on visual attention, tracking gaze behaviors is an important research problem in various applications including biometric identification, attention modeling and human-computer interaction. Most of the existing gaze tracking methods require a repetitive system calibration process and are sensitive to the user's head movements. Therefore, they cannot be easily implemented in current multimodal interfaces. This paper investigates an appearance-based approach for gaze estimation that requires minimum calibration and is robust against head motion. The approach consists in building an orthonormal basis, or eigenspace, of the eye appearance with principal component analysis (PCA). Unlike previous studies, we build the eigenspace using image patches displaying both eyes. The projections into the basis are used to train regression models which predict the gaze location. The approach is trained and tested with a new multimodal corpus introduced in this paper. We consider several variables such as the distance between user and the computer monitor, and head movement. The evaluation includes the performance of the proposed gaze estimation system with and without head movement. It also evaluates the results in subject-dependent versus subject-independent conditions under different distances. We report promising results which suggest that the proposed gaze estimation approach is a feasible and flexible scheme to facilitate gaze-based multimodal interfaces.

With the development of new in-vehicle technology, drivers are exposed to more sources of distraction, which can lead to an unintentional accident. Monitoring the driver attention level has become a relevant research problem. This is the precise aim of this study. A database containing 20 drivers was collected in real-driving scenarios. The drivers were asked to perform common secondary tasks such as operating the radio, phone and a navigation system. The collected database comprises of various noninvasive sensors including the controller area network-bus (CAN-Bus), video cameras and microphone arrays. The study analyzes the effects in driver behaviors induced by secondary tasks. The corpus is analyzed to identify multimodal features that can be used to discriminate between normal and task driving conditions. Separate binary classifiers are trained to distinguish between normal and each of the secondary tasks, achieving an average accuracy of 77.2%. When a joint, multi-class classifier is trained, the system achieved accuracies of 40.8%, which is significantly higher than chances (12.5%). We observed that the classifiers' accuracy varies across secondary tasks, suggesting that certain tasks are more distracting than others. Motivated by these results, the study builds statistical models in the form of Gaussian Mixture Models (GMMs) to quantify the actual deviations in driver behaviors from the expected normal driving patterns. The study includes task independent and task dependent models. Building upon these results, a regression model is proposed to obtain a metric that characterizes the attention level of the driver. This metric can be used to signal alarms, preventing collision and improving the overall driving experience.