Addressing hard cases in autonomous driving, such as anomalous road users, extreme weather conditions, and complex traffic interactions, presents significant challenges. To ensure safety, it is crucial to detect and manage these scenarios effectively for autonomous driving systems. However, the rarity and high-risk nature of these cases demand extensive, diverse datasets for training robust models. Vision-Language Foundation Models (VLMs) have shown remarkable zero-shot capabilities as being trained on extensive datasets. This work explores the potential of VLMs in detecting hard cases in autonomous driving. We demonstrate the capability of VLMs such as GPT-4v in detecting hard cases in traffic participant motion prediction on both agent and scenario levels. We introduce a feasible pipeline where VLMs, fed with sequential image frames with designed prompts, effectively identify challenging agents or scenarios, which are verified by existing prediction models. Moreover, by taking advantage of this detection of hard cases by VLMs, we further improve the training efficiency of the existing motion prediction pipeline by performing data selection for the training samples suggested by GPT. We show the effectiveness and feasibility of our pipeline incorporating VLMs with state-of-the-art methods on NuScenes datasets. The code is accessible at https://github.com/KTH-RPL/Detect_VLM.

3D object detection at long-range is crucial for ensuring the safety and efficiency of self-driving vehicles, allowing them to accurately perceive and react to objects, obstacles, and potential hazards from a distance. But most current state-of-the-art LiDAR based methods are range limited due to sparsity at long-range, which generates a form of domain gap between points closer to and farther away from the ego vehicle. Another related problem is the label imbalance for faraway objects, which inhibits the performance of Deep Neural Networks at long-range. To address the above limitations, we investigate two ways to improve long-range performance of current LiDAR-based 3D detectors. First, we combine two 3D detection networks, referred to as range experts, one specializing at near to mid-range objects, and one at long-range 3D detection. To train a detector at long-range under a scarce label regime, we further weigh the loss according to the labelled point’s distance from ego vehicle. Second, we augment LiDAR scans with virtual points generated using Multimodal Virtual Points (MVP), a readily available image-based depth completion algorithm. Our experiments on the long-range Argoverse2 (AV2) dataset indicate that MVP is more effective in improving long range performance, while maintaining a straightforward implementation. On the other hand, the range experts offer a computationally efficient and simpler alternative, avoiding dependency on image-based segmentation networks and perfect camera-LiDAR calibration.

This paper presents image-based quantitative evaluation of subtle variations in facial wrinkles for the same subject in response to a dermatological treatment. This is a novel application because the time series images of the same subject over a shorter time period of weeks are analyzed as compared to more prevalent inter-person analysis of facial skin/marks. We propose image features based on Gabor filter bank for an accurate quantitative evaluation of variations in facial wrinkles. Since variations in Gabor features are very small on a time period of weeks, we propose a framework to compare image features in key wrinkle sites only while excluding the noise introduced by non-wrinkle sites. The framework consists of finer registration of images using Large Deformation Diffeomorphic Metric Mapping (LDDMM) and detection of wrinkle sites using Gabor filter bank and morphological image processing. Preliminary experiments show that the framework is useful in calculating variations in Gabor features at detected sites and indicating trends in the response of facial wrinkles to the dermatological treatment.

Facial retouching is widely used in media and entertainment industry. Professional software usually require a minimum level of user expertise to achieve the desirable results. In this paper, we present an algorithm to detect facial wrinkles/imperfection. We believe that any such algorithm would be amenable to facial retouching applications. The detection of wrinkles/imperfections can allow these skin features to be processed differently than the surrounding skin without much user interaction. For detection, Gabor filter responses along with texture orientation field are used as image features. A bimodal Gaussian mixture model (GMM) represents distributions of Gabor features of normal skin versus skin imperfections. Then, a Markov random field model is used to incorporate the spatial relationships among neighboring pixels for their GMM distributions and texture orientations. An expectation-maximization algorithm then classifies skin versus skin wrinkles/imperfections. Once detected automatically, wrinkles/imperfections are removed completely instead of being blended or blurred. We propose an exemplar-based constrained texture synthesis algorithm to inpaint irregularly shaped gaps left by the removal of detected wrinkles/imperfections. We present results conducted on images downloaded from the Internet to show the efficacy of our algorithms.

In this paper, we present a new generative model for wrinkles on aging human faces based on Markov Point Processes (MPP) where wrinkles are considered as stochastic spatial arrangements of sequences of line segments. The model is then used in a Bayesian framework to localize the wrinkles in images. In aging human faces, wrinkles mostly appear as discontinuities in surrounding grayscale texture. The intensity gradients due to wrinkles are enhanced using filters and used as data to detect more probable locations and directions of line segments. Wrinkles are localized by sampling MPP using the Reversible Jump Markov Chain Monte Carlo (RJMCMC) algorithm. Experiments on images obtained from uncontrolled acquisition conditions are presented.