As a direct depth sensor, radar holds promise as a tool to improve monocular 3D object detection, which suffers from depth errors, due in part to the depth-scale ambiguity. On the other hand, leveraging radar depths is hampered by difficulties in precisely associating radar returns with 3D estimates from monocular methods, effectively erasing its benefits. This paper proposes a fusion network that addresses this radar-camera association challenge. We train our network to predict the 3D offsets between radar returns and object centers, enabling radar depths to enhance the accuracy of 3D monocular detection. By using parallel radar and camera backbones, our network fuses information at both the feature level and detection level, while at the same time leveraging a state-of-the-art monocular detection technique without retraining it. Experimental results show significant improvement in mean average precision and translation error on the nuScenes dataset over monocular counterparts. Our source code is available at https://github.com/longyunf/radiant.

While radar and video data can be readily fused at the detection level, fusing them at the pixel level is potentially more beneficial. This is also more challenging in part due to the sparsity of radar, but also because automotive radar beams are much wider than a typical pixel combined with a large baseline between camera and radar, which results in poor association between radar pixels and color pixel. A consequence is that depth completion methods designed for LiDAR and video fare poorly for radar and video. Here we propose a radar-to-pixel association stage which learns a mapping from radar returns to pixels. This mapping also serves to densify radar returns. Using this as a first stage, followed by a more traditional depth completion method, we are able to achieve image-guided depth completion with radar and video. We demonstrate performance superior to camera and radar alone on the nuScenes dataset. Our source code is available at https://github.com/longyunf/rc-pda.

A distinctive feature of Doppler radar is the measurement of velocity in the radial direction for radar points. However, the missing tangential velocity component hampers object velocity estimation as well as temporal integration of radar sweeps in dynamic scenes. Recognizing that fusing camera with radar provides complementary information to radar, in this paper we present a closed-form solution for the point-wise, full-velocity estimate of Doppler returns using the corresponding optical flow from camera images. Additionally, we address the association problem between radar returns and camera images with a neural network that is trained to estimate radar-camera correspondences. Experimental results on the nuScenes dataset verify the validity of the method and show significant improvements over the state-of-the-art in velocity estimation and accumulation of radar points.

This article describes a novel approach to the problem of developing explainable machine learning models. We consider a deep reinforcement learning (DRL) model representing a highway path planning policy for autonomous highway driving [1]. The model constitutes a mapping from the continuous multidimensional state space characterizing vehicle positions and velocities to a discrete set of actions in longitudinal and lateral direction. It is obtained by applying a customized version of the double deep Q-network learning algorithm [2]. The main idea is to approximate the DRL model with a set of IF-THEN rules that provide an alternative interpretable model, which is further enhanced by visualizing the rules. This concept is rationalized by the universal approximation properties of the rule-based models with fuzzy predicates. The proposed approach includes a learning engine composed of zero-order fuzzy rules, which generalize locally around the prototypes by using multivariate function models. The adjacent (in the data space) prototypes, which correspond to the same action, are further grouped and merged into the so-called MegaClouds reducing significantly the number of fuzzy rules. The input selection method is based on ranking the density of the individual inputs. Experimental results show that the specific DRL agent can be interpreted by approximating with families of rules of different granularity. The method is computationally efficient and can be potentially extended to addressing the explainability of the broader set of fully connected deep neural network models.

This paper describes a new self-organizing neuro-fuzzy approach to autonomously learn interpretable models by self-driving cars. A new explainable self-organizing architecture and a new density-based feature selection method are proposed. These new approaches are used to classify different action states occurring from different self-driving conditions. The proposed approach is able to provide human understandable IF ... THEN rules representation due to its learning engine which is composed of a massively parallel set of 0-order fuzzy rules. The proposed density-based feature selection method is based on the ranking of the densities of each feature in the data space, and takes advantage of the parallel characteristic of the proposed explainable self-organizing approach to create individualized subsets of features per class. The main goal of both proposed methods is to provide highly accurate models with high transparency, interpretability, and explainability for self-driving vehicles. In order to validate our proposal, experiments were realized using a real dataset provided by Ford Motor Company. The dataset contains different driving states occurring during self-driving performances. Results demonstrate that the proposed approach could surpass its state-of-the-art competitors in terms of accuracy for this challenge multiclass classification problem.

This paper presents an actively semi-supervised multi-layer neuro-fuzzy modeling method, ASSDRB, to classify different lighting conditions for driving scenes. ASSDRB is composed of a massively parallel ensemble of AnYa type 0-order fuzzy rules. It uses a recursive learning algorithm to update its structure when new data items are provided and, therefore, is able to cope with nonstationarities. Different lighting conditions for driving situations are considered in the analysis, which is used by self-driving cars as a safety mechanism. Differently from mainstream Deep Neural Networks approaches, the ASSDRB is able to learn from unseen data. Experiments on different lighting conditions for driving scenes, demonstrated that the deep neuro-fuzzy modeling is an efficient framework for these challenging classification tasks. Classification accuracy is higher than those produced by alternative machine learning methods. The number of algebraic calculations for the present method are significantly smaller and, therefore, the method is significantly faster than common Deep Neural Networks approaches. Moreover, DRB produced transparent AnYa fuzzy rules, which are human interpretable.