Autonomous racing has rapidly gained research attention. Traditionally, racing cars rely on 2D LiDAR as their primary visual system. In this work, we explore the integration of an event camera with the existing system to provide enhanced temporal information. Our goal is to fuse the 2D LiDAR data with event data in an end-to-end learning framework for steering prediction, which is crucial for autonomous racing. To the best of our knowledge, this is the first study addressing this challenging research topic. We start by creating a multisensor dataset specifically for steering prediction. Using this dataset, we establish a benchmark by evaluating various SOTA fusion methods. Our observations reveal that existing methods often incur substantial computational costs. To address this, we apply low-rank techniques to propose a novel, efficient, and effective fusion design. We introduce a new fusion learning policy to guide the fusion process, enhancing robustness against misalignment. Our fusion architecture provides better steering prediction than LiDAR alone, significantly reducing the RMSE from 7.72 to 1.28. Compared to the second-best fusion method, our work represents only 11% of the learnable parameters while achieving better accuracy. The source code and dataset are publicly available at: https://github.com/ZZY-Zhou/F1Tenth-Steering.

Moving object segmentation (MOS) in dynamic scenes is an important, challenging, but under-explored research topic for autonomous driving, especially for sequences obtained from moving ego vehicles. Most segmentation methods leverage motion cues obtained from optical flow maps. However, since these methods are often based on optical flows that are pre-computed from successive RGB frames, this neglects the temporal consideration of events occurring within the inter-frame, consequently constraining its ability to discern objects exhibiting relative staticity but genuinely in motion. To address these limitations, we propose to exploit event cameras for better video understanding, which provide rich motion cues without relying on optical flow. To foster research in this area, we first introduce a novel large-scale dataset called DSEC-MOS for moving object segmentation from moving ego vehicles, which is the first of its kind. For benchmarking, we select various mainstream methods and rigorously evaluate them on our dataset. Subsequently, we devise EmoFormer, a novel network able to exploit the event data. For this purpose, we fuse the event temporal prior with spatial semantic maps to distinguish genuinely moving objects from the static background, adding another level of dense supervision around our object of interest. Our proposed network relies only on event data for training but does not require event input during inference, making it directly comparable to frame-only methods in terms of efficiency and more widely usable in many application cases. The exhaustive comparison highlights a significant performance improvement of our method over all other methods. The source code and dataset are publicly available at: https://github.com/ZZYZhou/DSEC-MOS.

Moving Object Detection (MOD) is a critical vision task for successfully achieving safe autonomous driving. Despite plausible results of deep learning methods, most existing approaches are only frame-based and may fail to reach reasonable performance when dealing with dynamic traffic participants. Recent advances in sensor technologies, especially the Event camera, can naturally complement the conventional camera approach to better model moving objects. However, event-based works often adopt a pre-defined time window for event representation, and simply integrate it to estimate image intensities from events, neglecting much of the rich temporal information from the available asynchronous events. Therefore, from a new perspective, we propose RENet, a novel RGB-Event fusion Network, that jointly exploits the two complementary modalities to achieve more robust MOD under challenging scenarios for autonomous driving. Specifically, we first design a temporal multi-scale aggregation module to fully leverage event frames from both the RGB exposure time and larger intervals. Then we introduce a bi-directional fusion module to attentively calibrate and fuse multi-modal features. To evaluate the performance of our network, we carefully select and annotate a sub-MOD dataset from the commonly used DSEC dataset. Extensive experiments demonstrate that our proposed method performs significantly better than the state-of-the-art RGB-Event fusion alternatives. The source code and dataset are publicly available at: https://github.com/ZZY-Zhou/RENet.

The idea of cooperation has been introduced to self-driving cars about a decade ago with the aim to reduce the occlusion caused by other users or the scene. More recently, the research efforts turned toward cooperative infrastructure bringing a new kind of the point of view as well as more processing power. This paper lies in this new field providing a survey that addresses the cooperative environment. We provide an overview of the architectures available to create such a system as well as the challenges introduced by the cooperation. Later, we review the main blocks involved in the perception: localization, object detection & tracking, map generation. Each block is reviewed under the prism of cooperation. We also provide a Strengths, Weaknesses, Opportunities, and Threats (SWOT) analysis of the cooperative perception as well as a list of related scenarios alongside experimentations. Finally, we list some related datasets before concluding our paper, underlining the perspectives for further works.

In this paper, we explore the different minimal solutions for egomotion estimation of a camera based on homography knowing the gravity vector between calibrated images. These solutions depend on the prior knowledge about the reference plane used by the homography. We then demonstrate that the number of matched points can vary from two to three and that a direct closed-form solution or a Gröbner basis based solution can be derived according to this plane. Many experimental results on synthetic and real sequences in indoor and outdoor environments show the efficiency and the robustness of our approach compared to standard methods.

With the availability of the recent human skeleton extraction algorithm introduced by Shotton et al. [1], an interest for skeleton-based action recognition methods has been renewed. Despite the importance of the low-latency aspect in applications, it can be noted that the majority of recent approaches has not been evaluated in terms of computational cost. In this paper, a novel fast and accurate human action descriptor named Kinematic Spline Curves (KSC) is introduced. This descriptor is built by interpolating the kinematics of joints (position, velocity and acceleration). To overcome the anthropometric and the execution rate variability, we respectively propose the use of a skeleton normalization and a temporal normalization. For this purpose, a new temporal normalization method based on the Normalized Accumulated kinetic Energy (NAE) of the human skeleton is suggested. Finally, the classification step is performed using a linear Support Vector Machine (SVM). Experimental results on challenging benchmarks show the efficiency of our approach in terms of recognition accuracy and computational latency.

In this paper we present an unsynchronized camera network able to estimate the motion and the structure with accurate absolute scale. The proposed algorithm requires at least three frames: two frames from one camera and a frame from a neighbouring camera. The relative camera poses are estimated with classical Structure-from-Motion and the absolute scales between views are computed by assuming straight trajectories between consecutive views of one camera. We propose a final optimisation step to refine only the scale and the 3D points. Our method is evaluated in real conditions on the KITTI dataset. We show quantitative evaluation through comparisons against GPS/INS ground truth.

In this paper, we tackle the problem of map estimation from small set of vehicular GPS traces collected from low cost devices. Contrary to the existing works, we rely only on GPS information. First, we propose a fast implementation of Kalman filtering of spline-based road modeling. Our approach demonstrates a significant boost of the computation speed while maintained a good estimation error. Secondly, we perform an evaluation of our algorithm on real world data. Our estimation is compared to a high grade Inertial Navigation System and vectorial data gathered from major map providers. Our results suggest that a good performance can be achieved from the fusion of multiple GPS traces collected from multiple vehicles and drivers.

This article studies the feasibility of an exteroceptive system for the contactless control of a motion-compensated gangway which can be used for maintenance operations on offshore wind farms. Our study shows that current systems based only on inertial systems are not accurate enough to ensure the gangway is held in place without being secured mechanically. Using measurements from a 2D LIDAR system, we propose a method for the real-time monitoring of the position of the gangway in relation to the offshore wind turbine. Our algorithm involves detecting and estimating the position of the wind turbine pile in a 2D scatter diagram using robust approaches. To evaluate our method, we have installed a real-time 3D simulation chain fed with data from actual measurements. We obtain a measurement accuracy of the order of a centimeter, in real time, in representative sea state scenarios.

This paper presents a visual odometry with metric scale estimation of a multi-camera system in challenging un-synchronized setup. The intended application is in the field of intelligent vehicles. We propose a new algorithm named “triangle-based” method. The proposed algorithm employs the information from both extrinsic and intrinsic parameters of calibrated cameras. We assume that the trajectory between two consecutive frames of a camera is a linear segment (straight trajectory). The relative camera poses are estimated via classical Structure-from-Motion. Then, the scale factors are computed by imposing the known extrinsic parameters and the linearity assumption. We verify the validity of our method both in simulated and real conditions. For the real world, the motion trajectory estimated for image sequence of two cameras from KITTI dataset is compared against the GPS/INS ground truth.

For decades, scientists have dreamed of building autonomous cars that can drive without a human driver. Progress in this kind of research recently received an increasing attention in car industries. There are many autonomous car models recently developed. However, they are still infancy since they still lack efficiency and reliability. To obtain efficient and reliable systems, the validation process plays an important role. Nowadays, the validation is strongly related to the number of kilometers of drive. Thus, simulation techniques are used before going into real world driving. We focused our work on developing a methodology to smothly move from simulation into real world car driving. We defined a versatile architecture that simplifies the evaluation of different types of algorithms. Several evaluation systems are shown and discussed.