Beyond novel view synthesis, Neural Radiance Fields (NeRF) are useful for applications that interact with the real world. In this paper, we use them as an implicit map of a given scene and propose a camera relocalization algorithm tailored for this representation. The proposed method enables to compute in real-time the precise position of a device using a single RGB camera, during its navigation. In contrast with previous work, we do not rely on pose regression or photometric alignment but rather use dense local features obtained through volumetric rendering which are specialized on the scene with a self-supervised objective. As a result, our algorithm is more accurate than competitors, able to operate in dynamic outdoor environments with changing lightning conditions and can be readily integrated in any volumetric neural renderer.

Accurate velocity estimation is key to vehicle control. While the literature describes how model-based and learning-based observers are able to estimate a vehicle’s velocity in normal driving conditions, the challenge remains to estimate the velocity in near-limits maneuvers while using only conventional in-car sensors. In this paper, we introduce a novel neural network architecture based on Long Short-Term Memory (LSTM) networks to accurately estimate the vehicle’s velocity in different driving conditions, including maneuvers at the limits of handling. The approach has been tested on real vehicle data and it provides more accurate estimations than state-of-the-art model-based and learning-based methods, for both regular and near-limits driving scenarios. Our approach is robust since the performance of the state-of-the-art observers deteriorates with higher dynamics, while our method adapts to different maneuvers, providing accurate estimations even at the vehicle’s limits of handling.

Advanced driving functions, for assistance or full automation, require strong guarantees to be deployed. This means that such functions may not be available all the time, like now commercially available SAE Level 3 [1] modes that are made available only on some roads and at law speeds. The specification of such restriction is described technically in the Operational Design Domain (ODD) which is a fundamental concept for the design of automated driving systems (ADS). In this work, we focus on the example of trajectory planning and control which are crucial functions for SAE level 4+ vehicles and often rely on model-based methods. Hence, the quality of the underlying models has to be evaluated with respect to the ODD. Mathematical analyses such as uncertainty and sensitivity analysis support the quantitative assessment of model quality in general. In this paper, we present a new approach to assess the quality of vehicle dynamics models using an ODD-centric sensitivity analysis. The sensitivity analysis framework is implemented for a 10-DoF nonlinear double-track vehicle dynamics model used inside a model-predictive trajectory controller. The model sensitivity is evaluated with respect to given ODD and maneuver parameters. Based on the results, ODD-compliant behavior generation strategies with the goal of minimizing model sensitivity are outlined.

We propose a novel learning-based formulation for visual localization of vehicles that can operate in real-time in city-scale environments. Visual localization algorithms determine the position and orientation from which an image has been captured, using a set of geo-referenced images or a 3D scene representation. Our new localization paradigm, named Implicit Pose Encoding (ImPosing), embeds images and camera poses into a common latent representation with 2 separate neural networks, such that we can compute a similarity score for each image-pose pair. By evaluating candidates through the latent space in a hierarchical manner, the camera position and orientation are not directly regressed but incrementally refined. Very large environments force competitors to store gigabytes of map data, whereas our method is very compact independently of the reference database size. In this paper, we describe how to effectively optimize our learned modules, how to combine them to achieve real-time localization, and demonstrate results on diverse large scale scenarios that significantly outperform prior work in accuracy and computational efficiency.

Even after decades of research, dynamic scene background reconstruction and foreground object segmentation are still considered as open problems due to various challenges such as illumination changes, camera movements, or background noise caused by air turbulence or moving trees. We propose in this paper to model the background of a frame sequence as a low dimensional manifold using an autoencoder and compare the reconstructed background provided by this autoencoder with the original image to compute the foreground/background segmentation masks. The main novelty of the proposed model is that the autoencoder is also trained to predict the background noise, which allows to compute for each frame a pixel-dependent threshold to perform the foreground segmentation. Although the proposed model does not use any temporal or motion information, it exceeds the state of the art for unsupervised background subtraction on the CDnet 2014 and LASIESTA datasets, with a significant improvement on videos where the camera is moving. It is also able to perform background reconstruction on some non-video image datasets.

This paper proposes a new process for calibrating a magnetometer in indoor environments that is accurate, easily deployable and time efficient. Our approach simultaneously estimates the calibration of the magnetometer with the local variations of the magnetic field modeled by a Gaussian Process. To guarantee an accurate estimation of the magnetometer calibration, a two-step optimization algorithm is proposed. Experiments show that the proposed solution is as accurate as outdoor calibration algorithms, and more precise than state-of-the-art indoor calibration methods.

In this paper, we investigate visual-based camera re-localization with neural networks for robotics and autonomous vehicles applications. Our solution is a CNN-based algorithm which predicts camera pose (3D translation and 3D rotation) directly from a single image. It also provides an uncertainty estimate of the pose. Pose and uncertainty are learned together with a single loss function and are fused at test time with an EKF. Furthermore, we propose a new fully convolutional architecture, named CoordiNet, designed to embed some of the scene geometry.Our framework outperforms comparable methods on the largest available benchmark, the Oxford RobotCar dataset, with an average error of 8 meters where previous best was 19 meters. We have also investigated the performance of our method on large scenes for real time (18 fps) vehicle localization. In this setup, structure-based methods require a large database, and we show that our proposal is a reliable alternative, achieving 29cm median error in a 1.9km loop in a busy urban area.

Cameras and 2D laser scanners, in combination, are able to provide low-cost, light-weight and accurate solutions, which make their fusion well-suited for many robot navigation tasks. However, correct data fusion depends on precise calibration of the rigid body transform between the sensors. In this paper we present the first framework that makes use of Convolutional Neural Networks (CNNs) for odometry estimation fusing 2D laser scanners and mono-cameras. The use of CNNs provides the tools to not only extract the features from the two sensors, but also to fuse and match them without needing a calibration between the sensors. We transform the odometry estimation into an ordinal classification problem in order to find accurate rotation and translation values between consecutive frames. Results on a real road dataset show that the fusion network runs in real-time and is able to improve the odometry estimation of a single sensor alone by learning how to fuse two different types of data information.

Urban environments are still a challenge for Autonomous Vehicles, due to strong interactions with other vehicles and pedestrians. Machine learning methods are increasingly explored to tackle these situations, but their performances are highly conditioned on the availability of vehicle trajectory datasets. Such datasets are usually created with specific infrastructures and equipments, which is costly and not scalable. As a result, only a few datasets of vehicle trajectories are currently available, representing very specific situations such as highway driving, and containing a limited number of trajectories. We argue that to unleash the potential of behavior learning methods for autonomous vehicles, we need large datasets of accurate vehicle trajectories, representing interacting vehicles in very diverse situations. In this paper, we introduce a fully automatic and scalable framework for accurate vehicle trajectory extraction from single fixed monocular traffic cameras. We leverage the fact that traffic cameras represent a very large and cost-effective source of highly diverse vehicle trajectories, as they are generally located at places where traffic is dense and where a lot of interactions occur (e.g intersections). With this work, we aim at developing a framework for effortless and accurate vehicle trajectory dataset creation at large-scale. Furthermore, we provide an open-source implementation of this framework https://github.com/AubreyC/trajectory-extractor.

The use of 2D laser scanners is attractive for the autonomous driving industry because of its accuracy, light-weight and low-cost. However, since only a 2D slice of the surrounding environment is detected at each scan, it is a challenge to execute important tasks such as the localization of the vehicle. In this paper we present a novel framework that explores the use of deep Recurrent Convolutional Neural Networks (RCNN) for odometry estimation using only 2D laser scanners. The application of RCNNs provides the tools to not only extract the features of the laser scanner data using Convolutional Neural Networks (CNNs), but in addition it models the possible connections among consecutive scans using the Long Short-Term Memory (LSTM) Recurrent Neural Network. Results on a real road dataset show that the method can run in real-time without using GPU acceleration and have competitive performance compared to other methods, being an interesting approach that could complement traditional localization systems.

Automation driving techniques have seen tremendous progresses these last years, particularly due to a better perception of the environment. In order to provide safe yet not too conservative driving in complex urban environment, data fusion should not only consider redundant sensing to characterize the surrounding obstacles, but also be able to describe the uncertainties and errors beyond presence/absence (be it binary or probabilistic). This paper introduces an enriched representation of the world, more precisely of the potential existence of obstacles through an evidential grid map. A method to create this representation from 2 very different sensors, laser scanner and stereo camera, is presented along with algorithms for data fusion and temporal updates. This work allows a better handling of the dynamic aspects of the urban environment and a proper management of errors in order to create a more reliable map. We use the evidential framework based on the Dempster-Shafer theory to model the environment perception by the sensors. A new combination operator is proposed to merge the different sensor grids considering their distinct uncertainties. In addition, we introduce a new long-life layer with high level states that allows the maintenance of a global map of the entire vehicle's trajectory and distinguish between static and dynamic obstacles. Results on a real road dataset show that the environment mapping data can be improved by adding relevant information that could be missed without the proposed approach.

The self-localization of a vehicle is still an ongoing and challenging task for the autonomous driving development. At the same time, the correct understanding of the vehicle's surroundings and the creation of a map with the traversed trajectory is essential for driving in complex urban scenarios. This paper proposes a solution to create an enriched global map of the environment while localizing the vehicle within it. We use the Evidential framework based on the Dempster-Shafer theory to create a map able to distinguish between static and dynamic obstacles and to keep the information from the entire traversed path. Additionally, we propose a method that at each current map creation estimates the vehicle's position by a grid matching algorithm based on image registration techniques. The method transforms the grid maps into images using the evidential data to extract relevant information to perform a better localization. The proposed method was tested on the KITTI dataset and shows that the solution is able to reduce the drift and improve the localization compared to other methods using similar configurations.

Lane change is a crucial vehicle maneuver which needs coordination with surrounding vehicles. Automated lane changing functions built on rule-based models may perform well under pre-defined operating conditions, but they may be prone to failure when unexpected situations are encountered. In our study, we proposed a Reinforcement Learning based approach to train the vehicle agent to learn an automated lane change behavior such that it can intelligently make a lane change under diverse and even unforeseen scenarios. Particularly, we treated both state space and action space as continuous, and designed a Q-function approximator that has a closed-form greedy policy, which contributes to the computation efficiency of our deep Q-learning algorithm. Extensive simulations are conducted for training the algorithm, and the results illustrate that the Reinforcement Learning based vehicle agent is capable of learning a smooth and efficient driving policy for lane change maneuvers.

Accurate probabilistic prediction for intention and motion of road users is a key prerequisite to achieve safe and high-quality decision-making and motion planning for autonomous driving. Typically, the performance of probabilistic predictions was only evaluated by learning metrics for approximation to the motion distribution in the dataset. However, as a module supporting decision and planning, probabilistic prediction should also be evaluated from decision and planning perspective. Moreover, the evaluation of probabilistic prediction highly relies on the problem formulation variation and motion representation simplification, which lacks a formal foundation in a comprehensive framework. To address such concerns, we provide a systematic and unified framework for the analysis of three under-explored aspects of probabilistic prediction: problem formulation, representation simplification and evaluation metric. More importantly, we address the omitted but crucial problems in the three aspects from decision and planning perspective. In addition to a review of learning metrics, metrics to be considered from planning perspective are highlighted, such as planning consequence of inaccurate and erroneous prediction, as well as violations of predicted motions to planning constraints. We address practical formulation variations of prediction problems, such as decision-maker view and blind view for viewpoint, as well as reactive prediction for interaction, so that decision and planning can be facilitated.

With the emerging vehicular network and the possible diverse applications, Intelligent Transportation Systems (ITS) have been evolving to Cooperative ITS (C-ITS, CITS) with connected intelligent vehicles, and the topics in this domain also raise more and more research interesting recently. However, subjecting to the immaturity of V2X communication technology and the deployment of intelligent vehicles in large scale, such studies and the corresponding verification are all challenged within current situations. Focusing on several emerging features, such as cyber-physical fusion, vehicular networking, service-carrier etc, a new simulator QoS-CITS for such service-oriented C-ITS is designed. To enhance the adaptivity, an para-reconfigurable architecture is firstly adopted, within which all reservation-based models of traffic objects, state-driven behaviors, cooperation mechanisms, and policies proposed for service-oriented C-ITS are implemented. And then, a series of experiments are conducted via employing parameters within typical scenes, and all fundamental functions of QoS-CITS are verified. With this simulator, researchers can carry out various experiments, not only classic ones but service-oriented, via setting parameters according to their study and verification requirements, and then analyze the performance with statistics data automatically recorded.

In normal on-road situations, autonomous vehicles will be expected to have smooth trajectories with relatively little demand on the vehicle dynamics to ensure passenger comfort and driving safety. However, the occurrence of unexpected events may require vehicles to perform aggressive maneuvers, near the limits of their dynamic capacities. In order to ensure the occupant's safety in these situations, the ability to plan controllable but near-limits trajectories will be of very high importance. One of the main issues in planning aggressive maneuvers lies in the high complexity of the vehicle dynamics near the handling limits, which effectively makes state-of-the-art methods such as Model Predictive Control difficult to use. This article studies a highly precise model of the vehicle body to derive a simpler, constrained second-order integrator dynamic model which remains precise even near the handling limits of the vehicle. Preliminary simulation results indicate that our model provides better accuracy without increasing computation time compared to a more classical kinematic bicycle model. The proposed model can find applications for contingency planning, which may require aggressive maneuvers, or for trajectory planning at high speed, for instance in racing applications.

Most autonomous driving architectures separate planning and control phases in different layers, even though both problems are intrinsically related. Due to limitations on the available computational power, their levels of abstraction and modeling differ; in particular, vehicle dynamics are often highly simplified at the planning phase, which may lead to inconsistency between the two layers. In this paper, we study the kinematic bicycle model, which is often used for trajectory planning, and compare its results to a 9 degrees of freedom model. Modeling errors and limitations of the kinematic bicycle model are highlighted. Lastly, this paper proposes a simple and efficient consistency criterion in order to validate the use of this model for planning purposes.

Before reaching full autonomy, vehicles will gradually be equipped with more and more advanced driver assistance systems (ADAS), effectively rendering them semi-autonomous. However, current ADAS technologies seem unable to handle complex traffic situations, notably when dealing with vehicles arriving from the sides, either at intersections or when merging on highways. The high rate of accidents in these settings proves that they constitute difficult driving situations. Moreover, intersections and merging lanes are often the source of important traffic congestion and, sometimes, deadlocks. In this paper, we propose a cooperative framework to safely coordinate semi-autonomous vehicles in such settings, removing the risk of collision or deadlocks while remaining compatible with human driving. More specifically, we present a supervised coordination scheme that overrides control inputs from human drivers when they would result in an unsafe or blocked situation. To avoid unnecessary intervention and remain compatible with human driving, overriding only occurs when collisions or deadlocks are imminent. In this case, safe overriding controls are chosen while ensuring they deviate minimally from those originally requested by the drivers. Simulation results based on a realistic physics simulator show that our approach is scalable to real-world scenarios, and computations can be performed in real time on a standard computer for up to a dozen simultaneous vehicles.

As driverless vehicles are increasingly becoming possible, so does the use of such vehicles as intelligent carriers in different domains. Intelligent transportation systems (ITSs) show increasingly heterogeneous, cyber-physical, cooperative, and service-oriented features and are beginning to be merged with the emerging cyber-physical-social systems. Given this new trend, how to make these intelligent vehicles cooperate more safely and efficiently with one another according to novel constraints, such as mission type and quality-of-service (QoS), has become a vital aspect of cooperative ITS (C-ITS). With these emerging characteristics, the classical passing-through-intersection problem has gained new connotations, worth further exploring. After analyzing the essences of this new problem, service-oriented cooperation models and mechanisms for whole autonomous vehicles approaching intersections are investigated in this paper. First, related traffic objects and possible vehicular behaviors are abstracted and modeled with the cyber-physical cooperative features and QoS constraints. A new reservation-based scheduling procedure is then conducted by employing the concepts of vehicle-to-infrastructure communication, and typical vehicular passing-through behaviors and several spatial-temporal constraints are designed to coordinate vehicles passing through an intersection divided as a series of continuous static critical sections. Given these considerations, a priority-based centralized scheduling algorithm, named csPriorFIFO, which adopts a novel priority inheritance mechanism to promote the traffic QoS of emergent vehicles, is proposed. Finally, all these designs are implemented in a traffic simulator named QoS-CITS, and the functions and the performance of these studied methods are verified and compared.

This work presents a novel framework for the formation control of multiple autonomous ground vehicles in an on-road environment. Unique challenges of this problem lie in 1) the design of collision avoidance strategies with obstacles and with other vehicles in a highly structured environment, 2) dynamic reconfiguration of the formation to handle different task specifications. In this paper, we design a local MPC-based trajectory planner for each individual vehicle to follow a reference trajectory while satisfying the various kinematic and dynamic constraints of the vehicles as well as collision avoidance and formation-keeping requirements. The reference trajectory of a vehicle is computed from its leader's trajectory, based on a predefined formation tree. We use logic rules to organize the collision avoidance behaviors of member vehicles. Moreover, we propose a methodology to safely reconfigure the formation on-the-fly. The proposed framework has been validated using high-fidelity simulations.

In this paper, we address the problem of time-optimal coordination of mobile robots under kinodynamic constraints along specified paths. We propose a novel approach based on time discretization that leads to a mixed-integer linear programming (MILP) formulation. This problem can be solved using general-purpose MILP solvers in a reasonable time, resulting in a resolution-optimal solution. Moreover, unlike previous work found in the literature, our formulation allows an exact linear modeling (up to the discretization resolution) of second-order dynamic constraints. Extensive simulations are performed to demonstrate the effectiveness of our approach.

The deployment of Cooperative Intelligent Transportation Systems (C-ITS) raises the question of future traffic management systems, which will be operating with an increasing amount of information and control over the infrastructure and the vehicles. This topic of research shares some similarities with robot coordination problems, inspiring our research on autonomous intersection management. In this article, we use a mixed-integer linear programming formulation for time-optimal robots coordination along specified paths and apply it to intersection management for autonomous vehicles. Our formulation allows to simultaneously solve a discrete optimal vehicle ordering problem, and a (discretized) continuous optimal velocity planning problem taking into account kinodynamics constraints. This allows faster pruning of the decision tree for the discrete problem, thus reducing computation time. A possible application for ITS is to evaluate the efficiency loss from a given vehicle ordering policy, or dynamically adapt policies to improve their efficiency. Moreover, any intermediary solution found by the solver can be used as a heuristically good policy, with proved bounds on sub-optimality.

This paper presents an approach for the formation control of autonomous vehicles traversing along a multi-lane road with obstacles and traffic. A major challenge in this problem is a requirement for integrating individual vehicle behaviors such as lane-keeping and collision avoidance with a global formation maintenance behavior. We propose a hierarchical Model Predictive Control (MPC) approach. The desired formation is modeled as a virtual structure evolving curvilinearly along a centerline, and vehicle configurations are expressed as curvilinear relative longitudinal and lateral offsets from the virtual center. At high-level, the trajectory generation of the virtual center is achieved through an MPC framework, which allows various on-road driving constraints to be considered in the optimization. At low-level, a local MPC controller computes the vehicle inputs in order to track the desired trajectory, taking into account more personalized driving constraints. High-fidelity simulations show that the proposed approach drives vehicles to the desired formation while retains some freedom for individual vehicle behaviors.

Scheduling driverless vehicles with different priorities to pass through intersections efficiently and safely has been becoming an important passing-through intersection (PTI) problem in the field of novel intelligent traffic systems (ITS), which is increasingly becoming cyber-physical-fused and social-service-oriented. Considering new emerging features with possible priorities, a novel centralized priority scheduling mechanism is mainly explored in this paper. First, related pivotal aspects of environment and driverless vehicles are modeled by fusing their physical and kinematic characters. Based on these models, PTI-related motions are further abstracted as several reservation-oriented standard states and actions. Then, an event-triggered and state-driven autonomous control procedure is designed. By mapping vehicular relations in spatiotemporal domain into time-distance windows, a universal passing-through principle, rules, and priority-based scheduling mechanisms are proposed and described in detail. Finally, a priority scheduling algorithm sPriorFIFO is proposed and designed. These models and mechanisms are then implemented within an algorithm simulator, through which scheduling performances are verified and evaluated.

Networked cooperation is one primarily problem for autonomous Cyber-Physical Systems(CPS) that feature cyber-physical fusion, and smart behavior. After analyzing current contributions, a three-level task model, with which missions can be recognized and mapped to tasks, is studied firstly in this paper. And then, based on a designed uniform model of the surroundings, external missions and internal computational resources, two hybrid models, which are autonomously cooperative and real-time reactive by taking advantages of agent and adaptive control, are illuminated respectively. Further, a hierarchal cooperation mechanism of CPS fleet is put forward. With this mechanism, the dynamic topologies of a fleet will be more flexible and dependable, and by adopting an intelligent algorithm, all tasks mapped from a mission can be autonomously assigned to suitable members under decentralized consultation or centralized allocation mode. In this paper, an optimized genetic algorithm is employed to illustrate the process of proposed mechanisms.

A multi-homed vehicular mobile network uses different wireless and radio communication technologies to reach the Internet through multiple routing paths. Because of network mobility and lack of reliable quality-of-service (QoS) support in these communicating technologies, it is challenging to guarantee high quality in vehicle-to-infrastructure (V2I) communications. In this paper, we evaluate a novel policy to support resource reservation for mobile networks. We also define a probing mechanism based on IPv6 messages and traffic load estimation at mobile routers and home agent to enable NEMOs choose the most optimal egress routing path to reach Internet. Intelligent transportation service providers (ITSPs) can deploy this proposed QoS architecture to realize Service Level Agreements (SLAs) based on QoS negotiation with NEMOs or quality in provisioned multimedia services for on-board users.

We develop an intelligent distributed quality-of-service (QoS) control scheme which inter-operates between mobile routers, managing vehicular networks mobility, and road communication gateways (RCGs). This scheme manages vehicle-to-infrastructure (V2I) communications by enabling multi-homed vehicular networks to evenly distribute traffic among egress links of their mobile routers based on vehicular communication policies, available bandwidth and performance metrics of selected routing paths with correspondent nodes (CNs). In this scheme, the data control plane is considered as a collaborative entity and specifies detailed operations to be performed in the mobile routers and RCGs. Simulation experiments show that our proposed scheme can improve the congestion window (CWND) of TCP and the e2e packet loss of video traffic, despite network mobility. It also guarantees the service parameter settings of each uplink and downlink connection, while achieving reasonable utilization efficiency of network resources.

When considering the need for an intelligent transportation mode based upon the fact that car travel is still the consumer's first choice, we decided to explore the idea of collective taxis. More precisely, we are seeking to provide an autonomous high quality door-to-door service, affordable by almost everyone, covering the entire urban area, simply by allocating in an optimal way more than one passenger to each vehicle and using the latest available information regarding traffic conditions, etc. Because this system is far less unconstrained than most public transportation systems, it has high potentialities in terms of performance and flexibility, but it is consequently very difficult to manage and to design optimally. Therefore, we present a discrete event simulation tool and show how it can be used to conceive, optimise and compare decision algorithms, achieve sound trade-off between conflicting performance indicators and assess the efficiency of the system in various contexts.

Localization with respect to a reference model is a key feature for mobile robots. Urban environment offers numerous landmarks that can be used for the localization process. This paper deals with the use of an environment model stored in a geographic information system, to drive a vision system i.e. highlights what to look for and where to look for. This task is achieved by propagating uncertainties along the image acquisition system to highlight some region of interest in the image.

We introduce a class of Markov models, termed best-effort networks, designed to capture performance indices such as mean transfer times in data networks with best-effort service. We introduce the so-called min bandwidth sharing policy as a conservative approximation to the classical max-min policy. We establish necessary and sufficient ergodicity conditions for best-effort networks under the min policy. We then resort to the mean field technique of statistical physics to analyze network performance deriving fixed point equations for the stationary distribution of large symmetrical best-effort networks. A specific instance of such networks is the star-shaped network which constitutes a plausible model of a network with an overprovisioned backbone. Numerical and analytical study of the equations allows us to state a number of qualitative conclusions on the impact of traffic parameters (link loads) and topology parameters (route lengths) on mean document transfer time.

REALISE was designed to extract from sequences of images, acquired with a moving camera, the information necessary for determining the 3D (CAD-like) structure of a real-life scene together with information about the radiometric signatures of surfaces bounding the extracted 3D objects (e.g. reflectance behaviour). The retrieved information is then integrated in a virtual reality (VR) software environment. The R&D work is been performed principally in the following areas of computer vision and computer graphics: structure from motion, recovery of geometries, recovery of photometric and texture information, highly realistic rendering on the basis of empirically-based reflectance models, and the design and development of improved rendering processes together with a new VR system. Beside this innovative R&D work another key aspect of REALISE is to have computer vision & computer graphics cooperate to produce realistic 3D data efficiently.