Reinforcement Learning from Demonstrations (RLfD) has emerged as an effective method by fusing expert demonstrations into Reinforcement Learning (RL) training, harnessing the strengths of both Imitation Learning (IL) and RL. However, existing algorithms rely on offline demonstrations, which can introduce a distribution gap between the demonstrations and the actual training environment, limiting their performance. In this paper, we propose a novel approach, Reinforcement Learning from Online Demonstrations (RLfOLD), that leverages online demonstrations to address this limitation, ensuring the agent learns from relevant and up-to-date scenarios, thus effectively bridging the distribution gap. Unlike conventional policy networks used in typical actor-critic algorithms, RLfOLD introduces a policy network that outputs two standard deviations: one for exploration and the other for IL training. This novel design allows the agent to adapt to varying levels of uncertainty inherent in both RL and IL. Furthermore, we introduce an exploration process guided by an online expert, incorporating an uncertainty-based technique. Our experiments on the CARLA NoCrash benchmark demonstrate the effectiveness and efficiency of RLfOLD. Notably, even with a significantly smaller encoder and a single camera setup, RLfOLD surpasses state-of-the-art methods in this evaluation. These results, achieved with limited resources, highlight RLfOLD as a highly promising solution for real-world applications.

Current approaches of applied in urban focus on decoupling the perception training from the driving policy training. The main reason is to avoid training a convolution encoder alongside a policy network, which is known to have issues related to sample efficiency, degenerated feature representations, and catastrophic self-overfitting. However, this paradigm can lead to representations of the environment that are not aligned with the downstream task, which may result in suboptimal performances. To address this limitation, this paper proposes RLAD, the first method applied in the urban ad domain. We propose several techniques to enhance the performance of an rlfp algorithm in this domain, including: 1) an image encoder that leverages both image augmentations and layers; 2) WayConv1D, which is a waypoint encoder that harnesses the 2D geometrical information of the waypoints using 1D convolutions; and 3) an auxiliary loss to increase the significance of the traffic lights in the latent representation of the environment. Experimental results show that RLAD significantly outperforms all state-of-the-art rlfp methods on the NoCrash benchmark. We also present an infraction analysis on the NoCrash-regular benchmark, which indicates that RLAD performs better than all other methods in terms of both collision rate and red light infractions. The source code of RLAD is available at https://github.com/DanielCoelho112/rlad.Note to Practitioners—The practical problem that motivated our work is the application of RLfP in urban AD. Unlike common approaches in AD, our main goal is to learn intermediate feature representations of the environment that are aligned with the driving task. Our method is the first RLfP algorithm applied in the urban AD domain, and results show that it surpasses all state-of-the-art RLfP methods on the NoCrash benchmark. The key insights and contributions of our work lie in the development of effective techniques for RLfP in urban AD, improving the overall performance and safety of AD systems. Beyond AD, the techniques and insights from our work could be extended to other domains that involve RL from pixel inputs, such as robotics, and video game agents. These applications would benefit from the enhanced performance offered by our proposed methods.