The goal of offline reinforcement learning (RL) is to extract the best possible policy from the previously collected dataset considering the out-of-distribution (OOD) sample issue. Offline model-based RL (MBRL) is a captivating solution capable of alleviating such issues through a state-action transition augmentation with a learned dynamic model. Unfortunately, offline MBRL methods have been observed to fail in sparse rewarded and long-horizon environments for a long time. In this work, we propose a novel MBRL method, dubbed Temporal Distance-Aware Transition Augmentation (TempDATA), that generates additional transitions in a geometrically structured representation space, instead of state space. For comprehending long-horizon behaviors efficiently, our main idea is to learn state abstraction, which captures a temporal distance from both trajectory and transition levels of state space. Our experiments empirically confirm that TempDATA outperforms previous offline MBRL methods and achieves matching or surpassing the performance of diffusion-based trajectory augmentation and goal-conditioned RL on the D4RL AntMaze, FrankaKitchen, CALVIN, and pixel-based FrankaKitchen.

Skill Incremental Learning (SIL) is the process by which an embodied agent expands and refines its skill set over time by leveraging experience gained through interaction with its environment or by the integration of additional data. SIL facilitates efficient acquisition of hierarchical policies grounded in reusable skills for downstream tasks. However, as the skill repertoire evolves, it can disrupt compatibility with existing skill-based policies, limiting their reusability and generalization. In this work, we propose SIL-C, a novel framework that ensures skill-policy compatibility, allowing improvements in incrementally learned skills to enhance the performance of downstream policies without requiring policy re-training or structural adaptation. SIL-C employs a bilateral lazy learning-based mapping technique to dynamically align the subtask space referenced by policies with the skill space decoded into agent behaviors. This enables each subtask, derived from the policy's decomposition of a complex task, to be executed by selecting an appropriate skill based on trajectory distribution similarity. We evaluate SIL-C across diverse SIL scenarios and demonstrate that it maintains compatibility between evolving skills and downstream policies while ensuring efficiency throughout the learning process.

The Internet of Vehicles (IoV) is a network that connects various transportation devices, where autonomous vehicles play a crucial role. These vehicles must be able to coexist and drive alongside human-driven vehicles. To ensure a safer IoV, it is essential to develop adaptive decision-making capabilities, even under the uncertainty posed by human drivers. This study aims to develop an autonomous vehicle with the capacity for foresighted decision-making in complex multi-agent interactions by integrating predictive abilities of future observations. The proposed autonomous vehicle is equipped with episodic future thinking (EFT) capabilities. Specifically, EFT uses a prediction network to simulate future observations and guide adaptive decision-making of actor-critic networks based on envisioned scenarios. We use an offline reinforcement learning paradigm to train both the prediction network and the EFT-based actor-critic networks. To assess the generalized performance of the proposed solution, we integrate the EFT module with three popular offline reinforcement learning algorithms. We run an extensive series of performance evaluations across three driving scenarios involving multi-agent interactions and use four datasets for offline training, including a real-world dataset. Simulation results demonstrate that the proposed solution with the EFT module improves average performance in most cases compared to the baseline without the EFT module. Additionally, we compare the proposed solution with three existing model-based reinforcement learning approaches, confirming its superiority. Finally, we analyze driving behavior in terms of agility, safety, and stability.

The primary goal of an autonomous driving system is to achieve full autonomy by integrating the capability of adaptive decision-making across a wide range of driving scenarios. Despite recent advances in reinforcement learning (RL) enabling the development of policies for adaptive behaviors, current solutions are typically tailored for specific driving scenarios (e.g., highway, tollgate) rather than providing a scenario-free solution. To address this limitation, this study focuses on developing an autonomous driving policy that can operate across diverse driving scenarios through a generalized decision-making model. Specifically, an autonomous vehicle learns a unified multi-task policy by utilizing a shared replay buffer across all scenarios, thereby enhancing sample and learning efficiencies. Furthermore, we adopt an offline-to-online RL approach to leverage both the sample efficiency of offline RL and the performance improvements of online RL. The proposed solution involves an algorithmic shift aimed at maximizing the objectives of each RL method, incorporating three key techniques: Q re-initialization, Q adaptation, and policy variance re-initialization. To validate our solution, we compare its performance with existing RL methods and analyze driving behavior using objective-aware and safety-aware metrics. Our findings demonstrate that the proposed solution achieves superior performance across most metrics, irrespective of dataset quality.

Offline reinforcement learning has emerged as a promising technology by enhancing its practicality through the use of pre-collected large datasets. Despite its practical benefits, most algorithm development research in offline reinforcement learning still relies on game tasks with synthetic datasets. To address such limitations, this paper provides autonomous driving datasets and benchmarks for offline reinforcement learning research. We provide 19 datasets, including real-world human driver’s datasets, and seven popular offline reinforcement learning algorithms in three realistic driving scenarios. We also provide a unified decision-making process model that can operate effectively across different scenarios, serving as a reference framework in algorithm design. Our research lays the groundwork for further collaborations in the community to explore practical aspects of existing reinforcement learning methods. Dataset and codes can be found in https://sites.google.com/view/ad4rl.

Understanding cognitive processes in multi-agent interactions is a primary goal in cognitive science. It can guide the direction of artificial intelligence (AI) research toward social decision-making in multi-agent systems, which includes uncertainty from character heterogeneity. In this paper, we introduce *episodic future thinking (EFT) mechanism* for a reinforcement learning (RL) agent, inspired by the cognitive processes observed in animals. To enable future thinking functionality, we first develop a *multi-character policy* that captures diverse characters with an ensemble of heterogeneous policies. The *character* of an agent is defined as a different weight combination on reward components, representing distinct behavioral preferences. The future thinking agent collects observation-action trajectories of the target agents and leverages the pre-trained multi-character policy to infer their characters. Once the character is inferred, the agent predicts the upcoming actions of target agents and simulates the potential future scenario. This capability allows the agent to adaptively select the optimal action, considering the predicted future scenario in multi-agent scenarios. To evaluate the proposed mechanism, we consider the multi-agent autonomous driving scenario in which autonomous vehicles with different driving traits are on the road. Simulation results demonstrate that the EFT mechanism with accurate character inference leads to a higher reward than existing multi-agent solutions. We also confirm that the effect of reward improvement remains valid across societies with different levels of character diversity.

With the rapid development of modern high resolution video streaming services, providing high Quality of Experience (QoE) has become a crucial service for any media streaming platforms. Most often it is necessary of provide the QoE with NR-IQA, which is a daunting task for any present network system for it’s huge computational overloads and often inaccurate results. So in this research paper a new type of this NR-IQA was proposed that resolves these issues. In this work we have described a deep-learning based Convolutional Neural Network (CNN) to accurately predict image quality without a reference image. This model processes the RAW RGB pixel images as input, the CNN works in the spatial domain without using any hand-crafted or derived features that are employed by most previous methods. The proposed CNN is utilized to classify all images in a MOS category. This approach achieves state of the art performance on the KoniQ-10k dataset and shows excellent generalization ability in classifying proper images into proper category. Detailed processing on images with data augmentation revealed the high quality estimation and classifying ability of our CNN, which is a novel system by far in these field.

Recently, the researches on DNN (deep neural network) using GPUs has been actively conducted. The reason for using GPUs in DNN is that it reduces the learning time by using many computational cores. However, GPUs have no implements to support the reliable computing operations. It can exacerbate the reliability of the deep neural network. To ensure the reliability of the deep neural network, the proposed approach is to utilize the dropout technique used in MLP (Multi-Layer Perceptron) learning. In case of the dropout method, some threads in GPUs do not participate in the calculation and it causes GPUs underutilization. The proposed method uses the GPUs underutilization to support a reliable deep neural network. The proposed approach is available through using the idle neurons with adjacent calculating neurons in the dropout process. The experiment results show that the proposed approach is able to support the reliability issues in GPUs while executing deep neural network algorithms.