With the increasing focus on research on autonomous driving, road environments have evolved into mixed-autonomy traffic networks. In this context, developing an autonomous strategy that can reduce societal costs is important because autonomous vehicles have a direct impact on the entire traffic network. Deep reinforcement learning (RL), a promising autonomous decision-making process, typically leads to anegocentric strategycharacterized by a static target and disregards rapidly changing traffic conditions. However, this approach can incur significant societal costs in complex traffic scenarios. In this study, we propose afast-follower strategythat effectively reduces societal costs in a mixed-autonomy traffic network by dynamically adjusting the reward standards to accommodate varying traffic conditions. To assess the impact of autonomous strategies on transportation networks, we introduce a novel metric, theprice of autonomous strategy(PoAS), which is designed to quantify the societal costs associated with autonomous decision-making. Additionally, we provide a traffic-aware analysis using PoAS to identify the driving conditions under which the fast-follower strategy results in a lower societal cost than the egocentric strategy. This theoretical analysis is validated using PoAS comparisons across various societal metrics and traffic conditions. The simulation results confirm that the fast-follower strategy outperforms other autonomous strategies in mixed and fully autonomous traffic networks.

The expansion of machine learning into dynamic environments presents challenges in handling open-world problems where label shift, covariate shift, and unknown classes emerge concurrently. Post-training methods have been explored to address these challenges, adapting models to newly emerging data. However, these methods struggle when the initial pre-training is performed on class-imbalanced datasets, limiting generalization to minority classes. To address this, we propose OASIS, an Open-world Adaptive Self-supervised and Imbalanced-aware System. OASIS consists of two learning phases: pre-training and post-training. The pre-training phase aims to improve the classification performance of samples near class boundaries via a novel borderline sample refinement step. Notably, the borderline sample refinement step critically improves the robustness of the decision boundary in the representation space. Through this robustness of the pre-trained model, OASIS generates reliable pseudo-labels, adapting the model against open-world problems in the post-training phase. Extensive experiments demonstrate that our method significantly outperforms state-of-the-art post-training techniques in both accuracy and efficiency across diverse open-world scenarios.

In real-world applications, machine learning models face online label shift, where label distributions change over time. Effective adaptation requires careful learning rate selection: too low slows adaptation and too high causes instability. We propose ASAP (Adaptive Shift Aware Post-training), which dynamically adjusts the learning rate by computing the cosine distance between current and previous unlabeled outputs and mapping it within a bounded range. ASAP requires no labels, model ensembles, or past inputs, using only the previous softmax output for fast, lightweight adaptation. Experiments across multiple datasets and shift scenarios show ASAP consistently improves accuracy and efficiency, making it practical for unsupervised model adaptation.

The goal of offline reinforcement learning (RL) is to extract the best possible policy from the previously collected dataset considering the out-of-distribution (OOD) sample issue. Offline model-based RL (MBRL) is a captivating solution capable of alleviating such issues through a state-action transition augmentation with a learned dynamic model. Unfortunately, offline MBRL methods have been observed to fail in sparse rewarded and long-horizon environments for a long time. In this work, we propose a novel MBRL method, dubbed Temporal Distance-Aware Transition Augmentation (TempDATA), that generates additional transitions in a geometrically structured representation space, instead of state space. For comprehending long-horizon behaviors efficiently, our main idea is to learn state abstraction, which captures a temporal distance from both trajectory and transition levels of state space. Our experiments empirically confirm that TempDATA outperforms previous offline MBRL methods and achieves matching or surpassing the performance of diffusion-based trajectory augmentation and goal-conditioned RL on the D4RL AntMaze, FrankaKitchen, CALVIN, and pixel-based FrankaKitchen.

Tumor tissues are composed of malignant subclones with diverse genetic profiles. Reconstructing the evolutionary trajectory of these subclones is crucial for understanding how tumors acquire malignant traits. However, current approaches to subclonal tree reconstruction are limited either by their reliance on single-cell DNA sequencing (scDNA-seq) that involve a small number of cells and thus yield low-resolution results, or using single-cell RNA sequencing (scRNA-seq) data, which despite including larger cell populations, remain susceptible to bias from high dropout rates and technical noise. Here, we introduce CluVar, an autoencoder-based framework for inferring the phylogeny of cancer subclones from scRNA-seq data using mutation profile analysis. To address the extensive missing variant information inherent in scRNA-seq datasets, CluVar incorporates a customized loss function and multiple hidden layers optimized for clustering. CluVar demonstrated superior performance in reconstructing phylogenetic trees of cancer subclones under a range of erroneous conditions. When applied to cancer scRNA-seq data, the phylogenetic tree predicted using CluVar aligned well with the transcriptomic profiles. These findings highlight its utility for tracing evolutionary trajectories and identifying novel variants associated with cancer progression.

Advancements in artificial intelligence (AI) have enabled Internet of Things (IoT) devices to offer intelligent services, improving system adaptability and scalability. Split federated learning (SFL) has emerged as a promising approach for privacy-sensitive and resource-constrained IoT devices, addressing computational and privacy challenges. In the SFL framework, IoT devices serve as clients and do not share raw client data with the server. Instead, they offload computationally intensive tasks to the server. However, an SFL-based IoT system faces three key challenges. First, it struggles to personalize client models when client data distributions are heterogeneous. Second, it encounters a trade-off between communication overhead and data privacy. Third, it suffers from severe performance degradation when label distributions shift after deploying the pre-trained model. To address these issues, we propose a novel early-exit SFL framework consisting of a pre-training phase and an online learning phase. In the pre-training phase, we introduce a personalized SFL training method to tailor each client model to its data distribution and a surrogate target generation method to train the server’s large model. In the online learning phase, client models are updated to handle label distribution shifts and maintain performance by leveraging the server’s large model as a teacher through knowledge distillation. For the real-time inference, early-exit is available by passing through only the client’s model. Extensive simulations demonstrate that the proposed framework achieves superior accuracy and lower communication costs compared to state-of-the-art methods. Notably, our method outperforms existing methods with average improvements of 18.49% in pre-training settings and 30.96% under diverse label distribution shift scenarios.

The Internet of Vehicles (IoV) is a network that connects various transportation devices, where autonomous vehicles play a crucial role. These vehicles must be able to coexist and drive alongside human-driven vehicles. To ensure a safer IoV, it is essential to develop adaptive decision-making capabilities, even under the uncertainty posed by human drivers. This study aims to develop an autonomous vehicle with the capacity for foresighted decision-making in complex multi-agent interactions by integrating predictive abilities of future observations. The proposed autonomous vehicle is equipped with episodic future thinking (EFT) capabilities. Specifically, EFT uses a prediction network to simulate future observations and guide adaptive decision-making of actor-critic networks based on envisioned scenarios. We use an offline reinforcement learning paradigm to train both the prediction network and the EFT-based actor-critic networks. To assess the generalized performance of the proposed solution, we integrate the EFT module with three popular offline reinforcement learning algorithms. We run an extensive series of performance evaluations across three driving scenarios involving multi-agent interactions and use four datasets for offline training, including a real-world dataset. Simulation results demonstrate that the proposed solution with the EFT module improves average performance in most cases compared to the baseline without the EFT module. Additionally, we compare the proposed solution with three existing model-based reinforcement learning approaches, confirming its superiority. Finally, we analyze driving behavior in terms of agility, safety, and stability.

The primary goal of an autonomous driving system is to achieve full autonomy by integrating the capability of adaptive decision-making across a wide range of driving scenarios. Despite recent advances in reinforcement learning (RL) enabling the development of policies for adaptive behaviors, current solutions are typically tailored for specific driving scenarios (e.g., highway, tollgate) rather than providing a scenario-free solution. To address this limitation, this study focuses on developing an autonomous driving policy that can operate across diverse driving scenarios through a generalized decision-making model. Specifically, an autonomous vehicle learns a unified multi-task policy by utilizing a shared replay buffer across all scenarios, thereby enhancing sample and learning efficiencies. Furthermore, we adopt an offline-to-online RL approach to leverage both the sample efficiency of offline RL and the performance improvements of online RL. The proposed solution involves an algorithmic shift aimed at maximizing the objectives of each RL method, incorporating three key techniques: Q re-initialization, Q adaptation, and policy variance re-initialization. To validate our solution, we compare its performance with existing RL methods and analyze driving behavior using objective-aware and safety-aware metrics. Our findings demonstrate that the proposed solution achieves superior performance across most metrics, irrespective of dataset quality.

Offline reinforcement learning has emerged as a promising technology by enhancing its practicality through the use of pre-collected large datasets. Despite its practical benefits, most algorithm development research in offline reinforcement learning still relies on game tasks with synthetic datasets. To address such limitations, this paper provides autonomous driving datasets and benchmarks for offline reinforcement learning research. We provide 19 datasets, including real-world human driver’s datasets, and seven popular offline reinforcement learning algorithms in three realistic driving scenarios. We also provide a unified decision-making process model that can operate effectively across different scenarios, serving as a reference framework in algorithm design. Our research lays the groundwork for further collaborations in the community to explore practical aspects of existing reinforcement learning methods. Dataset and codes can be found in https://sites.google.com/view/ad4rl.

Understanding cognitive processes in multi-agent interactions is a primary goal in cognitive science. It can guide the direction of artificial intelligence (AI) research toward social decision-making in multi-agent systems, which includes uncertainty from character heterogeneity. In this paper, we introduce *episodic future thinking (EFT) mechanism* for a reinforcement learning (RL) agent, inspired by the cognitive processes observed in animals. To enable future thinking functionality, we first develop a *multi-character policy* that captures diverse characters with an ensemble of heterogeneous policies. The *character* of an agent is defined as a different weight combination on reward components, representing distinct behavioral preferences. The future thinking agent collects observation-action trajectories of the target agents and leverages the pre-trained multi-character policy to infer their characters. Once the character is inferred, the agent predicts the upcoming actions of target agents and simulates the potential future scenario. This capability allows the agent to adaptively select the optimal action, considering the predicted future scenario in multi-agent scenarios. To evaluate the proposed mechanism, we consider the multi-agent autonomous driving scenario in which autonomous vehicles with different driving traits are on the road. Simulation results demonstrate that the EFT mechanism with accurate character inference leads to a higher reward than existing multi-agent solutions. We also confirm that the effect of reward improvement remains valid across societies with different levels of character diversity.

The mathematical modeling of infectious diseases aims to evaluate the transmissibility of the on-going spread of disease and guide the government's control strategies and interventions. In this paper, we propose a novel transmissibility indicator, reproduction factor, which evaluates the number of secondary infections from a single nonisolated infectious individual. In contrast to classic reproduction numbers, the reproduction factor explicitly considers the fraction of susceptible individuals (who are not immune to disease naturally or through vaccination) and the nonisolated population to evaluate near real-time transmissibility. Thus, it can be an effective indicator when the spread of disease has progressed and control strategies have been implemented. Other merits of the proposed reproduction factor include data-driven inference based on a Markov chain, which enables the inference of latent information, such as the number of nondetected infectious individuals and the number of daily new infections. We performed an extensive simulation using the COVID-19 datasets of Germany, Italy, South Korea, and California (the U.S.) to verify our model. We further compared the results with other transmissibility measures, including reproduction numbers, and the results of state-of-the-art epidemic models. Through the results, we confirmed that the proposed reproduction factor and corresponding inference model explained the COVID-19 datasets.

With the development of the Internet and networks, various types of network data have been shared. Intelligent and various cyber attacks are continuously increasing as the amount of network data increases rapidly. Although anomaly detection based on autoencoders worked actively, autoencoder has a limitation that it cannot utilize hidden space and detects abnormal data with various anomalies only at one point. We propose a step-by-step anomaly detection using the hidden space of the autoencoder. The proposed system improves the performance of anomaly detection by utilizing hidden spaces. The pre-detection rate of the abnormal data is 20%, enabling proactive response. The total detection rate of the abnormal data is 99%, which outperforms other existing solutions.

We propose a partial federated learning based network intrusion system to utilize the limited communication resources of mobile devices. The key to our algorithm is that we share only part of the model to enable efficient communication and strengthen data privacy. The proposed method is evaluated using two network traffic datasets and three performance measures. Our simulation results confirm that the model can reach the performance of centralized learning, although the ratio of a shared layer is reduced up to 25%.

With the increasing amount of Internet traffic, a significant number of network intrusion events have recently been reported. In this letter, we propose a network intrusion detection system that enables hierarchical detection based on self-supervised learning. The proposed solution consists of multiple stages of detection, including the early detection of extreme outliers, which may cause severe damage to the system. Furthermore, it performs thorough reexaminations using the hidden spaces with specialized anomaly scores, which leads to high detection accuracy. Extensive simulation results confirm that the proposed solution can preemptively detect 20$\%$of abnormal data, thereby enabling a proactive response, and can detect 99$\%$of abnormal data at the final stage.

A fundamental question in neuroscience is how the brain creates an internal model of the world to guide actions using sequences of ambiguous sensory information. This is naturally formulated as a reinforcement learning problem under partial observations, where an agent must estimate relevant latent variables in the world from its evidence, anticipate possible future states, and choose actions that optimize total expected reward. This problem can be solved by control theory, which allows us to find the optimal actions for a given system dynamics and objective function. However, animals often appear to behave suboptimally. Why? We hypothesize that animals have their own flawed internal model of the world, and choose actions with the highest expected subjective reward according to that flawed model. We describe this behavior as {\it rational} but not optimal. The problem of Inverse Rational Control (IRC) aims to identify which internal model would best explain an agent's actions. Our contribution here generalizes past work on Inverse Rational Control which solved this problem for discrete control in partially observable Markov decision processes. Here we accommodate continuous nonlinear dynamics and continuous actions, and impute sensory observations corrupted by unknown noise that is private to the animal. We first build an optimal Bayesian agent that learns an optimal policy generalized over the entire model space of dynamics and subjective rewards using deep reinforcement learning. Crucially, this allows us to compute a likelihood over models for experimentally observable action trajectories acquired from a suboptimal agent. We then find the model parameters that maximize the likelihood using gradient ascent. Our method successfully recovers the true model of rational agents. This approach provides a foundation for interpreting the behavioral and neural dynamics of animal brains during complex tasks.

In this paper, we aim to find a robust network formation strategy that can adaptively evolve the network topology against network dynamics in a distributed manner. We consider a network coding deployed wireless ad hoc network where source nodes are connected to terminal nodes with the help of intermediate nodes. We show that mixing operations in network coding can induce packet anonymity that allows the inter-connections in a network to be decoupled. This enables each intermediate node to consider complex network inter-connections as a node-environment interaction such that the Markov decision process (MDP) can be employed at each intermediate node. The optimal policy that can be obtained by solving the MDP provides each node with the optimal amount of changes in transmission range given network dynamics (e.g., the number of nodes in the range and channel condition). Hence, the network can be adaptively and optimally evolved by responding to the network dynamics. The proposed strategy is used to maximize long-term utility, which is achieved by considering both current network conditions and future network dynamics. We define the utility of an action to include network throughput gain and the cost of transmission power. We show that the resulting network of the proposed strategy eventually converges to stationary networks, which maintain the states of the nodes. Moreover, we propose to determine initial transmission ranges and initial network topology that can expedite the convergence of the proposed algorithm. Our simulation results confirm that the proposed strategy builds a network which adaptively changes its topology in the presence of network dynamics. Moreover, the proposed strategy outperforms existing strategies in terms of system goodput and successful connectivity ratio.

In this paper, we propose a distributed solution based on game-theoretic approaches to the topology formation problem for mobile wireless sensor networks with multi-source multicast flows. Our solution significantly reduces computational complexity by taking advantage of network coding. Finding an optimal topology for network coding in multi-source multicast flows is NP-hard problem, so the proposed algorithm provides a suboptimal solution with low computational complexity. We formulate the problem of distributed network topology formation as a network formation game by considering the nodes in the network as players that can take actions for making outgoing links. The proposed game, which consists of multiple players and multicast flows, can be decomposed into independent link formation games played by only two players with a unicast flow. The proposed algorithm is also guaranteed to converge, i.e., a stable network topology can be always formed. Our simulation results confirm that the computational complexity of the proposed solution is low enough for practical deployment in large-scale mobile, wireless sensor networks.

In this letter, we propose a distributed network formation solution for network coding deployed wireless networks which includes multisource multicast flows. This is an attempt to solve an open problem of network coding based multisource multicast flow design based on a game theoretic approach, which can eventually form a network in a distributed way. The network is in particular constructed by individual decision makings of the nodes, while taking advantages of network coding techniques. The decisions made by the nodes include the transmission powers and the use of network coding operations. In each stage game, nodes update the parameters based on feedbacks such as rewards, penalties, and evaluate their prior actions, which enables the nodes to make best responses in the next stage game. Our simulations confirm that the resulting network can reduce overall power consumption compared to direct transmission, and improve system throughput with less power consumption compared to no coding strategy.

In this paper, we consider a delay-sensitive data transmission strategy based on network coding technique in finite fields over error-prone networks. In order to solve all-or-nothing problem inherited from network coding, compressed network coding is proposed by jointly considering network coding techniques and compressed sensing technique. While network coding techniques have been jointly used with the compressed sensing techniques, network coding operations are performed in the field of real numbers, and thus, the payload of transmitted data can be enlarged as the data traverse more hops in networks. In this paper, however, we propose to use network coding techniques in finite fields, such that the size of payload does not increase as more hops are traversed. With the help of compressed sensing technique, a destination node is able to approximately recover the source data based on l1-norm minimization approach, in case of innovative packet loss. It is analytically shown that the payload size of the proposed approach is always smaller than that of the conventional approach, while the proposed approach can achieve comparable decoding performances. We evaluate the effectiveness of the proposed approach based on an illustrative application of image delivery system.

This paper proposes a robust decoding algorithm in delivery of network coded data which is in particular correlated and delay-sensitive. We consider ad-hoc sensor network topologies, where a correlated data is delivered based on network coding techniques in conjunction with approximate decoding algorithm in order for efficient and robust data delivery. The approximate decoding algorithm has been developed as a decoding solution to ill-posed problems for network coded correlated data sources. In this paper, we improve the performance of approximate decoding algorithm by explicitly considering more information, which is used to additionally refine the recovered data. The information includes potential results that are from finite field operations and the set of such information is referred to as position information matrix in this paper. We deploy the position information matrix into approximate decoding algorithm and investigate its corresponding properties. We then analytically show that this improves the performance of approximate decoding algorithm. Our simulation results confirm the properties of the proposed approximate decoding algorithm with position information matrix and improved performance.