Recent popularization of the Internet of Vehicles (IoVs) and vehicles-to-everything (V2X) enables the emergence of real-time vehicular applications, posing challenges to resourcelimited vehicles. Toward this end, vehicle edge computing (VEC) has been proposed to alleviate the computational burden on vehicles by leveraging resources from roadside units (RSUs) and VEC servers. While existing works mainly focus on the task requirement for either vehicles or RSUs, the joint task offloading for both V2X and RSUs-to-everything (R2X) has not been fully studied. In this paper, we aim at optimizing the task offloading strategies for both vehicles and RSUs, and adopt a multi-hop task offloading manner to fully utilize the VEC network resources. This problem introduces a severe state-action space shift issue with varying dimensions and representation, which poses challenges for conventional DRL approaches. To address it, we propose a Bidirectional Encoder Representations from Transformers (Bert)-based matching Q-network (BMQN) algorithm. First, we design the BMQN model to efficiently capture correlations among all vehicles and RSUs through bidirectional attention. Then, we propose type-embedded grouped attention and available action embedding to mitigate the overfitting sequence length issue, thereby enhancing generalization capacity. Moreover, we propose to address the state-action space shift issue through a matching-based manner, which can significantly enhance the task offloading ability by matching the states among devices. Simulation results demonstrate that: 1) the BMQN can achieve much better performance than other approaches in scenarios comprising various numbers of vehicles and RSUs as well as diverse road lengths; 2) the BMQN has sufficient generalization capacity to adapt to inexperienced scenarios through matching-based architecture and available action embedding.

Industrial Cyber-Physical Systems (ICPS) have emerged as a critical component in the industrial domain. To facilitate seamless collaboration among massive devices, cloud-edge computing architectures have emerged as a key enabler for ICPS, leveraging distributed intelligence to orchestrate devices and computational tasks. In cloud-edge computing, efficient task offloading and resource management are essential for optimizing task performance and reducing energy costs. However, conventional centralized resource management strategies struggle to satisfy the real-time, adaptability, and performance demands of dynamic ICPS systems. Industrial Cyber-Physical Systems (ICPS) have emerged as a critical component in the industrial domain. To facilitate seamless collaboration among massive devices, cloud-edge computing architectures have emerged as a key enabler for ICPS, leveraging distributed intelligence to orchestrate devices and computational tasks. In cloud-edge computing, efficient task offloading and resource management are essential for optimizing task performance and reducing energy costs. However, conventional centralized resource management strategies struggle to satisfy the real-time, adaptability, and performance demands of dynamic ICPS systems. In this paper, we propose the Distributed Transformer-based Actor-Critic (DTAC) algorithm to jointly determine task offloading and resource management decisions in cloud-edge computing networks, particularly for delay-sensitive applications in ICPS. The DTAC algorithm integrates the powerful transformer model with the popular actor-critic architecture to address the challenge of a hybrid high-dimensional action space. We first train a centralized model to learn coordination among user equipments (UEs) and then introduce a decentralized transfer learning (TL) approach to efficiently adapt the centralized model into the DTAC framework. Using the DTAC model, each UE can independently manage its local resources based solely on local information, avoiding the significant signaling overhead inherent in centralized approaches. Simulation results demonstrate that DTAC not only outperforms other MARL and TL schemes in both small-and large-scale scenarios, but also exhibits strong generalization capabilities in inexperienced settings. Furthermore, DTAC and decentralized TL approaches significantly reduce training costs by 73% compared to other methods, making them more practical for ICPS deployment.

Next-generation multiple access techniques are crucial for providing low-latency and highly efficient data transmission services. Recently, Deep Reinforcement Learning (DRL) has emerged as a prevalent approach in the multiple access domain, aiming to facilitate user coordination and enhance transmission efficiency. However, current DRL approaches face challenges, including limited generalization ability, low sample efficiency, and the complexities associated with Partially Observable Markov Decision Processes (POMDP), which hinder their application in heterogeneous networks with varying numbers of nodes and configurations. In this paper, we propose a foundation model-based multiple access (FMA) algorithm. To address severe POMDP and sample inefficiency issues, we decompose the multiple access problem into two parts: a transmission decision part and a configuration estimation part. We leverage the strong generalization and inference capabilities of the foundation model, utilizing a Deep Learning (DL) approach instead of DRL for training, and adopt the Low-Rank Adaptation (LoRA) technique to fine-tune the foundation model for downstream multiple access tasks. Simulation results demonstrate that: 1) through the decomposition, the FMA approach exhibits sufficient generalization and inference abilities to adapt to various scenarios with various protocols, configurations, and numbers of heterogeneous nodes; 2) by incorporating expert knowledge, the FMA approach can significantly enhance network performance while ensuring certain fairness requirement for heterogeneous nodes.

Mobile-edge Computing (MEC) has now emerged as a complement to cloud computing, providing computational capacity for the resources-constrained edge devices. Recently, intelligent computation offloading and cache placement stands as effective approaches to enhance the performance of dynamic MEC networks. In this paper, we propose an online centralized joint resource management approach, named Transformer-based Actor-Critic (TAC), to minimize the task execution time subject to resource constraints. We decouple this mixed-integer non-linear programming (MINLP) problem into a non-convex offloading decision part and a convex joint resources allocation part, and propose the TAC approach to address the non-convex task offloading problem with low computational complexity. In the joint resources management problem, the high-dimensional state-action space is addressed by the transformer-based actor-critic architecture. Through the proposed TAC, the joint cache, communication and computation resource management can be obtained without the knowledge of future task arrivals. Simulation results demonstrate that the TAC can save 48.4% average task execution time with only 2.3% additional computation delay compared to Random with lowest computational complexity. In particular, it further demonstrates great generalization ability to enhance the performance in untrained scenarios.

With the development of wireless communication systems, the large-scale deployment of Internet of Things (IoT) devices becomes popular. Due to limited energy, the multiple access approaches without carrier sensing requirement are widely deployed in IoT devices, including Aloha and time-division multiple access (TDMA). However, these approaches encounter the transmission inefficiency issue, especially in dynamic heterogeneous networks comprising nodes with diverse protocols and varying numbers and transmission configurations over time. In this article, combining curriculum learning (CL) and multitask reinforcement learning (MTRL), we propose the curriculum multitask heterogeneous-agent proximal policy optimization (CMHA) algorithm to improve the throughput performance while guaranteeing fairness in dynamic heterogeneous networks. We introduce the elastic weight consolidation (EWC) in the CMHA to further enhance generalization capacity, which can better address the challenging MTRL problem in dynamic heterogeneous networks. Combining the monotonic improvement feature of heterogeneous-agent proximal policy optimization (HAPPO) and the generalization capacity of EWC, the proposed CMHA can achieve a nearly monotonic improvement in all possible scenarios. The simulations show that the CMHAalgorithm 1) has sufficient generalization capacity for massive scenarios in dynamic heterogeneous networks; 2) can significantly enhance the network throughput; and 3) can guarantee the fairness of both agents and heterogeneous nodes.

In the IEEE 802.11ax Wireless Local Area Networks (WLANs), Orthogonal Frequency Division Multiple Access (OFDMA) has been applied to enable the high-throughput WLAN amendment. However, with the growth of the number of devices, it is difficult for the Access Point (AP) to schedule uplink transmissions, which calls for an efficient access mechanism in the OFDMA uplink system. Based on Multi-Agent Proximal Policy Optimization (MAPPO), we propose a Mean-Field Multi-Agent Proximal Policy Optimization (MFMAPPO) algorithm to improve the throughput and guarantee the fairness. Motivated by the Mean-Field games (MFGs) theory, a novel global state and action design are proposed to ensure the convergence of MFMAPPO in the massive access scenario. The Multi-Critic Single-Policy (MCSP) architecture is deployed in the proposed MFMAPPO so that each agent can learn the optimal channel access strategy to improve the throughput while satisfying fairness requirement. Extensive simulation experiments are performed to show that the MFMAPPO algorithm 1) has low computational complexity that increases linearly with respect to the number of stations 2) achieves nearly optimal throughput and fairness performance in the massive access scenario, 3) can adapt to various diverse and dynamic traffic conditions without retraining, as well as the traffic condition different from training traffic.