Mobile edge computing (MEC) is a promising paradigm that provides abundant computation and storage resources at the edge close to mobile devices (MDs). In MEC networks, MDs offload compute-heavy tasks to nearby edge servers (ESs) for delay-sensitive processing, where relevant services are stored to support task execution. However, the limited computation and storage capacities of ESs make joint optimization of service caching and computation offloading challenging due to coupled decisions, a large solution space, and dynamic environments. In this paper, we investigate the joint optimization of service caching and computation offloading in MEC networks, aiming to maximize the cache hit ratio and minimize the average service latency. To tackle this problem, the original formulation is decomposed into two hierarchical subproblems, namely high-level service caching and low-level computation offloading. We propose a novel hierarchical deep reinforcement learning (DRL) algorithm with active inference, termed HADRL. At the high-level, we adopt a deep deterministic policy gradient (DDPG) based DRL approach to maximize the cache hit ratio. At the low-level, we employ an active inference based DRL approach to minimize the average service latency. Unlike conventional DRL, the active inference based DRL approach selects policies by minimizing expected free energy instead of relying only on explicit rewards, making it well suited for highly dynamic low-level computation offloading. According to the simulation outcomes, the HADRL scheme surpasses the benchmark algorithms with respect to cache hit ratio as well as average service latency.

Intelligent transportation systems demand efficient resource allocation and task offloading to ensure low-latency, high-bandwidth vehicular services. The dynamic nature of vehicular environments, characterized by high mobility and extensive interactions among vehicles, necessitates considering time-varying statistical regularities, especially in scenarios with sharp variations. Despite the widespread use of traditional reinforcement learning for resource allocation, its limitations in generalization and interpretability are evident. To overcome these challenges, we propose an Intelligence-based Reinforcement Learning (IRL) algorithm. This algorithm utilizes active inference to infer the real world and maintain an internal model by minimizing free energy. Enhancing the efficiency of active inference, we incorporate prior knowledge as macro guidance, ensuring more accurate and efficient training. By constructing an intelligence-based model, we eliminate the need for designing reward functions, aligning better with human thinking, and providing a method to reflect the learning, information transmission and intelligence accumulation processes. This approach also allows for quantifying intelligence to a certain extent. Considering the dynamic and uncertain nature of vehicular scenarios, we apply the IRL algorithm to environments with constantly changing parameters. Extensive simulations confirm the effectiveness of IRL, significantly improving the generalization and interpretability of intelligent models in vehicular networks.

The rapid advancement of intelligent transportation systems necessitates efficient resource allocation for low-latency and high-bandwidth vehicular services. While traditional reinforcement learning has been widely utilized for resource allocation, it suffers from limitations such as poor generalization and interpretability. To overcome these challenges, we propose a novel Intelligence-based Reinforcement Learning (IRL) al-gorithm, which uses active inference to infer the real world and maintain an internal model of the world by minimizing free energy. We address the inefficiency of active inference by incorporating prior knowledge as macro guidance, ensuring more accurate and efficient training. By constructing the intelligence-based model, we eliminate the need for designing reward functions, which aligns better with human thinking and provides a method to reflect the learning, information transmission, and intelligence accumulation processes. Considering the dynamic and uncertain nature of vehicular scenarios, we apply the IRL algorithm to continuously evolving environments where environmental parameters are not fixed. Extensive simulations confirm the effectiveness of IRL, significantly enhancing the generalization and interpretability of intelligent models.

“Read-write-own” based Web3 has been proposed as a promising user-centric Internet to open the new generation of the World Wide Web, where Web3 users can independently manage data and derive value from creating content without relying on intermediaries. Connected and autonomous vehicles (CAVs) in Web3 can trade models in a self-controlled and decentralized credible way, which is a fundamentally and principally innovation based on novel architecture. Effectively implementing such paradigms involves proper model trading strategies. However, reinforcement learning (RL)-based strategies face challenges of poor generalization ability, low feasibility, and the exploration-exploitation dilemma. It is also difficult to define an explicit and appropriate reward function. Therefore, in this paper, we propose an intelligence-based reinforcement learning (IRL) approach for CAVs in Web3. We present a framework to enable model transactions between CAVs. Also, we provide a decentralized identifier (DID)-based identity management system for resource description and data verification to access Web3, followed by the mechanism and supporting smart contracts. Furthermore, we formulate the model trading issue as an active inference to form higher-level cognition about the environment without rewards. Then we use IRL to solve it. And we use “intelligence”, a high-level indicator, to quantify the efficiency of such cognition. It can evaluate the difference between the predicted state and the real state in policy exploration. The proposed scheme shows good generalization and can auto-balance exploration and exploitation, simultaneously achieving outperforming performance on the model trading issue with no rewards. In simulations, the performance of the proposed scheme is compared with existing methods.

With the rapid development of intelligent transportation systems, there is an increasingly strong demand for low-latency and high-bandwidth vehicular services. Unmanned aerial vehicles (UAVs) can be used as a supplement to the ground networks, to relieve the communication pressure on ground facilities, such as base stations. In this paper, we use multiple UAVs to provide services for vehicles and model the multi-UAV scenario as a collaborative multi-agent system. In addition, we take vehicle safety as the top priority and the delay requirement as the constraints. Then we exploit the Lagrange multiplier to combine the constraint function and cost function, so as to reduce the resource consumption as much as possible on the premise of ensuring the safety of the vehicles. The influence of spectrum efficiency and computing power should also be taken into account when allocating resources. We adopt the multi-agent reinforcement learning to train the UAVs, and meanwhile introduce the attention mechanism so that each UAV can optimize itself better with the information of other UAVs. Through extensive simulations, the effectiveness of our proposed method is verified. Particularly, the limited resources can allocated efficiently according to the vehicle’s needs under the premise of ensuring vehicle safety.

In recent years, multi-access edge computing (MEC) is emerging to provide computation and storage resources to the Internet of things (IoT) devices to assist them in high-performance demanding tasks. Real-time task requests from the IoT devices often have strict delay constraints. However, in practice, the delay requirements of task requests often fail to be satisfied because of the inappropriate computation processing method and inefficient resource allocation in MEC networks. In this article, we present a novel framework for MEC networks with unmanned aerial vehicles (UAVs) and intelligent reflecting surfaces (IRSs) to facilitate computation offloading with delay constraints. In addition, we propose a novel constrained reinforcement learning method with a dynamic balance mechanism named$Q_{c}-DQN$. Finally, we conduct extensive simulations to verify the effectiveness of our proposed method. Compared to the benchmark schemes, our scheme not only improves the overall network performance and reduces the task completion time, but also meets the delay constraints.

With the rapid development of intelligent transportation systems, there is an increasing demand for a variety of vehicular services, such as automated driving assistance, emergency alert, infotainment, etc. However, in some situations (e.g., remote areas or maritime scenarios), the terrestrial networks alone cannot serve the vehicular applications very well due to the infrastructure deployment and maintenance issues. Satellite networks have become an effective supplement to terrestrial networks, which complement well in terms of coverage, flexibility, reliability, and availability. In this paper, we consider the low orbit multi-beam satellite-terrestrial networks to serve for vehicles. We model this problem as a cooperative multi-agent reinforcement learning process, where each beam acts as an agent, and the global bandwidth is cooperatively shared among all the agents. A multi-agent actor-critic method with attention mechanism is proposed to allocate resources for vehicles with strict delay requirements and minimum bandwidth consumption. When allocating bandwidth, the channel efficiency, the angle of the beams and the priorities of requests in different regions are also considered. Centralized training and distributed execution is performed in the training of the agents. Extensive simulation results verify the effectiveness of our proposed method, where all the agents can well cooperative to achieve efficient resource allocation on-demand for the vehicles under strictly limited bandwidth resources.

With the rapid development of vehicular networks, there is an increasing demand for extensive networking, computing and caching resources. In fact, vehicular networks are nonstationary, and how to allocate multiple resources effectively and efficiently for dynamic vehicular networks is extremely important, however, really challenging. In this paper, we propose a general framework that can enable fast-adaptive edge resource allocation for dynamic vehicular environment. Specifically, we model the dynamics of the vehicular environment as a series of related Markov Decision Processes (MDPs). We combine hierarchical reinforcement learning with meta learning, which makes our proposed framework available to quickly adapt to a new environment by only fine-tuning the top-level master network, and meanwhile the low-level sub-networks can make the right resource allocation policy. The extensive simulation results show the effectiveness of our proposed framework, which can quickly adapt to different scenarios. This is consistent with the real-world situations and can significantly improve the performance of resource allocation in dynamic vehicular networks.

With the exponential growth in the number of Internet-of-Things (IoT) devices, the cloud-centric computing paradigm can hardly meet the increasingly high requirements for low latency, high bandwidth, ease of availability, and more intelligent services. Therefore, a distributed and decentralized computing architecture is imperative, where edge-centric computing, such as fog computing and mist computing, has been recently proposed. Edge-centric computing resources can be managed locally and personally rather than being administered by a remote centralized third party. However, security and privacy issues are the main challenges due to the absence of trust between the IoT devices and edge computing nodes (ECNs). A blockchain, as a decentralized, trustless, and immutable public ledger, can well solve the trust-absence issue. In this article, we first elaborate on the security and privacy issues of edge-computing-enabled IoT, and then present the key characteristics of blockchains, which make blockchains well suited for the edge-centric IoT scenarios. Furthermore, we propose a general framework for blockchain-based edge-computing-enabled IoT scenarios that specifies the step-by-step procedure of a single transaction between an IoT end and an ECN. In addition, we design a smart contract within a private blockchain network that exploits the state-of-the-art machine learning algorithm, asynchronous advantage actor-critic (A3C), to allocate the edge computing resources, which exemplifies how artificial intelligence (AI) can be combined with blockchains. We further discuss the benefits of the convergence of AI and blockchains. Finally, simulation results are presented.