Data collection and distribution through crowdsensing has become an emerging trend in smart city scenarios. By leveraging existing vehicle resources without deploying dedicated infrastructure, Vehicular CrowdSensing (VCS) provides low-cost and high-mobility data collection on road networks. Typically, the Crowdsensing Platform (CP) issues data collection tasks, recruits Sensing Vehicles (SVs) to complete tasks, and sells the collected data to Data Demanders (DDs). Here, the goal of CP is to maximize profits through data collection and sales, and the goal of DDs is to improve satisfaction by purchasing high-quality sensing data. It can be seen that both CP and DD hope that SVs can complete more sensing tasks at a limited cost (high efficiency) while ensuring the accuracy of data collection (high quality). However, due to individual rationality and selfishness, not all SVs are willing to complete the sensing task. Therefore, how to motivate SVs to complete sensing tasks with high quality and efficiency, while handling the relationship among CP, DDs, and SVs, is a problem that needs to be considered. To solve the above problems, this paper proposes a Reputation-based Hybrid Incentive Approach (RHIA), with the goal of maximizing the utility of CP, SVs, and DDs. Specifically, in order to improve the task completion quality of SVs, we introduce vehicle reputation to measure SVs. Then, we propose a one-to-one bargaining game between CP and each SV, and use the reputation value as the sequential basis of the game. Meanwhile, in order to improve the task completion efficiency of SVs, we also design a unique SV Trajectory Planning Algorithm (STPA). Further, in order to meet the needs of DDs, a one-to- multi Stackelberg game between CP and DDs is proposed. Here, the existence and uniqueness of Nash equilibrium is proved through backward induction. Finally, based on real-world datasets, the effectiveness of our proposed RHIA and STPA is verified. Our proposed method can ensure the long-term stability of the VCS system, which also improves the utility of participating individuals.

This paper investigates dynamic model partitioning and resource allocation in split federated learning, aiming to maximize the utility of clients and the Central Server (CS). We first model the interactions between the CS and clients as a Stackelberg game, where the CS acts as the leader to set payment and allocate computation resources, while clients as followers to determine model partitioning strategies. Then, we transform the problem into a bi-level optimization and propose a Nash-Equilibrium-based Stackelberg Algorithm (NESA) to solve it. Finally, the experimental results indicate that a Stackelberg equilibrium exists between the CS and clients, and NESA achieves higher utility and improves accuracy and convergence speed.

The Internet of Vehicles (IoV) has witnessed a significant growth in the number of participants. This rapid expansion has increased demands for computing resources and quality of service (QoS), posing challenges for mobile edge computing (MEC) in the IoV domain. Efficiently allocating computing power to meet these service demands has become a crucial concern. Therefore, joint optimization of offloading decisions and power allocation is required to achieve the tradeoff between task latency and energy consumption. To address the above challenge, we propose a multi-agent reinforcement learning (MARL) method called multi-agent twin delayed deep deterministic policy gradient (MA-TD3) in this paper. Compared to its predecessor, multi-agent deep deterministic policy gradient (MADDPG), this algorithm improves performance and execution speed. It solves the slow convergence problem caused by Q-value overestimation and reduces the computational cost. The experimental results illustrate that the proposed algorithm reaches an observable performance improvement.

Edge caching is a promising technology to reduce backhaul strain and content access delay in Internet of Vehicles (IoV). It precaches frequently used contents close to vehicles through intermediate roadside units. Previous edge caching works often assume that content popularity is known in advance or obeys simplified models. However, such assumptions are unrealistic, as content popularity varies with uncertain spatial-temporal traffic demands in IoVs. Federated learning (FL) enables vehicles to predict popular content with distributed training. It preserves the training data remain local, thereby addressing privacy concerns and communication resource shortages. This article investigates a mobility-aware edge caching strategy by exploiting asynchronous FL and deep reinforcement learning (DRL). We first implement a novel asynchronous FL framework for local updates and global aggregation of stacked autoencoder (SAE) models. Then, utilizing the latent features extracted by the trained SAE model, we adopt a hybrid filtering model for predicting and recommending popular content. Furthermore, we explore intelligent caching decisions after content prediction. Based on the formulated Markov decision process (MDP) problem, we propose a DRL-based solution, and adopt neural network-based parameter approximations for the curse of dimensionality in RL. Extensive simulations are conducted based on real-world data trajectory. Especially, our proposed method outperforms federated averaging, least recently used, and NoDRL, and the edge hit rate is improved by roughly 6%, 21%, and 15%, respectively, when the cache capacity reaches 350 MB.

Due to vehicle mobility, the topology of Vehicle Ad-hoc Networks (VANETs) may change dynamically. High mobility, limited bandwidth, and dynamic network topology pose challenges for communication in the Internet of Vehicles (IoVs). Literature works have attempted to promote efficient (e.g., lower end-to-end latency) message forwarding. However, due to the uncertain direction of message forwarding and vehicle mobility, they suffer from unreachable destinations and unstable connections. This paper explores the efficient method of message forwarding to alleviate network congestion in IoVs. We propose a Spatio-Temporal domain Autonomous Load Balancing (STALB) routing protocol. Specifically, STALB is a trajectory-based method for controlling the direction of message forwarding. STALB can significantly reduce the end-to-end latency and overload ratio, since it considers the local status of network relay devices (i.e., buffer score, congestion status) from the spatio-temporal domain. Then, we present a path reconstruction mechanism, which ensures that messages are forwarded to destinations within limited Time-To-live (TTL). Extensive simulation results show that STALB significantly outperforms other baseline methods (BSaW, TDOR, and TBHGR) regarding overhead ratio, average delivery latency, and average buffer time. Especially, the delivery rate of STALB can reach 99.9% under the sparse network scenario (4,500 messages), at least 0.7% higher than other baseline methods. Similarly, the average delivery delay of STALB is at least 84.31% lower than that of other baseline methods under the dense network scenario (18,000 messages).

Edge caching is a promising approach to reduce duplicate content transmission in Internet-of-Vehicles (IoVs). Several Reinforcement Learning (RL) based edge caching methods have been proposed to improve the resource utilization and reduce the backhaul traffic load. However, they only obtain the local sub-optimal solution, as they neglect the influence from environments by other agents. This paper investigates the edge caching strategies with consideration of the content delivery and cache replacement by exploiting the distributed Multi-Agent Reinforcement Learning (MARL). A hierarchical edge caching architecture for IoVs is proposed and the corresponding problem is formulated with the goal to minimize the long-term content access cost in the system. Then, we extend the Markov Decision Process (MDP) in the single agent RL to the context of a multi-agent system, and tackle the corresponding combinatorial multi-armed bandit problem based on the framework of a stochastic game. Specifically, we firstly propose a Distributed MARL-based Edge caching method (DMRE), where each agent can adaptively learn its best behaviour in conjunction with other agents for intelligent caching. Meanwhile, we attempt to reduce the computation complexity of DMRE by parameter approximation, which legitimately simplifies the training targets. However, DMRE is enabled to represent and update the parameter by creating a lookup table, essentially a tabular-based method, which generally performs inefficiently in large-scale scenarios. To circumvent the issue and make more expressive parametric models, we incorporate the technical advantage of the Deep-$Q$Network into DMRE, and further develop a computationally efficient method (DeepDMRE) with neural network-based Nash equilibria approximation. Extensive simulations are conducted to verify the effectiveness of the proposed methods. Especially, DeepDMRE outperforms DMRE,$Q$-learning, LFU, and LRU, and the edge hit rate is improved by roughly 5%, 19%, 40%, and 35%, respectively, when the cache capacity reaches 1, 000 MB.

Mobile-edge computing (MEC) has emerged as a promising computing paradigm in the 5G architecture, which can empower user equipments (UEs) with computation and energy resources offered by migrating workloads from UEs to the nearby MEC servers. Although the issues of computation offloading and resource allocation in MEC have been studied with different optimization objectives, they mainly focus on facilitating the performance in the quasistatic system, and seldomly consider time-varying system conditions in the time domain. In this article, we investigate the joint optimization of computation offloading and resource allocation in a dynamic multiuser MEC system. Our objective is to minimize the energy consumption of the entire MEC system, by considering the delay constraint as well as the uncertain resource requirements of heterogeneous computation tasks. We formulate the problem as a mixed-integer nonlinear programming (MINLP) problem, and propose a value iteration-based reinforcement learning (RL) method, named$Q$-Learning, to determine the joint policy of computation offloading and resource allocation. To avoid the curse of dimensionality, we further propose a double deep$Q$network (DDQN)-based method, which can efficiently approximate the value function of$Q$-learning. The simulation results demonstrate that the proposed methods significantly outperform other baseline methods in different scenarios, except the exhaustion method. Especially, the proposed DDQN-based method achieves very close performance with the exhaustion method, and can significantly reduce the average of 20%, 35%, and 53% energy consumption compared with offloading decision, local first method, and offloading first method, respectively, when the number of UEs is 5.

Mobile Edge Computing (MEC) has emerged as a promising computing paradigm in 5G networks, which can empower User Equipments (UEs) with computation and energy resources offered by migrating workloads from the UEs to the MEC servers. Although the issues of computation offloading and resource allocation in MEC have been studied with different optimization objectives, they mainly investigate quasi-static system environments, without considering the different resource requirements and time-varying system conditions in a dynamic system. In this paper, we exploit a multi-user MEC system, and investigate the task execution scheme for dynamic joint optimization of offloading decision and resource assignment. Our objective is to minimize the energy consumption of all UEs, with considering the delay constraint as well as the dynamic resource requirements of heterogeneous computation tasks. Accordingly, we formulate the problem as a mixed integer non-linear programming problem (MINLP), and propose a value iteration based Reinforcement Learning (RL) approach, named Q-Learning, to obtain the optimal policy of computation offloading and resource allocation. Simulation results demonstrate that the proposed approach can significantly decrease UEs' energy consumption in different scenarios, compared with other baseline methods.

Edge caching has been emerged as a promising solution to alleviate the redundant traffic and the content access latency in the future Internet of Vehicles (IoVs). Several Reinforcement Learning (RL) based edge caching methods have been proposed to improve the cache utilization and reduce the backhaul traffic load. However, they can only obtain the local sub-optimal solution, as they neglect the influence of environment by other agents. In this paper, we investigate the edge caching strategy with consideration of the content delivery and cache replacement by exploiting the distributed Multi-Agent Reinforcement Learning (MARL). We first propose a hierarchical edge caching architecture for IoVs and formulate the corresponding problem with the objective to minimize the long-term cost of content delivery in the system. Then, we extend the Markov Decision Process (MDP) in the single agent RL to the multi-agent system, and propose a distributed MARL based edge caching algorithm to tackle the optimization problem. Finally, extensive simulations are conducted to evaluate the performance of the proposed distributed MARL based edge caching method. The simulation results show that the proposed MARL based edge caching method significantly outperforms other benchmark methods in terms of the total content access cost, edge hit rate and average delay. Especially, our proposed method greatly reduces an average of 32% total content access cost compared with the conventional RL based edge caching methods.