Federated learning (FL) in multi-service provider (SP) ecosystems is fundamentally hampered by non-cooperative dynamics, where privacy constraints and competing interests preclude the centralized optimization of multi-SP communication and computation resources. In this paper, we introducePAC-MCoFL, a game-theoretic multi-agent reinforcement learning (MARL) framework where SPs act as agents to jointly optimize client assignment, adaptive quantization, and resource allocation. Within the framework, we integrate Pareto Actor-Critic (PAC) principles with expectile regression, enabling agents to conjecture optimal joint policies to achieve Pareto-optimal equilibria while modeling heterogeneous risk profiles. To manage the high-dimensional action space, we devise a ternary Cartesian decomposition (TCAD) mechanism that facilitates fine-grained control. Further, we developPAC-MCoFL-p, a scalable variant featuring a parameterized conjecture generator that substantially reduces computational complexity with a provably bounded error. Alongside theoretical convergence guarantees, our framework's superiority is validated through extensive simulations –PAC-MCoFLachieves approximately$5.8\%$and$4.2\%$improvements in total reward and hypervolume indicator (HVI), respectively, over the latest MARL solutions. The results also demonstrate that our method can more effectively balance individual SP and system performance in scaled deployments and under diverse data heterogeneity.

Satellite network is envisioned as a key enabler for wide-area coverage and seamless connectivity. Nonetheless, satellite-terrestrial communications are confronted with critical challenges, including significant signal attenuation and complex inter-satellite interference, particularly in ensuring uplink performance for power-constrained ground users. Reconfigurable intelligent surface (RIS) is considered as a promising solution to compensate for the severe path loss, owing to their high-gain and dynamic beamforming capabilities. In addition, with the deployment of ultra-dense satellite constellations, multi-satellite cooperation is expected to effectively mitigate inter-satellite interference and enhance channel gains. In this paper, a novel RIS-aided multi-satellite cooperative reception scheme is proposed for multi-user uplink transmission. We first analyze five types of multi-satellite cooperative reception framework within cell-free paradigm and formulate a multi-user total energy efficiency (EE) maximization problem. Based on block coordinate descent method, this nonconvex optimization problem is decomposed into three subproblems: receiver detection vector design, RIS phase shift optimization, and multi-user transmit power control. Maximum ratio combining, zero forcing, and minimum mean square error are adopted for linear detection vector designs, respectively. Then the RIS phase shift is optimized based on Riemannian conjugate gradient algorithm. Furthermore, multi-user transmit power control is designed using Dinkelbach’s algorithm. Finally, the total EE optimization problem is solved in an iterative manner. Extensive simulation results validate the effectiveness of the proposed reception scheme and algorithm.

As the essential complements to terrestrial networks, satellite networks have attracted widespread attention due to their potential to enable seamless coverage. However, satellite-terrestrial communications face several critical challenges, including severe path loss and inter-satellite interference, which pose significant obstacles, especially for energy-constrained uplink transmissions. To overcome these problems, a reconfigurable intelligent surfaces (RIS)-aided multi-satellite cooperative reception scheme is proposed in this paper. Specifically, an energy efficiency (EE) optimization problem for multi-user uplink transmission is first formulated. Based on block coordinate descent method, the total EE maximization problem is decomposed into three subproblems: receive beamforming design, RIS phase shift optimization, and multi-user transmit power optimization. Furthermore, the aforementioned subproblems are respectively tackled by applying linear receiver design, the Riemannian conjugate gradient algorithm, and the Dinkelbach’s algorithm. The superiority of the proposed scheme is validated by numerical results, which reveal significant gains in sum rate and total EE.

Low earth orbit (LEO) satellite networks are becoming increasingly important for the development of sixth-generation mobile communication systems along with the concept of integrated space-air-ground networks. However, LEO satellite networks face numerous challenges due to their dynamic topology, large number of satellite nodes, unbalanced load distribution, and differences in communication capabilities. These factors make it difficult for traditional routing strategies to effectively handle the complex network environment and meet the requirements for low latency. To solve these problems, this paper proposes a deep reinforcement learning (DRL)-based routing algorithm for LEO satellite networks, aimed at minimizing the overall end-to-end delay through joint optimization of routing and wavelength resources. The algorithm combines DRL with a wavelength switching model and queuing theory to dynamically optimize routing decisions in response to the existing network state. Simulation results demonstrate that the algorithm exhibits strong convergence properties and effectively reduces full-path delay compared to traditional routing algorithms.

In decentralized learning systems over wireless device-to-device (D2D) networks, training latency is a key metric that needs to be minimized by link selection and resource allocation, thereby accelerating model training. However, it may cause large computational complexity in general. To tackle the challenge, this paper proposes a graph neural network (GNN)-based algorithm to minimize the training latency. Under modeling the D2D network as a graph, the link selection and resource allocation can be efficiently obtained based on the local computing power and link quality. By the constraint on the network connectivity, the training latency can be significantly reduced while guaranteeing accuracy with a low complexity. The simulation results demonstrate that the GNN-based approach outperforms traditional approaches, offering superior scalability and robustness in heterogeneous large-scale D2D networks.

In the context of Internet of Things edge devices, pretrained models are often sourced directly from cloud computing platforms due to the unavailability of training data. This lack of access during the training phase makes these models susceptible to backdoor attacks. To address this challenge, we introduce a novel data-free backdoor removal method that operates effectively even when only the poisoned model is accessible. Our innovative approach employs two end-to-end generators with identical architectures to create both clean and poisoned samples. These samples are crucial for transferring knowledge from the teacher model—the fixed poisoned model—to the student model, which is initialized with the poisoned model. Our method utilizes a channel shuffling technique during the distillation process to disrupt and eliminate the backdoor knowledge embedded in the teacher model. This process involves iterative updates of the generators and meticulous distillation of the student model, leading to efficient backdoor removal. We conducted extensive experiments on five sophisticated backdoor attacks across two benchmark datasets. The results demonstrate that our method not only significantly bolsters the model’s resistance to backdoor attacks but also maintains high recognition accuracy for clean samples, thereby outperforming existing methods. Additionally, the code for our method is available athttps://github.com/gaoyafeiyoo/ADBR.

Edge computing is near end users and provides them with fast computing services. Compared to the cloud, edge provides services with low communication delays but can only service limited users due to constrained storage and computing resources. Thus, allocating more computing resources to some jobs will block the execution of others, and the edge sends them either to the cloud or back to the users. This article focuses on minimizing the average system time by exploring blocked job scheduling (BJS) and redundant computing resource allocation (RCRA) in the cloud–edge–user system. Since edge nodes often operate in highly unpredictable environments, we adopt the replication redundancy to use the resources to shorten the job execution time. First, we propose an approximate model to evaluate the average system time theoretically. Second, we analyze the optimal scheduling for the blocked jobs and the optimal resource allocation for the redundant computing resources. Finally, we propose an algorithm combining BJS and RCRA. Simulation results show that the proposed model approximates the cloud–edge–user system well, and the combined algorithm significantly outperforms the other algorithms under different service time distributions.

This letter proposes a computing resource sharing strategy based on blocked job offloading in the low-Earth orbit (LEO) satellite network. The proposed strategy allows a satellite to share all or part of its computing resources with other satellites, and each satellite offloads or receives the blocked jobs from the adjacent satellites on the same meridian and latitude lines. Furthermore, we analyze the job execution probability, which evaluates the likelihood of the job being executed in the satellite network, for resource sharing strategies with different blocked job offloading hops. The numerical results validate the performance advantages of the computing resource sharing strategies and indicate a way to select the proper strategy.

Non-Terrestrial Networks (NTNs), as one core infrastructure of the sixth-generation (6G) communication technology, integrate heterogeneous nodes, such as High Earth Orbit (HEO) satellites, to achieve three-dimensional ubiquitous connectivity. However, NTNs face with spectrum scarcity, imposing stringent demands on spectral efficiency and interference management. To address these challenges, we propose a NOMA-oriented spectrum sensing technique for uplink scenarios, where High-Altitude Platforms (HAPs) serve as dynamic aerial nodes for opportunistic transmission within HEO coverage. Specifically, we derive multi-user sensing thresholds to optimize detection accuracy and suppress false alarms. Numerical simulations demonstrate the technique achieves a 33.5% throughput gain over benchmarks at 10 dB and maintains satisfactory performance across PUs’ varying elevation angles and transmission willingness.

The evolution of intelligent transportation systems necessitate the integration of advanced technologies to address challenges in data processing, real-time decision-making, and network coverage. In this paper, we present a novel intelligent transportation system architecture that synergizes Unmanned Aerial Vehicles (UAVs), Digital Twins (DTs), and an Integrated Sensing, Communication, and Computation (ISCC) framework. In this system, UAVs equipped with edge computing capabilities collaborate with the ground-based cloud center to process data collected by perception devices (PDs). Each UAV and PD is mirrored by a Digital Twin (DT) at the base station, enabling real-time monitoring and predictive analytics. We intend to maximize the network lifetime and minimize the economic overhead, which is achieved through the joint optimization of the association policies of UEs, input data caching decisions, UAVs’ flight trajectory and speed, task processing mode selection and data unload proportion allocation under joint processing. To tackle the inherent challenges of nonlinearity, dynamic network conditions, and heterogeneous data sources, the problem is modeled as a Markov Decision Process (MDP) and solved using an enhanced Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm, adapted to handle both continuous and discrete action spaces. The experimental results show that compared to the baseline algorithm, this approach not only exhibits faster convergence but also achieves outstanding performance in maximizing system utility for intelligent transportation systems.

Deep mutual learning (DML) is one of the emerging technologies for mobile intelligent applications that has attracted much attention in recent years. To effectively deploy DML at a large scale, in this paper, we propose a novel hierarchical asynchronous deep mutual learning (HADML) system that enables devices to collaborate in model training without the exchange of local datasets. To further improve the learning efficiency, the average energy cost for model exchanging is minimized by jointly optimizing the link scheduling and communication resource allocation. To efficiently solve this problem, the graph neural network and deep unfolding network are employed to obtain the link scheduling and resource allocation, respectively. Finally, the simulation results demonstrate that our proposed algorithm can achieve a balance between knowledge sharing and communication energy consumption.

The digital twin technology facilitates the application of Metaverse in autonomous driving. Particularly, this paper focuses on investigating an unmanned aerial vehicle (UAV)-aided vehicular Metaverse. In specific, the moving vehicles in physical world collect the real-time traffic data, which is synchronized through the UAV to the virtual world to help the autonomous vehicle (AV) simulation. For such a physical-virtual synchro-nization process, we define the age of incorrect information (AOII) to measure the traffic data freshness. Accounting for the randomness in the physical world, we jointly optimize the UAV trajectory, the vehicle scheduling and the semantic extraction of collected data under the Markov decision process (MDP) framework. Our objective is to minimize the expected long-term system AOII. Without the statistical knowledge of physical-world randomness, we propose to leverage a proximal policy optimization based deep reinforcement learning algorithm to solve the optimal control policy to the MDP formulation. We conduct numerical experiments to verify the accuracy of the theoretical analysis, and the results demonstrate the performance gains from our proposed algorithm.

In the context of the large-scale deployment of 5G base stations, atmospheric ducts cause remote interference in time division duplex systems. Addressing the impact of remote interference on communication systems necessitates timely prediction and emergency mitigation of atmospheric ducts. In this paper, a genetic programming-based multiclass remote interference prediction network model is proposed. Firstly, the proposed model can directly learn to make predictions from extensive databases without relying on any assumptions. Secondly, it presents a genetic programming strategy capable of automatically adjusting the model's structure, thereby enhancing the prediction accuracy of various interference classes. Numerical results demonstrate that the multiclass remote interference prediction network (MRIPNet) outperforms state-of-the-art interference prediction models when tested on real-world datasets. Further-more, MRIPNet excels in accurately predicting a small number of severe interference, which help operators promptly execute interference avoidance measures.

Information sharing among vehicles plays a critical role in improving driving safety and traffic loads in Internet of Vehicles (IoV). However, due to the existence of malicious vehicles, information sharing among vehicles lacks a trustworthy environment. It is challenging for a vehicle to assess the credibility of the received information. Blockchain has attracted extensive attention because of the decentralization and tamper-proof characteristics. Nevertheless, the high mobility of vehicles leads to rapid network topology changes, which makes it difficult to reach a consensus. In this article, we propose a consortium blockchain-based trust-value management approach to build a trustworthy environment for information sharing among vehicles. A new incentive mechanism is designed to encourage vehicles to actively participate in blockchain maintenance. Furthermore, an Enhanced Proof of Work (EPoW) consensus algorithm is designed to reduce the amount of information that needs to be transmitted over the network to reach consensus quickly. Comprehensive analysis and simulation results verify that the proposed trust-value management approach and EPoW consensus algorithm realize secure and efficient information sharing in IoV.

As a vital enabler for ubiquitous connectivity in space-air-ground integrated networks, the satellite-terrestrial communication system has been envisioned to be a crucial complement to terrestrial networks because of its superior capability of providing wide coverage. However, there are many practical limitations that degrade system performance, including on-board power constraints, severe path loss, high delay, and Doppler frequency shift. Reconfigurable intelligent surface (RIS), an attractive candidate technology for future networks, has been expected to be a promising solution to tackle these challenges. In this paper, we propose an RIS-assisted multi-satellite cooperative downlink transmission scheme, where an RIS is deployed near the terrestrial receiver to enhance the communication via joint design of beamforming vectors at satellites and phase shift optimization at RIS. Specifically, we first optimize the channel gain of the proposed transmission scheme under the continuous-time propagation model. Then, we formulate the problem as a maximization optimization of the received signal power during the duration of multi-satellite communication service, and we apply an alternating optimization (AO) algorithm to solve the non-convex problem. To reduce computational complexity, we further propose a low-complexity optimization algorithm under line-of-sight (LoS) transmission conditions and a scalable distributed optimization algorithm without the need to share the multi-satellite individual channel state information. Simulation results validate the superiority of our proposed transmission scheme and demonstrate that the low-complexity algorithm and the distributed algorithm can achieve system performance comparable to that of the AO algorithm under LoS channel conditions.

The surging demand for high-definition video streaming services and large neural network models implies a tremendous explosion of Internet traffic. To mitigate the traffic pressure, architectures with in-network storage have been proposed to cache popular contents at devices in closer proximity to users. Correspondingly, in order to maximize caching utilization, it becomes essential to devise an effective popularity prediction method. In that regard, predicting popularity with dynamic graph neural network (DGNN) models achieves remarkable performance. However, DGNN models still suffer from tackling sparse datasets where most users are inactive. Therefore, we propose a reformative temporal graph network, named semantics-enhanced temporal graph network (STGN), which attaches extra semantic information into the user-content bipartite graph and could better leverage implicit relationships behind the superficial topology structure. On top of that, we customize its temporal and structural learning modules to further boost the prediction performance. Specifically, in order to efficiently aggregate the diversified semantics that a content might possess, we design a user-specific attention (UsAttn) mechanism for the temporal learning. Unlike the attention mechanism that only analyzes the influence of genres on content, UsAttn also considers the attraction of semantic information to a specific user. Meanwhile, as for the structural learning, we introduce the concept of positional encoding into our attention-based graph learning and novelly adopt a semantic positional encoding (SPE) function, which effectively boost the performance of lightweight algorithms. Finally, extensive simulations verify the superiority of our models and demonstrate their effectiveness in content caching.

Deep mutual learning (DML) is one of the most high-profile technologies emerging in the field of machine learning during the past few years. DML has the potential of exchanging knowledge on the premise of ensuring data privacy, while retaining the characteristics of local models. In this paper, we design a novel system named as decentralized mutual learning over unlicensed spectrum (DML-U), which allows neighbor mobile devices to learn from each other via bidirectional device-to-device links over unlicensed spectrum (D2D-U). On this basis, we formulate a non-convex optimization problem for the one-to-one pairing scenario with the goal of minimizing the average communication energy cost for sharing knowledge. We further propose a two-layer iterative algorithm that includes the outer layer based on the enumeration method and the inner layer based on the sum-of-ratios optimization, aiming to find the optimal pairing scheme between devices and obtain the global optimal communication resource allocation scheme, respectively. The numerical results validate the effectiveness of the proposed algorithm in improving the DML performance.

Open radio access network (O-RAN) slicing allows the flexible control of network components and resources to satisfy the ever increasing demand of mobile applications. To optimize service provisioning, efficient management of limited radio resources is challenging due to the orchestration among network slices in the long-timescale and the slice configurations according to the mobile user (MU) statistics in the short-timescale. In this paper, we first propose a novel meta Markov decision process framework to mathematically formulate the problem of two-timescale radio resource management (RRM) in O-RAN slicing. The original RRM problem is then decoupled into a long-timescale master problem and a short-timescale subproblem, which are solved by a hierarchical reinforcement learning (RL) mechanism. Our proposed hierarchical RL mechanism includes a deep RL algorithm, solving the optimal long-timescale RRM policy, and a linear-decomposition based meta-RL algorithm, solving the optimal short-timescale RRM policy. Numerical experiments verify the theoretical analysis and show that our proposed hierarchical RL mechanism outperforms the most representative state-of-the-art baselines.

Achieving seamless connectivity is one of the key visions for sixth-generation (6G) wireless communication systems. Satellite communication (SatCom) system, especially the multi-satellite cooperative communication, has become an indispensable part of space-air-ground integrated communication network due to its superior capability in providing wide area coverage. However, the performance of satellite-terrestrial communication is severely limited by high path loss. Reconfigurable Intelligent Surface (RIS) has been considered as a potential enabling technology in 6G to improve the service quality of various communication systems. In this paper, a RIS-aided multi-beam cooperative transmission scheme is proposed to boost the system performance of satellite-terrestrial communication. By optimizing the RIS reflection phase dynamically, the system capacity is considerably improved during the continuous service time of dual satellites. Based on the simulation results, we also point out the impact of the discrete phase optimization and RIS's inclination angle on the system performance considering the deployment of RIS in three-dimensional (3D) space.

In this paper, we design a novel system named as decentralized mutual learning over unlicensed spectrum (DML-U), where edge devices are allowed to learn from each other via bidirectional device-to-device communications over unlicensed spectrum. We further formulate a non-convex optimization problem to minimize energy consumption and accelerate knowledge sharing with constrained power, bandwidth and transmission latency. Under this context, we propose a two-layer iterative algorithm, which contains an enumeration-based outer layer for the pairing scheme and a sum-of-ratios-based inner layer for obtaining a globally optimal allocation of communication resources. Simulation results verify that our obtained algorithm converges fast and finds efficiently the balance between knowledge sharing and communication energy consumption.

Semantic communication is a novel communication paradigm that focuses on recognizing and delivering the desired meaning of messages to the destination users. Most existing works in this area focus on delivering explicit semantics, labels or signal features that can be directly identified from the source signals. In this paper, we consider the implicit semantic communication problem in which hidden relations and closely related semantic terms that cannot be recognized from the source signals need to also be delivered to the destination user. We develop a novel adversarial learning-based implicit semantic-aware communication (iSAC) architecture in which the source user, instead of maximizing the total amount of information transmitted to the channel, aims to help the recipient learn an inference rule that can automatically generate implicit semantics based on limited clue information. We prove that by applying iSAC, the destination user can always learn an inference rule that matches the true inference rule of the source messages. Experimental results show that the proposed iSAC can offer up to a 19.69 dB improvement over existing non-inferential communication solutions, in terms of symbol error rate at the destination user.

The enormous amount of network equipment and users implies a tremendous growth of Internet traffic for multi-media services. To mitigate the traffic pressure, architectures with in-network storage have been proposed to cache popular content at devices in close proximity to users in order to decrease the number of backhaul hops. Meanwhile, the reduced transmission distance also contributes to energy saving. However, due to limited storage, only a fraction of the content can be cached, while caching the most popular content is cost-effective. Correspondingly, it becomes essential to devise an effective popularity prediction method. In this regard, some existing efforts manifest the effectiveness of dynamic graph neural network (DGNN) models, but it remains challenging to tackle sparse datasets. Herein, we first propose a reformative temporal graph network, named STGN, to address the challenge and improve prediction performance. Specifically, the STGN model leverages extra semantic messages to help establish implicit paths within the sparse interaction graph and enhance the temporal and structural learning of a DGNN model. Furthermore, we devise a user-specific attention mechanism to aggregate various semantics in a fine-grained manner. Finally, extensive simulations verify the superiority of our STGN models and demonstrate the potential in terms of energy-saving.

Previous works have proven the superior performance of ensemble-based black-box attacks on transferability. However, existing methods require significant difference in architecture among the source models to ensure gradient diversity. In this paper, we propose a Diverse Gradient Method (DGM), verifying that knowledge distillation is able to generate diverse gradients from unchangeable model architecture for boosting transferability. The core idea behind our DGM is to obtain transferable adversarial perturbations by fusing diverse gradients provided by a single source model and its distilled versions through an ensemble strategy. Experimental results show that DGM successfully crafts adversarial examples with higher transferability, only requiring extremely low training cost. Furthermore, our proposed method could be used as a flexible module to improve transferability of most of existing black-box attacks.

Semantic communication, a promising candidate for 6G technology, has become a research hot spot. However, existing studies tend to focus more on image reconstruction rather than accurately transmitting semantic information at the pixel level. This paper introduces a novel approach using codec-based Masked AutoEncoders (MAE) for efficient image transmission. The proposed system compresses local information into low-dimensional latent vectors, improving system efficiency. We also design a selective module for enhanced image reconstruction and implement Noise Adversarial Training (NAT) to increase the system’s resilience to channel noise. Experimental results show that our method effectively improves downstream tasks while preserving image quality.

Wi-Fi-based contactless activity recognition is of great importance to computer–human interaction, accounting for convenience concerns. However, it remains challenging to recognize activities from multiple users due to the multipath distortion and disruption of Wi-Fi signals. In this article, we propose a highly universal framework, namely, WISDOM, for Wi-Fi-based multiuser activity recognition. Specifically, we first leverage an existing model to identify the number of users from the input Wi-Fi signals. Then, we develop a subcarrier correlation and inversion-based sorting algorithm to extract the signal for each user. Finally, we design a neural network, i.e., WISDOM-Net, which is built on a bidirectional gated recurrent unit network incorporated with the attention mechanism and the one dimension convolutional neural network, to recognize the corresponding user activities. Experimental results show that our proposed WISDOM-Net outperforms the existing baselines on both the public and our own data sets. In particular, WISDOM-Net can reach an average recognition accuracy of up to 98.19% and 90.77% in 2-user and 3-user scenarios, respectively.

Mobile-edge computing (MEC) brings the potential to address the ever increasing computation demands from the mobile users (MUs). In addition to local processing, the resource-constrained MUs in an MEC system can also offload computation to the nearby servers for remote execution. With the explosive growth of mobile devices, computation offloading faces the challenge of spectrum congestion, which, in turn, deteriorates the overall quality of computation experience. This article, hence, investigates computation task scheduling in a heterogeneous cellular and WiFi MEC system. Such a system provides both licensed and unlicensed spectrum opportunities. Due to the sharing of communication and computation resources as well as the uncertainties, we formulate the problem of computation task scheduling among the competing MUs in a stationary heterogeneous edge computing system as a noncooperative stochastic game. We propose an approximation-based multiagent Markov decision process without the global system state observations, under which a multiagent proximal policy optimization (PPO) algorithm is derived to solve the corresponding Nash equilibrium. When expanding to a nonstationary heterogeneous edge computing system, the obtained algorithm suffers from the slow convergence due to constrained adaptability. Accordingly, we explore meta-learning and propose a multiagent meta-PPO algorithm, which rapidly adapts the control policy learning to the nonstationarity. Numerical experiments demonstrate performance gains from our proposed algorithms.

With the rapid increase of the demand for the Internet of Things (IoT), spectrum resources have incremental challenges. Nonorthogonal multiple access (NOMA) and spectrum sensing (SS) are considered key candidate technologies for next-generation wireless communications to improve spectrum utilization. Nevertheless, using both technologies at the same time makes the system more complex and brings new challenges to user differentiation. In order to make better use of these advantages, we creatively propose a feature detection-based SS method for NOMA systems. To better distinguish the relationship between the presence or absence of signals from different NOMA users, we employ feature detection to obtain the feature values of each user. We propose workflows and transceiver architectures combining the two technologies. Based on the relationship among users’ priorities, power, and transmission in common scenarios, we design a downlink mode and two uplink modes and deduce the threshold settings of the corresponding modes. Meanwhile, we also customarily propose enhanced algorithms, to have a marked increase in the performance for the proposed method in various modes. Experimental results illustrate that the proposed technique is feasible and has prominent detection performance and satisfying throughput performance.

This article studies joint power, spectrum and computational resource allocation in mobile edge computing (MEC) systems. Considering that the licensed spectrum resources are not sufficient, the computing tasks can also be uploaded to the remote MEC server (MECS) via the unlicensed spectrum. To facilitate fair coexistence with Wi-Fi networks, we adopt the duty-cycle-muting mechanism with adaptive adjustment of the duty cycle on unlicensed channels. We propose a Stackelberg game formulation, where the aim is to minimize the long-term energy consumption of the noncooperative user terminals (UEs) while guaranteeing the stability of task buffers. In the game, the MECS prices the licensed spectrum to indirectly adjust the proportion of bandwidth for each UE. In particular, we develop a distributed resource management algorithm, which enables the UEs to behave independently and adaptively. Theoretical analysis and simulations demonstrate the effectiveness of our proposed algorithm with respect to energy saving under constrained signaling overheads.

Volumetric video provides users with a good viewing experience of six degrees of freedom (DoF) and has wide applications in many fields such as teleconferencing and online games. However, the huge data volume and strict latency requirements of point cloud video, the most popular representative of volumetric video, pose a challenge to its transmission. Existing point cloud video transmission algorithms usually segment a long video by every one or several group of frames, predict network bandwidth and field of view (FoV) information, then perform adaptive transmission by solving the quality of experience (QoE) optimization problem. However, such segmentation neglects the impact of current optimization decisions on the subsequent video streaming process, as well as the accumulated prediction error across a long interval, severely degrading user’s QoE. Moreover, the complex constrained optimization problem makes the solution time too long to meet the real-time video streaming requirements. To this end, in this paper, we propose a rolling prediction-optimization-transmission (POT) framework, which makes predictions of network bandwidth and FoV in each short rolling window to reduce prediction error. And our framework takes into account the upper bounded QoE contribution of the subsequent point cloud video to improve the system performance. In addition, we design a deep reinforcement learning based real-time solver to make decisions for the fixed structure optimization problem in each roll, allowing our system to run in real-time. We have performed simulations and experiments, and the results show that our solution outperforms existing methods.

To improve the transportation efficiency of advanced manufacturing, cameras have been extensively deployed to enhance the on-road perception of autonomous ground vehicles in smart industrial parks. Considering the informative yet substantial data volume of contents generated by those cameras, we employ vehicle-to-everything links to deliver the captured video frames to neighboring vehicles, road-side units, or base stations, in order to respond to vehicle-control-related video queries. To help vehicles obtain low-latency and high-accuracy on-road information for autonomous driving, an optimization problem is formulated, taking into account the impact of vehicle mobility and diverse resource demands of different video queries. Then, a two-stage algorithm is proposed to determine the frame rate of video cameras, as well as the destination of the associated video frames based on matching theory. Extensive simulation results show that this approach can improve the accuracy by up to 18% and reduce the response delay by 13.2%.

In this paper, we consider task offloading in edge computing systems, where tasks are offloaded by the base station to resourceful mobile users. With the consideration of unique characteristics in practical edge computing systems, such as dynamic arrival of computation tasks, and energy constraints at battery-powered mobile users, we formulate an incentive mechanism design problem by jointly optimizing task offloading decisions, and allocation of both communications (i.e., power and bandwidth), and computation resources. In order to tackle the nonlinear issue in the designed mechanism, a novel online incentive mechanism is proposed. We first convert the original mechanism design problem into several one-shot design problems by temporally removing the energy constraint. Then, we propose a new mechanism design framework, called the Integrate Rounding Scheme based Maxima-in-distributional Range (IRSM), and based on that, design a new incentive mechanism for each one-shot problem. Finally, we reconsider energy constraints to design a new nonlinear online incentive mechanism by rationally combining the previously derived one-shot ones. Theoretical analyses show that our proposed nonlinear online incentive mechanism can guarantee individual rationality, truthfulness, a sound competitive ratio, and computational efficiency. We further conduct comprehensive simulations to validate the effectiveness and superiority of our proposed mechanism.

Device-to-device (D2D)-assisted decentralized learning has been proposed for mobile devices to collaboratively train artificial intelligence networks without the centralized parameter server. However, a densely connected network will cause large learning latency and energy consumption due to the limited computation and communication resources. In addition, link selection and aggregation weight have a significant impact on the learning performance. To cope with these challenges, we propose a joint computing power adjustment, wireless resource allocation, link selection, and aggregation weight adaptation mechanism to improve both communication and energy efficiencies. Specifically, the learning performances including the convergence rate, per-iteration learning latency, and per-iteration energy consumption are first analyzed. Then, an optimization problem is formulated to minimize the total learning cost, which is defined as the weighted sum of total learning latency and energy consumption. Given a network topology, the computing power and wireless resource allocation are optimized by the alternating optimization algorithm. Moreover, the optimal aggregation weight is obtained by semidefinite programming. With respect to link selection, we propose a tabu search based meta-heuristic algorithm to approximately achieve feasible solutions with a low computational complexity. Finally, extensive experiments demonstrate that the proposed link selection algorithm can significantly reduce the learning cost under the given learning accuracy requirement.

In this paper, a novel decentralized mutual learning (DML) network is designed, where each mobile device can share knowledge with its neighbour devices via bidirectional device-to-device (D2D) communication. We subdivide and discuss mutual learning scenarios, and investigate the user association and resource allocation problems for the one-to-many scenario. With constraints on power, bandwidth and communication latency, we formulate a non-convex optimization problem to minimize the average communication energy consumption for sharing new knowledge. On the basis, a two-layer iterative algorithm is proposed, which consists of an outer layer algorithm based on particle swarm optimisation (PSO) for searching a suitable user association strategy and an inner layer algorithm based on sum-of-ratios optimization for achieving a globally optimal allocation of communication resource. Numerical results are presented to verify the fast convergence and the effectiveness of the proposed algorithm in terms of a trade-off between energy consumption and knowledge sharing efficiency.

Massive data drives the development of machine learning (ML) for a long time. However, at present, data is starting to hinder ML's development. The first reason is that the privacy of data is increasingly valued by the public. Therefore, Federated Learning (FL) has emerged, which realizes model training through distributed computing and centralized aggregation. Second, due to the popularity of FL, edge devices need to store all data, which may quickly occupy the entire storage space of edge devices, resulting in fatal errors. To address these challenges, we proposed a storage-aware joint user scheduling and spectrum allocation algorithm, named FedSUS, to reduce the storage stress of each device and guarantee traditional FL metrics, i.e., learning accuracy and training latency. First, a probabilistic framework is adopted for user scheduling. Second, we introduce a data influence evaluation method to FL and analyze its convergence. Based on this, two problems are formulated to tradeoff the storage resource, the influence of data, and the learning latency and to minimize the transmission latency, respectively. Then, the closed-form results to the above problems are both developed. Finally, FedSUS is validated by using a popular convolutional neural network (CNN) and datasets (CIFAR-10). And numerical results demonstrate that our algorithm can effectively reduce the local data size while keeping (even improving) the learning accuracy as compared with baseline.

In point cloud video streaming systems, the field of view (FoV) prediction is critical for selecting the tiles, the objective of which is to optimize the expected long-term quality-of-experience (QoE) from the perspective of a user. On one hand, a satisfactory QoE accounts for not only the playback quality but also the playback smoothness. On the other hand, the large data volume of a selected tile requires the transmission to be adaptive to the system uncertainties. This paper applies a Markov decision process to formulate the problem of tile selection across the infinite discrete time horizon. In particular, a system state includes the FoV information, which is predicted from the Transformer. To alleviate the dependence on system uncertainty statistics, a deep reinforcement learning approach is derived for solving the optimal control policy. Under different settings, we conduct experiments based on the real throughput and head-mounted display data. The results show that compared to the existing baselines, our proposed prediction-control approach achieves a higher FoV prediction accuracy, better playback quality as well as smoothness, and hence a better average QoE for the user.

Wireless hierarchical federated learning (HFL) has been proposed for large-scale model training over multi-cell network while preserving the data privacy. However, the imbalanced data distribution and load have a significant impact on the convergence rate, the learning accuracy, and the learning latency in wireless HFL with non-independent identically distributed training data. To cope with these challenges, we first derive the learning latency and the upper bound of the model error. Then, an optimization problem is formulated to minimize the weighted sum of total data distribution distance and learning latency. Joint user association and wireless resource allocation algorithms are investigated to achieve the optimal learning performance. Finally, the effectiveness of the proposed algorithms are demonstrated by the simulations.

This paper investigates an air-ground integrated multi-access edge computing system, which is deployed by an infrastructure provider (InP). Under a business agreement with the InP, a third-party service provider provides computing services to the subscribed mobile users (MUs). MUs compete for the shared spectrum and computing resources over time to achieve their distinctive goals. From the perspective of an MU, we deliberately define the age of update to capture the staleness of information from refreshing computation outcomes. Given the system dynamics, we model the interactions among MUs as a stochastic game. In the Nash equilibrium without cooperation, each MU behaves in accordance with the local system states and conjectures. We can hence transform the stochastic game into a single-agent Markov decision process. As another major contribution, we develop an online deep reinforcement learning (RL) scheme that adopts two separate double deep Q-networks to approximate the Q-factor and the post-decision Q-factor, respectively. The deep RL scheme allows each MU to optimize the behaviours with unknown dynamic statistics. Numerical experiments show that our proposed scheme outperforms the baselines in terms of the average utility under various system conditions.

In this work, hierarchical federated learning (HFL) over wireless multi-cell networks is proposed for large-scale model training while preserving data privacy. However, the imbalanced data distribution has a significant impact on the convergence rate and learning accuracy. In addition, a large learning latency is incurred due to the traffic load imbalance among base stations (BSs) and limited wireless resources. To cope with these challenges, we first provide an analysis of the model error and learning latency in wireless HFL. Then, joint user association and wireless resource allocation algorithms are investigated under independent identically distributed (IID) and non-IID training data, respectively. For the IID case, a learning latency aware strategy is designed to minimize the learning latency by optimizing user association and wireless resource allocation, where a mobile device selects the BS with the maximal uplink channel signal-to-noise ratio (SNR). For the non-IID case, the total data distribution distance and learning latency are jointly minimized to achieve the optimal user association and resource allocation. The results show that both data distribution and uplink channel SNR should be taken into consideration for user association in the non-IID case. Finally, the effectiveness of the proposed algorithms are demonstrated by the simulations.

This paper addresses a joint radio and computational resources allocation problem for mobile edge computing (MEC) networks. To alleviate the shortage of licensed spectrum resources, computing tasks can be offloaded to the MEC server through not only the licensed channels but also the unlicensed channels, where the adaptive duty-cycle-muting (DCM) mechanism is employed at the user terminals (UTs) to guarantee the fair coexistence with the WiFi networks. Moreover, Stackelberg game formulation is used to build up a decentralized radio and computational resources allocation framework, where the MEC server is modeled as a leader to set the price of the licensed spectrum, while UTs as followers compete for the radio and computational resources as a non-cooperative game. The objective of each UT is to minimize the long-term energy consumption with the guarantee of task buffer stability. Accordingly, we develop a distributed algorithm to achieve the equilibrium solution for the formulated Stackelberg game. Numerical results are presented to demonstrate that the proposed scheme is effective with respect to the reduction on energy consumption of UTs with limited signaling overheads.

Mobile edge computing (MEC) is a promising paradigm for alleviating the computation burden of resource-constrained mobile devices. Nevertheless, the majority of existing efforts concentrate on offloading computations from mobile de-vices to an edge computing server through the cellular networks only. With the development of wireless connectivity technologies, WiFi networks over unlicensed spectrum provide a “green” (i.e., cost-efficient and economical) alternative for computation offloading. In this paper, we investigate the problem of computation offloading in a heterogeneous WiFi and cellular MEC system, where both the WiFi and the cellular networks are possible for offloading the arriving computation tasks at a mobile user (MU). The objective of an MU is to minimize the long-term cost, which can be described as a single-agent Markov decision process (MDP) by accounting for the inherent system dynamics in the MU mobility, sporadic computation task arrivals and wireless connectivity variations. To solve the optimal strategy for the formulated MDP with a high-dimensional state space but without the statistical knowledge of system dynamics, we resort to a model-free deep reinforcement learning algorithm. Numerical experiments verify that the proposed algorithm is able to significantly reduce the average computation offloading cost compared with other baselines.

In this paper, we investigate a cloud radio access network (Cloud-RAN) in which both fronthaul and radio access links use massive MIMO millimeter-wave (mmWave) transmissions. Such an all-mmWave Cloud-RAN architecture provides a flexible and cost-effective means for deployment of next-generation (5G and beyond) cellular networks to meet the demands of fast-growing mobile data traffic. Nevertheless, the design of transmit power allocation schemes for multiple massive MIMO beams on both the fronthaul and access links in a mmWave Cloud-RAN is challenging. In particular, the traffic and wireless channel states of multiple mobile terminals (MTs) change over time, while their statistics may not be known a priori. We formulate the joint fronthaul-access link massive MIMO beam power allocation problem as a Markov decision process (MDP) with an objective to optimize the long-term quality of service to all the MTs in a Cloud-RAN. A reinforcement learning algorithm is designed, which learns the optimal beam power allocation policy on the fly and adapts to the network dynamics. Further, by leveraging the structure of the underlying problem, a post-decision state is introduced and a function decomposition technique is developed to reduce the search space during the learning process. The evaluation results validate the convergence of our proposed scheme and demonstrate its superior performance over the state-of-the-art baselines.

The point cloud videos, thanks to the multi-view and immersive experiences, have recently attracted notable attentions from both academia and industry. Due to the high data volume, a point cloud video also raises the challenge of quality-of-experience (QoE), which is in terms of the balance between playback quality and buffering delay during the transmission under time-varying system conditions. In this paper, we propose a deep reinforcement learning (DRL) approach to optimize the expected long-term QoE for the client. Over the time horizon, the proposed approach learns to select the tiles of the corresponding video for transmissions in an iterative way. Under various settings, numerical experiments based on real throughput data traces are conducted to evaluate the proposed approach. Compared to the baselines, our approach not only enhances the video quality but also reduces the re-buffering time, obtaining an improvement of average QoE for the client by 9%–14%.

In this paper, a distributed power and spectrum allocation scheme is proposed to maximize the system energy efficiency (EE) for unlicensed device-to-device (D2D-U) networks. A non-convex optimization problem is formulated while considering the co-channel interference on licensed bands as the global constraint. To deal with the non-convex situation, the object function is converted into an equivalent convex one. Then, a distributed algorithm is developed to solve the optimization problem in which each D2D-U pair can find the optimal allocation strategy independently. The signaling overheads are analyzed and numerical results are presented to show that the proposed scheme is capable of achieving the optimal EE while confining the cochannel interference and guaranteeing the fair coexistence.

WiFi-based identity recognition is predominant because of its noninvasive and ubiquitous advantages. However, existing approaches show slow training speed and limited applicability. In this article, we propose a lightweight deep learning model, named as lightweight WiFi-based identification (LW-WiID), to address these technical challenges. LW-WiID reconstructs original data of channel state information into frequency energy graph, which contains not only the temporal feature of the gait but also the spatial feature among subcarriers, ensuring the accuracy of identity recognition. Furthermore, a novel Balloon mechanism is designed to achieve the lightweight. Through information integration crossing both layers and channels, the Balloon mechanism effectively reduces the number of model parameters. Experimental results demonstrate that LW-WiID achieves an accuracy of 99.7% on a 50-person gait data set while the model size is compressed to 5.53% of the existing identity recognition approaches with the same accuracy.

Unlike the conventional device-to-device (D2D) networks, the unlicensed D2D (D2D-U) pairs can not only reuse the licensed channels with the base station (BS) but also share the unlicensed channels with the WiFi stations. One challenge arises from the fact that the co-channel interference on licensed channels and the collision probability on unlicensed channels may cause extra power consumption at the terminals. Accordingly, we first propose a channel access method for the D2D-U pairs on unlicensed channels. Then, a decentralized joint spectrum and power allocation scheme is designed to minimize the power consumption at D2D-U pairs. Different from the existing distributed schemes, the proposed scheme can guarantee the global minimization of power consumption across the D2D-U pairs. Simulation results validate the theoretical analysis and verify the performance from the proposed scheme.

The Internet of Vehicles (IoV) enables real-time data exchange among vehicles and roadside units and thus provides a promising solution to alleviate traffic jams in the urban area. Meanwhile, better traffic management via efficient traffic light control can benefit the IoV as well by enabling a better communication environment and decreasing the network load. As such, IoV and efficient traffic light control can formulate a virtuous cycle. Edge computing, an emerging technology to provide low-latency computation capabilities at the edge of the network, can further improve the performance of this cycle. However, while the collected information is valuable, an efficient solution for better utilization and faster feedback has yet to be developed for edge-empowered IoV. To this end, we propose a Decentralized Reinforcement Learning at the Edge for traffic light control in the IoV (DRLE). DRLE exploits the ubiquity of the IoV to accelerate traffic data collection and interpretation towards better traffic light control and congestion alleviation. Operating within the coverage of the edge servers, DRLE aggregates data from neighboring edge servers for city-scale traffic light control. DRLE decomposes the highly complex problem of large area control into a decentralized multi-agent problem. We prove its global optima with concrete mathematical reasoning and demonstrate its superiority over several state-of-the-art algorithms via extensive evaluations.

This paper investigates the problem of age of information (AoI)-aware resource awareness in an unmanned aerial vehicle (UAV)-assisted mobile-edge computing (MEC) system, which is deployed by an infrastructure provider (InP). A service provider leases resources from the InP to serve the mobile users (MUs) with sporadic computation requests. Due to the limited number of channels and the finite shared I/O resource of the UAV, the MUs compete to schedule local and remote task computations in accordance with the observations of system dynamics. The aim of each MU is to selfishly maximize the expected long-term computation performance. We formulate the non-cooperative interactions among the MUs as a stochastic game. To approach the Nash equilibrium solutions, we propose a novel online deep reinforcement learning (DRL) scheme, which enables each MU to behave using its local conjectures only. The DRL scheme employs two separate deep Q-networks to approximate the Q-factor and the post-decision Q-factor for each MU. Numerical experiments show the potentials of the online DRL scheme in balancing the tradeoff between AoI and energy consumption.

After a disaster occurred, it is extremely important to reconstruct the network and provide the communication services to the victims immediately. Deploying MDRU (Movable and Deployable Resource Unit) in the disaster area, along with multiple access points to extend the service area of MDRU is a very promising solution. In this kind of heterogeneous disaster response networks, it is of great importance to minimize the packet delay from user terminals by performing optimal radio access control. In this paper, we propose a deep reinforcement learning based radio access control mechanism, which enables the smart relay selection and transmitting power control. We evaluate the performance by extensive simulations, and validate the superiority of the proposed mechanism by comparing with baseline schemes.

Network Function Virtualization (NFV) enables flexible deployment of network services as applications. Network operators expect to use a limited number of Network Function (NF) instances to handle the fluctuating traffic load and provide network services. However, it is a big challenge to guarantee the Quality of Service (QoS) under the unpredictable network traffic while minimizing the processing resources. One typical solution is to realize NF scale-out, scale-in and load balancing by elastically migrating the related traffic flows with SoftwareDefined Networking (SDN). However, it is difficult to optimally migrate flows since many real-time statuses of NF instances should be considered to make accurate decisions. In this paper, we propose DeepMigration to solve the problem by efficiently and dynamically migrating traffic flows among different NF instances. DeepMigration is a Deep Reinforcement Learning (DRL)-based solution coupled with Graph Neural Network (GNN). By taking advantages of the graph-based relationship deduction ability from our customized GNN and the self-evolution ability from the experience training of DRL, DeepMigration can accurately model the cost (e.g., migration latency) and the benefit (e.g., reducing the number of NF instances) of flow migration among different NF instances and generate dynamic and effective flow migration policies to improve the QoS. Experiment results show that DeepMigration requires less migration cost and saves up to 71.6{%} of the computation time than existing solutions.

This paper investigates an unmanned aerial vehicle (UAV)-assisted mobile-edge computing (MEC) system, in which the UAV provides complementary computation resource to the terrestrial MEC system. The UAV processes the received computation tasks from the mobile users (MUs) by creating the corresponding virtual machines. Due to finite shared I/O resource of the UAV in the MEC system, each MU competes to schedule local as well as remote task computations across the decision epochs, aiming to maximize the expected long-term computation performance. The non-cooperative interactions among the MUs are modeled as a stochastic game, in which the decision makings of a MU depend on the global state statistics and the task scheduling policies of all MUs are coupled. To approximate the Nash equilibrium solutions, we propose a proactive scheme based on the long short-term memory and deep reinforcement learning (DRL) techniques. A digital twin of the MEC system is established to train the proactive DRL scheme offline. Using the proposed scheme, each MU makes task scheduling decisions only with its own information. Numerical experiments show a significant performance gain from the scheme in terms of average utility per MU across the decision epochs.

Network slicing is a key technology in 5G communications system. Its purpose is to dynamically and efficiently allocate resources for diversified services with distinct requirements over a common underlying physical infrastructure. Therein, demand-aware resource allocation is of significant importance to network slicing. In this paper, we consider a scenario that contains several slices in a radio access network with base stations that share the same physical resources (e.g., bandwidth or slots). We leverage deep reinforcement learning (DRL) to solve this problem by considering the varying service demands as the environment state and the allocated resources as the environment action. In order to reduce the effects of the annoying randomness and noise embedded in the received service level agreement (SLA) satisfaction ratio (SSR) and spectrum efficiency (SE), we primarily propose generative adversarial network-powered deep distributional Q network (GAN-DDQN) to learn the action-value distribution driven by minimizing the discrepancy between the estimated action-value distribution and the target action-value distribution. We put forward a reward-clipping mechanism to stabilize GAN-DDQN training against the effects of widely-spanning utility values. Moreover, we further develop Dueling GAN-DDQN, which uses a specially designed dueling generator, to learn the action-value distribution by estimating the state-value distribution and the action advantage function. Finally, we verify the performance of the proposed GAN-DDQN and Dueling GAN-DDQN algorithms through extensive simulations.

In this paper, we investigate the problem of age of information (AoI)-aware radio resource management for expected long-term performance optimization in a Manhattan grid vehicle-to-vehicle network. With the observation of global network state at each scheduling slot, the roadside unit (RSU) allocates the frequency bands and schedules packet transmissions for all vehicle user equipment-pairs (VUE-pairs). We model the stochastic decision-making procedure as a discrete-time single-agent Markov decision process (MDP). The technical challenges in solving the optimal control policy originate from high spatial mobility and temporally varying traffic information arrivals of the VUE-pairs. To make the problem solving tractable, we first decompose the original MDP into a series of per-VUE-pair MDPs. Then we propose a proactive algorithm based on long short-term memory and deep reinforcement learning techniques to address the partial observability and the curse of high dimensionality in local network state space faced by each VUE-pair. With the proposed algorithm, the RSU makes the optimal frequency band allocation and packet scheduling decision at each scheduling slot in a decentralized way in accordance with the partial observations of the global network state at the VUE-pairs. Numerical experiments validate the theoretical analysis and demonstrate the significant performance improvements from the proposed algorithm.

Network slicing is a proposing technology to support diverse services from mobile users (MUs) over a common physical network infrastructure. In this paper, we consider radio access network (RAN)-only slicing, where the physical RAN is tailored to accommodate both computation and communication functionalities. Multiple service providers (SPs, i.e., multiple tenants) compete with each other to bid for a limited number of channels across the scheduling slots, aiming to provide their subscribed MUs the opportunities to access the RAN slices. An eavesdropper overhears data transmissions from the MUs. We model the interactions among the non-cooperative SPs as a stochastic game, in which the objective of a SP is to optimize its own expected long-term payoff performance. To approximate the Nash equilibrium solutions, we first construct an abstract stochastic game using the channel auction outcomes. Then we linearly decompose the per-SP Markov decision process to simplify the decision- makings and derive a deep reinforcement learning based scheme to approach the optimal abstract control policies. TensorFlow-based experiments verify that the proposed scheme outperforms the three baselines and yields the best performance in average utility per MU per scheduling slot.

Network slicing is a key technology in 5G communications system, which aims to dynamically and efficiently allocate resources for diversified services with distinct requirements over a common underlying physical infrastructure. Therein, demand-aware allocation is of significant importance to network slicing. In this paper, we consider a scenario that contains several slices in one base station on sharing the same bandwidth. Deep reinforcement learning (DRL) is leveraged to solve this problem by regarding the varying demands and the allocated bandwidth as the environment state and action, respectively. In order to obtain better quality of experience (QoE) satisfaction ratio and spectrum efficiency (SE), we propose generative adversarial network (GAN) based deep distributional Q network (GAN-DDQN) to learn the distribution of state-action values. Furthermore, we estimate the distributions by approximating a full quantile function, which can make the training error more controllable. In order to protect the stability of GAN-DDQN's training process from the widely-spanning utility values, we also put forward a reward-clipping mechanism. Finally, we verify the performance of the proposed GAN-DDQN algorithm through extensive simulations.

This paper targets at the problem of radio resource management for expected long-term delay-power tradeoff in vehicular communications. At each decision epoch, the road side unit observes the global network state, allocates channels and schedules data packets for all vehicle user equipment-pairs (VUE-pairs). The decision-making procedure is modelled as a discrete-time Markov decision process (MDP). The technical challenges in solving an optimal control policy originate from highly spatial mobility of vehicles and temporal variations in data traffic. To simplify the decision-making process, we first decompose the MDP into a series of per-VUE-pair MDPs. We then propose an online long short-term memory based deep reinforcement learning algorithm to break the curse of high dimensionality in state space faced by each per-VUE-pair MDP. With the proposed algorithm, the optimal channel allocation and packet scheduling decision at each epoch can be made in a decentralized way in accordance with the partial observations of the global network state at the VUE-pairs. Numerical simulations validate the theoretical analysis and show the effectiveness of the proposed online learning algorithm.

To improve the quality of computation experience for mobile devices, mobile-edge computing (MEC) is a promising paradigm by providing computing capabilities in close proximity within a sliced radio access network (RAN), which supports both traditional communication and MEC services. Nevertheless, the design of computation offloading policies for a virtual MEC system remains challenging. Specifically, whether to execute a computation task at the mobile device or to offload it for MEC server execution should adapt to the time-varying network dynamics. This paper considers MEC for a representative mobile user in an ultradense sliced RAN, where multiple base stations (BSs) are available to be selected for computation offloading. The problem of solving an optimal computation offloading policy is modeled as a Markov decision process, where our objective is to maximize the long-term utility performance whereby an offloading decision is made based on the task queue state, the energy queue state as well as the channel qualities between mobile user and BSs. To break the curse of high dimensionality in state space, we first propose a double deep Q-network (DQN)-based strategic computation offloading algorithm to learn the optimal policy without knowing a priori knowledge of network dynamics. Then motivated by the additive structure of the utility function, a Q-function decomposition technique is combined with the double DQN, which leads to a novel learning algorithm for the solving of stochastic computation offloading. Numerical experiments show that our proposed learning algorithms achieve a significant improvement in computation offloading performance compared with the baseline policies.

With the cellular networks becoming increasingly agile, a major challenge lies in how to support diverse services for mobile users (MUs) over a common physical network infrastructure. Network slicing is a promising solution to tailor the network to match such service requests. This paper considers a system with radio access network (RAN)-only slicing, where the physical infrastructure is split into slices providing computation and communication functionalities. A limited number of channels are auctioned across scheduling slots to MUs of multiple service providers (SPs) (i.e., the tenants). Each SP behaves selfishly to maximize the expected long-term payoff from the competition with other SPs for the orchestration of channels, which provides its MUs with the opportunities to access the computation and communication slices. This problem is modelled as a stochastic game, in which the decision makings of a SP depend on the global network dynamics as well as the joint control policy of all SPs. To approximate the Nash equilibrium solutions, we first construct an abstract stochastic game with the local conjectures of channel auction among the SPs. We then linearly decompose the per-SP Markov decision process to simplify the decision makings at a SP and derive an online scheme based on deep reinforcement learning to approach the optimal abstract control policies. Numerical experiments show significant performance gains from our scheme.

Edge computing is an emerging concept based on distributed computing, storage, and control services closer to end network nodes. Edge computing lies at the heart of the fifth-generation (5G) wireless systems and beyond. While the current state-of-the-art networks communicate, compute, and process data in a centralized manner (at the cloud), for latency and compute-centric applications, both radio access and computational resources must be brought closer to the edge, harnessing the availability of computing and storage-enabled small cell base stations in proximity to the end devices. Furthermore, the network infrastructure must enable a distributed edge decision-making service that learns to adapt to the network dynamics with minimal latency and optimize network deployment and operation accordingly. This paper will provide a fresh look to the concept of edge computing by first discussing the applications that the network edge must provide, with a special emphasis on the ensuing challenges in enabling ultrareliable and low-latency edge computing services for mission-critical applications such as virtual reality (VR), vehicle-to-everything (V2X), edge artificial intelligence (AI), and so on. Furthermore, several case studies where the edge is key are explored followed by insights and prospect for future work.

Millimeter wave (mmWave) communications provide great potential for next-generation cellular networks to meet the demands of fast-growing mobile data traffic with plentiful spectrum available. However, in a mmWave cellular system, the shadowing and blockage effects lead to the intermittent connectivity, and the handovers are more frequent. This paper investigates an "all- mmWave" cloud radio access network (cloud-RAN), in which both the fronthaul and the radio access links operate at mmWave. To address the intermittent transmissions, we allow the mobile users (MUs) to establish multiple connections to the central unit over the remote radio heads (RRHs). Specifically, we propose a multipath transmission framework by leveraging the "all- mmWave" cloud-RAN architecture, which makes decisions of the RRH association and the packet transmission scheduling according to the time- varying network statistics, such that a MU experiences the minimum queueing delay and packet drops. The joint RRH association and transmission scheduling problem is formulated as a Markov decision process (MDP). Due to the problem size, a low-complexity online learning scheme is put forward, which requires no a priori statistic information of network dynamics. Simulations show that our proposed scheme outperforms the state-of- art baselines, in terms of average queue length and average packet dropping rate.

To improve the quality of computation experience for mobile devices, mobile-edge computing (MEC) is emerging as a promising paradigm by providing computing capabilities within radio access networks in close proximity. Nevertheless, the design of computation offloading policies for a MEC system remains challenging. Specifically, whether to execute an arriving computation task at local mobile device or to offload a task for cloud execution should adapt to the environmental dynamics in a smarter manner. In this paper, we consider MEC for a representative mobile user in an ultra dense network, where one of multiple base stations (BSs) can be selected for computation offloading. The problem of solving an optimal computation offloading policy is modelled as a Markov decision process, where our objective is to minimize the long-term cost and an offloading decision is made based on the channel qualities between the mobile user and the BSs, the energy queue state as well as the task queue state. To break the curse of high dimensionality in state space, we propose a deep Q-network-based strategic computation offloading algorithm to learn the optimal policy without having a priori knowledge of the dynamic statistics. Numerical experiments provided in this paper show that our proposed algorithm achieves a significant improvement in average cost compared with baseline policies.

Traffic prediction plays an important role in evaluating the performance of telecommunication networks and attracts intense research interests. A significant number of algorithms and models have been put forward to analyse traffic data and make prediction. In the recent big data era, deep learning has been exploited to mine the profound information hidden in the data. In particular, Long Short-Term Memory (LSTM), one kind of Recurrent Neural Network (RNN) schemes, has attracted a lot of attentions due to its capability of processing the long-range dependency embedded in the sequential traffic data. However, LSTM has considerable computational cost, which can not be tolerated in tasks with stringent latency requirement. In this paper, we propose a deep learning model based on LSTM, called Random Connectivity LSTM (RCLSTM). Compared to the conventional LSTM, RCLSTM makes a notable breakthrough in the formation of neural network, which is that the neurons are connected in a stochastic manner rather than full connected. We apply the RCLSTM to predict traffic and validate that the RCLSTM with even 35% neural connectivity still shows a satisfactory performance. When we gradually add training samples, the performance of RCLSTM becomes increasingly closer to the baseline LSTM. Moreover, for the input traffic sequences of enough length, the RCLSTM exhibits even superior prediction accuracy than the baseline LSTM.

How to allocate the limited wireless resource in dense radio access networks (RANs) remains challenging. By leveraging a software-defined control plane, the independent base stations (BSs) are virtualized as a centralized network controller (CNC). Such virtualization decouples the CNC from the wireless service providers (WSPs). We investigate a virtualized RAN, where the CNC auctions channels at the beginning of scheduling slots to the mobile terminals (MTs) based on bids from their subscribing WSPs. Each WSP aims at maximizing the expected long-term payoff from bidding channels to satisfy the MTs for transmitting packets. We formulate the problem as a stochastic game, where the channel auction and packet scheduling decisions of a WSP depend on the state of network and the control policies of its competitors. To approach the equilibrium solution, an abstract stochastic game is proposed with bounded regret. The decision making process of each WSP is modeled as a Markov decision process (MDP). To address the signalling overhead and computational complexity issues, we decompose the MDP into a series of single-agent MDPs with reduced state spaces, and derive an online localized algorithm to learn the state value functions. Our results show significant performance improvements in terms of per-MT average utility.

This paper addresses the problem of wireless resource scheduling in a vehicle-to-vehicle (V2V) communication network. The technical challenges lie in the fast changing network dynamics, namely, the channel quality and the data traffic variations. For a road segment covered by a road side unit (RSU), especially in a dense urban area, the vehicle density tends to be stable. The incoming service requests from the vehicle user equipment (VUE)-pairs compete with each other for the limited frequency resource in order to deliver data packets. Such competitions are regulated by the RSU via a sealed second-price auction at the beginning of scheduling slots. Each incumbent service request aims at maximizing the expected long-term payoff from bidding the frequency resource for packet transmissions. Markov perfect equilibrium (MPE) can be utilized to characterize the optimal competitive behaviors of the service requests. When the number of incumbent VUE-pairs becomes large, solving the MPE becomes infeasible. We adopt an oblivious equilibrium to approximate the MPE, which is theoretically proven to be error-bounded. The decision making process at each service request is hence transformed into a single-agent Markov decision process, for which we propose an on-line auction based learning scheme. Through simulation experiments, we show the potential performance gains from our proposed scheme, in terms of per-service request average utility.

Network virtualization facilitates radio access network (RAN) sharing by decoupling the physical network infrastructure from the wireless services. This paper considers a scenario in which a virtual network operator (VNO) leases wireless resources from a software-defined networking based virtualized RAN set up by a third-party infrastructure provider (InP). In order to optimize the revenue, the VNO explores jointly the delay tolerance in mobile traffic and the weak load coupling across the base stations (BSs) when making the resource scheduling decisions to serve its mobile users (MUs). The problem faced by the VNO can be straightforwardly transformed to the problem of minimizing the payments to the InP, which is formulated as a finite time horizon constrained Markov decision process (MDP). However, for a large-scale network with a huge number of MUs, the problem solving becomes extremely challenging. Through the dual decomposition approach, we decompose the problem into a series of per-MU MDPs, which can be solved distributedly. Moreover, the independence of channel conditions between a MU and the BSs is expected to further simplify solving each per-MU MDP. The simulations carried out in this paper show that our proposed scheme achieves minimal average payments compared with other existing approaches in literature.

The proof-of-concept software-defined radio access network (RAN) is not flexible enough due to the inherent delay and the necessity of high-capacity fronthaul links. We are hence motivated to propose a hierarchical software-defined RAN architecture, over which the base stations (BSs) are abstracted into multiple virtual local controllers while these local controllers are administered by a high-level controller. Under such a hierarchical network architecture, we particularly investigate in this paper how to adapt the BS transmit power over a long term according to the network dynamics under the constraints of mobile user queue stability and limited fronthaul capacity. We first formulate an off-line stochastic power adaptation problem. Through developing the Lyapunov method, we transform the problem into an approximate on-line optimization task. However, the challenge arises from the introduced per-cluster fronthaul capacity constraint. To solve the task efficiently and avoid extensive information exchange between the high-level controller and the local controllers, we put forward a novel low-complexity algorithm by designing a non- cooperative power adaptation game among the local controllers. Simulations are provided to evaluate the efficacy of the proposed studies.

Decentralized communication technologies (i.e., ad hoc networks) provide more opportunities for emerging wireless Internet of Things (IoT) due to the flexibility and expandability of distributed architecture. However, the performance degradation of wireless communications with the increase of the number of hops becomes the main obstacle in the development of decentralized wireless IoT systems. The main challenges come from the difficulty in designing a resource and energy efficient multihop communication protocol. Transmission control protocol (TCP), the most frequently used transport layer protocol for achieving reliable end-to-end communications, cannot achieve a satisfactory result in multihop wireless scenarios as it uses end-to-end acknowledgment which could not work well in a lossy scenario. In this paper, we propose a multihop data delivery virtualization approach which uses multiple one-hop reliable transmissions to perform multihop data transmissions. Since the proposed protocol utilizes hop-by-hop acknowledgment instead of end-to-end feedback, the congestion window size at each TCP sender node is not affected by the number of hops between the source node and the destination node. The proposed protocol can provide a significantly higher throughput and shorter transmission time as compared to the end-to-end approach. We conduct real-world experiments as well as computer simulations to show the performance gain from our proposed protocol.

With increasing mobile data services in the wireless communication networks, the LTE unlicensed is put forward as an effective way to offload the crowded traffic in the licensed spectrum. However, when there are multiple wireless cellular operators (WCOs) sharing the common unlicensed spectrum together, how to manage the competitions among all WCOs while guaranteeing the performance of the Wi-Fi networks still remains to be a technical challenge. In this paper, we model a layered power control game among all the WCOs and the Wi-Fi access point (WAP). In the game, we first fix the behaviors of all the other WCOs and propose the zero- determinant strategy for the power control of one considered WCO. Based on the predicted strategies of every WCO in all situations, all WCOs then play a non- cooperative game and determine their optimal strategies to achieve the Nash equilibrium results. Simulation results show the correctness of the analysis and the high performance gain achieved with the proposed zero- determinant strategies.

In this work, we investigate the problem of context-aware data caching in the heterogeneous small cell networks (HSCNs) to provide satisfactory to the end-users in reducing the service latency. In particular, we explore the storage capability of base stations (BSs) in HSCNs and propose a data caching model consists of edge caching elements (CAEs), small cell base stations (SBSs), and macro cell BS (MBS). Then, we concentrate on how to efficiently match the data contents to the different cache entities in order to minimize the overall system service latency. We model it as a distributed college admission (CA) stable matching problem and tackle this issue by utilizing contextual information to generate the preference lists of cache entities and contents, respectively. In the CA model, we leverage the resident-oriented Gale-Shapley (RGS) algorithm to find a stable matching between the contents and the cache entities. Through numerical results, we illustrate the advantages of of our proposed methods.

The ever increasing video demands from mobile users have posed great challenges to cellular networks. To address this issue, video caching in radio access networks (RANs) has been recognized as one of the enabling technologies in future 5G mobile networks, which brings contents near the end-users, reducing the transmission cost of duplicate contents, meanwhile increasing the Quality-of-Experience (QoE) of users. Inspired by the emerging software-defined networking technology, recent proposals have employed centralized collaborative caching among cells to further increase the caching capacity of the RAN. In this paper, we explore a new dimension in video caching in software-defined RANs to expand its capacity. We enable the controller with the capability to adaptively select the bitrates of videos received by users, in order to maximize the number and quality of video requests that can be served, meanwhile minimizing the transmission cost. To achieve this, we further incorporate Scalable Video Coding (SVC), which enables caching and serving sliced video layers that can serve different bitrates. We formulate the problem of joint video caching and scheduling as a reward maximization (cost minimization) problem. Based on the formulation, we further propose a 2-stage rounding-based algorithm to address the problem efficiently. Simulation results show that using SVC with collaborative caching greatly improves the cache capacity and the QoE of users.

The explosively growing demands for mobile traffic service bring both challenges and opportunities to wireless networks, among which, wireless network virtualization is proposed as the main evolution towards 5G. In this paper, we first propose a Software Defined Network (SDN) based wireless virtualization architecture for enabling multi-flow transmission in order to save capital expenses (CapEx) and operation expenses (OpEx) significantly with multiple Infrastructures Providers (InPs) and multiple Mobile Virtual Network Operators (MVNOs). We formulate the virtual resource allocation problem with diverse QoS requirements as a social welfare maximization problem with transaction cost. Due to the high computational complexity of formulated problem and hidden information of InPs and MVNOs for SDN controller, we introduce the shadow price for ensuring the desirable economic properties as well as the total welfare of system. Simulations are conducted with different system configurations to show the effectiveness of the proposed SDN based wireless virtualization framework and double auction mechanism.

Small cell networks are key components in 5G networks to boost the network capacity, improve spectrum and energy efficiency, and enable flexible and new services. Due to the flexible spectrum access among and flexible deployment of small cells, the inter- cell coordination becomes critical for the performance of the network. In this paper, based on the key concept in software defined networking (SDN) for Internet, we first introduce the network graph approach as a tool for the control and coordination among small cells. The network graph is constructed from the abstracted network state information extracted from underlying base stations. It shields the logical centralized control unit from implementation details of the underlying physical layer and thus reduces the control overhead in a centralized solution. We use the network graph for network energy saving in small cell networks, in which network graphs are used to decide the optimal set of small cells in the network. For cells outside this set we can switch them off for energy saving. We propose three types of network graphs with different network state details. Based on these graphs, we formulate the energy saving problem as an integer linear programming (ILP) problem, and propose the practical algorithms to solve the problem. The performance of the algorithms are studied by simulation. It shows the potential of the proposed network graph approach for the inter-cell resource coordination in small cell networks.

This paper addresses energy-aware traffic offloading in stochastic heterogeneous cellular networks (HCNs). The objective is to minimize energy consumption of the HCN while maintaining Quality-of-Service experienced by the mobile users. For each cell, the energy consumption depends on its associated system load, which is coupled with system loads in other cells due to the sharing over a common spectrum band. Such a traffic offloading problem is modeled by a discrete-time Markov decision process (DTMDP). Based on the traffic observations and the traffic offloading operations, the network controller learns to solve the optimal traffic offloading strategy with no prior knowledge of the DTMDP statistics. To deal with the curse of dimensionality, we design a centralized Q-learning with compact state representation algorithm, which is named as QC-learning. Moreover, a decentralized QC-learning algorithm is developed such that the macro-cell base stations (BSs) can independently manage the operations of small-cell BSs by making use of the network information obtained from the network controller. Simulations validate the proposed studies.

Designing an efficient multi-hop broadcast protocol is very important for the realization of collision avoidance systems and other many interesting applications in vehicular ad hoc networks (VANETs). Existing protocols are optimized for a specific scenario, and are not capable of working in various scenarios. Therefore, designing an intelligent protocol which can tune itself in relation to the change of network environment is particularly important. In this paper, we propose a broadcast protocol which is able to make forwarding decision based on a self-learning mechanism. The protocol employs a fuzzy logic-based relay node selection approach to take into account multiple metrics for the forwarding algorithm. The parameters used for the fuzzy logic are tuned online using a reinforcement learning approach. Transfer learning is used to transfer knowledge to new arriving vehicles (agents) in order to shorten the convergence time. The combination of reinforcement learning, transfer learning and fuzzy logic can provide an intelligent solution for broadcasting in VANETs. We conduct computer simulations to evaluate the proposed protocol.

Mobile data offloading has been introduced to alleviate the congestion of cellular networks and to improve the quality of service for mobile end users. This paper presents a distributed mechanism for mobile data offloading in software defined network (SDN) at the network edge. In SDN, the data traffic of base stations (BSs) can be dynamically offloaded to access points (APs), which is enabled by the SDN controller. The SDN controller formulates a revenue maximization problem to optimize the data offloading decision, and solves the problem in a fully distributed fashion. The proposed mechanism is based on the proximal Jacobian multi-block alternating direction method of multipliers (ADMM). BSs and APs perform the offloading decision update concurrently, and are coordinated by the SDN controller through dual variables to reach a consensus on the offloading demand and supply. Numerical simulations validate the effectiveness of the proposed algorithm.

This paper first provides a brief survey on existing traffic offloading techniques in wireless networks. Particularly as a case study, we put forward an online reinforcement learning framework for the problem of traffic offloading in a stochastic heterogeneous cellular network (HCN), where the time-varying traffic in the network can be offloaded to nearby small cells. Our aim is to minimize the total discounted energy consumption of the HCN while maintaining the quality-of-service (QoS) experienced by mobile users. For each cell (i.e., a macro cell or a small cell), the energy consumption is determined by its system load, which is coupled with system loads in other cells due to the sharing over a common frequency band. We model the energy-aware traffic offloading problem in such HCNs as a discrete-time Markov decision process (DTMDP). Based on the traffic observations and the traffic offloading operations, the network controller gradually optimizes the traffic offloading strategy with no prior knowledge of the DTMDP statistics. Such a model-free learning framework is important, particularly when the state space is huge. In order to solve the curse of dimensionality, we design a centralized Q-learning with compact state representation algorithm, which is named QC-learning. Moreover, a decentralized version of the QC-learning is developed based on the fact the macro base stations (BSs) can independently manage the operations of local small-cell BSs through making use of the global network state information obtained from the network controller. Simulations are conducted to show the effectiveness of the derived centralized and decentralized QC-learning algorithms in balancing the tradeoff between energy saving and QoS satisfaction.

Multihop data dissemination in vehicular ad hoc networks (VANETs) is very important for the realization of collision avoidance systems and many other interesting applications. However, designing an efficient data dissemination protocol for VANETs has been a challenging issue due to vehicle movements, limited wireless resources, and the lossy characteristics of wireless communication. In this paper, we propose a protocol that can provide a lightweight and reliable solution for data dissemination in VANETs. The protocol employs dynamically generated backbone vehicles to disseminate broadcast packets to reduce the MAC-layer contention time at each node while maintaining a high packet dissemination ratio by taking into account vehicle movement dynamics and the link quality between vehicles for the backbone selection. The protocol also uses network coding to reduce the protocol overhead and to improve the packet reception probability as compared with conventional approaches. We use theoretical analysis and computer simulations to show the advantage of the proposed protocol over other existing alternatives.

This paper proposes a distributed cooperative framework for improving the energy efficiency of green cellular networks. Based on the traffic load, neighboring base stations (BSs) cooperate to optimize the BS switching (sleeping) strategies so as to maximize the energy saving while guaranteeing users' minimal service requirements. The inter-BS cooperation is formulated following the principle of ecological self-organization. An interaction graph is defined to capture the network impact of the BS switching operation. Then, we formulate the problem of energy saving as a constrained graphical game, where each BS acts as a game player with the constraint of traffic load. The constrained graphical game is proved to be an exact constrained potential game. Furthermore, we prove the existence of a generalized Nash equilibrium (GNE), and the best GNE coincides with the optimal solution of total energy consumption minimization. Accordingly, we design a decentralized iterative algorithm to find the best GNE (i.e., the global optimum), where only local information exchange among the neighboring BSs is needed. Theoretical analysis and simulation results finally illustrate the convergence and optimality of the proposed algorithm.

Recent works have validated the possibility of improving energy efficiency in radio access networks (RANs), achieved by dynamically turning on/off some base stations (BSs). In this paper, we extend the research over BS switching operations, which should match up with traffic load variations. Instead of depending on the dynamic traffic loads which are still quite challenging to precisely forecast, we firstly formulate the traffic variations as a Markov decision process. Afterwards, in order to foresightedly minimize the energy consumption of RANs, we design a reinforcement learning framework based BS switching operation scheme. Furthermore, to speed up the ongoing learning process, a transfer actor-critic algorithm (TACT), which utilizes the transferred learning expertise in historical periods or neighboring regions, is proposed and provably converges. In the end, we evaluate our proposed scheme by extensive simulations under various practical configurations and show that the proposed TACT algorithm contributes to a performance jump start and demonstrates the feasibility of significant energy efficiency improvement at the expense of tolerable delay performance.

This paper investigates energy efficiency for the two-tier femtocell networks through combining game theory and stochastic learning. With the Stackelberg game formulation, a hierarchical reinforcement learning framework is developed to study the joint expected utility maximization of macrocells and femtocells. The macrocells behave as the leaders and the femtocells are followers during the learning procedure. At each time step, the leaders commit to dynamic strategies based on the best responses of the followers, while the followers compete against each other with no further information but the leaders' strategy information. In this paper, two learning algorithms are proposed to schedule each cell's transmission power. Numerical results are presented to validate the proposed studies and show that the two learning algorithms substantially improve the energy efficiency of the femtocell networks.

The cross- and co-tier interference creates the challenges to facilitate the concept of heterogeneous cellular networks (HCNs) in practice. In this paper, we establish a combined learning framework to autonomously mitigate the destructive interference. The macrocell is modeled as the leader and protects itself through pricing the interference from small-cells, which are the followers in the stochastic learning process. During each epoch (an epoch consists of T time slots), the leader commits to a pricing policy by knowing the resource allocation policies of all followers, while the followers compete against each other in each time slot only with the leader's price information. In general, for any two consecutive epochs, the HCN states are highly correlated. The previous policy information can thus be leveraged to improve the learning performance. Numerical results support that the proposed study substantially protects the macrocell and at the same time, optimizes the energy efficiency in small-cells.

This paper considers designing efficient medium access strategies for secondary users (SUs) to select frequency channels to sense and access in cognitive radio networks. The interaction among the SUs is considered as a learning problem, in which every SU behaves as an intelligent agent. Each SU believes that its competitors alter their future medium access strategies in proportion to its own current strategy change. These beliefs adapt in accordance with limited information exchange. In this way, each SU can obtain the behavior feature of other users through conjecture, optimize the medium access strategy, and finally achieve the goal of reciprocity, based on which two learning algorithms are proposed. We show that the SUs' stochastic behaviors and beliefs converge to a steady state under some conditions. Numerical results are provided to evaluate the performance of the two algorithms, and show that the achieved system performance gain outperforms some existing protocols.

As the scarce spectrum resource is becoming overcrowded, cognitive radio indicates great flexibility to improve the spectrum efficiency by opportunistically accessing the authorized frequency bands. One of the critical challenges for operating such radios in a network is how to efficiently allocate transmission powers and frequency resource among the secondary users (SUs) while satisfying the quality-of-service constraints of the primary users. In this paper, we focus on the noncooperative power allocation problem in cognitive wireless mesh networks formed by a number of clusters with the consideration of energy efficiency. Due to the SUs' dynamic and spontaneous properties, the problem is modeled as a stochastic learning process. We first extend the single-agent Q-learning to a multiuser context, and then propose a conjecture-based multiagent Q-learning algorithm to achieve the optimal transmission strategies with only private and incomplete information. An intelligent SU performs Q-function updates based on the conjecture over the other SUs' stochastic behaviors. This learning algorithm provably converges given certain restrictions that arise during the learning procedure. Simulation experiments are used to verify the performance of our algorithm and demonstrate its effectiveness of improving the energy efficiency.

Recent works have validated the possibility of energy efficiency improvement in radio access networks (RAN), depending on dynamically turn on/off some base stations (BSs). In this paper, we extend the research over BS switching operation, matching up with traffic load variations. However, instead of depending on the predicted traffic loads, which is still quite challenging to precisely forecast, we formulate the traffic variation as a Markov decision process (MDP). Afterwards, in order to foresightedly minimize the energy consumption of RAN, we adopt the actor-critic method and design a reinforcement learning framework based BS switching operation scheme. In the end, we evaluate our proposed scheme by extensive simulations under various practical configurations and prove the feasibility of significant energy efficiency improvement.

As energy saving and environmental protection become an inevitable trend, researchers need to shift their focus to “green” oriented architecture design. Recent advances in the area of cognitive radio (CR) have significant potential towards “green” communications. One of the critical challenges for operating CRs in a wireless mesh network is how to efficiently allocate transmission powers and frequency resource among the secondary users (SUs) while satisfying the quality-of-service constraints of primary users. Due to the SUs' intelligent and selfish properties, this paper focuses on the non-cooperative spectrum sharing in cognitive wireless mesh networks formed by a number of clusters. In order to study the competition behaviors of SUs in a dynamic environment, the problem is modeled as a stochastic learning process. We first extend the single-agent reinforcement learning (RL) to a multi-user context, based on which a conjecture based multi-agent RL algorithm is proposed. A rational SU learns the optimal transmission strategy from the conjecture over the other SUs' responses.

Cognitive wireless mesh networks have great flexibility to improve the spectrum utilization by opportunistically accessing the authorized frequency bands, within which the secondary users (SUs) should not violate the quality of service (QoS) requirement of the primary users (PUs) while transmitting. In this paper, we consider inter-cluster connection among neighboring clusters under the framework of cognitive wireless mesh networks. Corresponding to the neighboring clusters, all nodes operate in half-duplex mode; hence exchanging control message usually needs four time slots by traditional scheme, which leads to a loss in networking and spectral efficiency especially at the gateway node. A novel scheme based on network coding is proposed, which needs only two time slots. Our simulation experiments reveal the following findings: the performances of traditional inter-cluster connection and network coding based inter-cluster connection are comparable. Next, how to choose optimal signal amplification factor at the gateway node according to the wireless environment is discussed. And we present an intelligent policy based on reinforcement learning to solve the problem. Theoretical analysis and numerical results both show the policy can achieve optimal throughput for the SUs in the long run.