With the advancement of Mobile Edge Computing (MEC), effective solutions for communication scenarios, including the industrial Internet of Things (IoT) and the Internet of Vehicles (IoV), are becoming increasingly feasible. Unmanned aerial vehicles (UAVs) can further enhance flexibility in delivering computational services within MEC contexts. Addressing the urgent need for information freshness, we propose a three-tier IoV-MEC system, supported by multiple UAVs and a cloud center, aiming to minimize the system's average age of information (AoI). We propose a heterogeneous multi-agent reinforcement learning algorithm based on the actor-critic framework, where vehicles act as data sources, UAVs serve as edge devices, and the cloud acts as a control center. All three tiers learn interaction strategies cooperatively based on the observations. To further enhance system performance, we implement an efficient federated learning method, allowing same-tier agents to share learning parameters, thus improving system performance and convergence speed. Extensive simulation results demonstrate that the proposed algorithm outperforms baseline algorithms in terms of average AoI and convergence speed.

Industrial control networks are frequent targets of cyber attacks, calling for autonomous defense strategies to combat threats effectively and promptly. We propose to leverage Reinforcement Learning (RL) to train defenders how to select the most appropriate actions in response to attackers’ behavior. We model the defender and attacker as agents in RL environments and define action space and reward function. We focus on evaluating value-based and policy gradient-based RL algorithms and propose enhancements to Proximal Policy Optimization (PPO) method through normalization. We then leverage CybORG, a simulation tool to study how the RL algorithms perform to achieve autonomous defense. Our extensive experiments reveal that the proposed normalized PPO outperforms other models in terms of stability, robustness, and average rewards. The results also demonstrate the effectiveness of the defender in protecting the operational server upon changing the attacker’s position. Additionally, increasing the number of attackers had a significant impact on policy-based algorithms, particularly PPO, with decreased reward values highlighting the increased vulnerability of the network.

In this paper, we investigate jamming-resilient UAV path planning strategies for data collection in Internet of Things (IoT) networks, in which the typical UAV can learn the optimal trajectory to elude such jamming attacks. Specifically, the typical UAV is required to collect data from multiple distributed IoT nodes under collision avoidance, mission completion deadline, and kinematic constraints in the presence of jamming attacks. We first design a fixed ground jammer with continuous jamming attack and periodical jamming attack strategies to jam the link between the typical UAV and IoT nodes. Defensive strategies involving a reinforcement learning (RL) based virtual jammer and the adoption of higher SINR thresholds are proposed to counteract against such attacks. Secondly, we design an intelligent UAV jammer, which utilizes the RL algorithm to choose actions based on its observation. Then, an intelligent UAV anti-jamming strategy is constructed to deal with such attacks, and the optimal trajectory of the typical UAV is obtained via dueling double deep Q-network (D3QN). Simulation results show that both non-intelligent and intelligent jamming attacks have significant influence on the UAV’s performance, and the proposed defense strategies can recover the performance close to that in no-jammer scenarios.

In this paper, we investigate jamming-resilient unmanned aerial vehicle (UAV) path planning strategies for data collection in Internet of Things (IoT) networks, in which the typical UAV can learn the optimal trajectory to elude such jamming attacks. Specifically, the typical UAV is required to collect data from multiple distributed IoT nodes under collision avoidance, mission completion deadline, and kinematic constraints in the presence of jamming attacks. We first design an intelligent UAV jammer, which utilizes reinforcement learning to choose actions based on its observation. Then, an intelligent UAV anti-jamming strategy is constructed to deal with such attacks, and the optimal trajectory of the typical UAV is obtained via dueling double deep Q-network (D3QN). Simulation results show that the intelligent jamming attack has great influence on the UAV's performance, and the proposed defense strategy can recover the performance close to that in no-jammer scenarios.

Lithium-ion batteries (LIBs) will accelerate the degradation of capacity after long-term cycling, showing nonlinear aging features. The onset of the nonlinear aging feature is called the “knee-point”. The appearance of nonlinear aging will not only lead to a rapid drop in the overall performance of LIBs, but also the serious collapse of battery safety, which makes the identification and early prediction of nonlinear aging knee-point particularly important. In this paper, we propose a nonlinear aging knee-point prediction method of LIBs based on sliding window with stacked long short-term memory (LSTM) neural network. We compared our method with other common machine learning algorithms, and found that the prediction results of knee-point under this method are significantly better than other algorithms. When the sliding window size is 20, the root mean square error (RMSE) of prediction result is 72.3 and the mean absolute error (MAE) is 51.6. In addition, we further study the effect of different sliding window sizes on the prediction results. By predicting the knee-point, it can be recognized when the nonlinear aging begins so that the user can be reminded whether to replace the battery, which greatly reduces the risk of battery safety problems.

Unmanned aerial vehicles (UAVs) are expected to be an integral part of wireless networks, and determining collision-free trajectory in multi-UAV noncooperative scenarios while collecting data from distributed Internet of Things (IoT) nodes is a challenging task. In this article, we consider a path-planning optimization problem to maximize the collected data from multiple IoT nodes under realistic constraints. The considered multi-UAV noncooperative scenarios involve a random number of other UAVs in addition to the typical UAV, and UAVs do not communicate or share information among each other. We translate the problem into a Markov decision process (MDP) with parameterized states, permissible actions, and detailed reward functions. Dueling double deep$Q$-network (D3QN) is proposed to learn the decision-making policy for the typical UAV, without any prior knowledge of the environment (e.g., channel propagation model and locations of the obstacles) and other UAVs (e.g., their missions, movements, and policies). The proposed algorithm can adapt to various missions in various scenarios, e.g., different numbers and positions of IoT nodes, different amount of data to be collected, and different numbers and positions of other UAVs. Numerical results demonstrate that real-time navigation can be efficiently performed with high success rate, high data collection rate, and low collision rate.

Cross-network anchor link discovery is an important research problem and has many applications in heterogeneous social network. Existing schemes of cross-network anchor link discovery can provide reasonable link discovery results, but the quality of these results depends on the features of the platform. Therefore, there is no theoretical guarantee to the stability. This article employs user embedding feature to model the relationship between cross-platform accounts, that is, the more similar the user embedding features are, the more similar the two accounts are. The similarity of user embedding features is determined by the distance of the user features in the latent space. Based on the user embedding features, this article proposes an embedding representation-based method Con&Net(Content and Network) to solve cross-network anchor link discovery problem. Con&Net combines the user’s profile features, user-generated content (UGC) features, and user’s social structure features to measure the similarity of two user accounts. Con&Net first trains the user’s profile features to get profile embedding. Then it trains the network structure of the nodes to get structure embedding. It connects the two features through vector concatenating, and calculates the cosine similarity of the vector based on the embedding vector. This cosine similarity is used to measure the similarity of the user accounts. Finally, Con&Net predicts the link based on similarity for account pairs across the two networks. A large number of experiments in Sina Weibo and Twitter networks show that the proposed method Con&Net is better than state-of-the-art method. The area under the curve (AUC) value of the receiver operating characteristic (ROC) curve predicted by the anchor link is 11% higher than the baseline method, and Precision@30 is 25% higher than the baseline method.

Unmanned aerial vehicles (UAVs) are expected to be an integral part of wireless networks, and determining collision-free trajectories for multiple UAVs while satisfying requirements of connectivity with ground base stations (GBSs) is a challenging task. In this paper, we consider non-cooperative multi-UAV scenarios, in which multiple UAVs need to fly from initial locations to destinations, while satisfying collision avoidance, wireless connectivity, and kinematic constraints. We aim to find trajectories for the UAVs with the goal to minimize their mission completion time. We first formulate the multi-UAV trajectory optimization problem as a sequential decision making problem. We, then, propose a decentralized deep reinforcement learning approach to solve the problem. More specifically, a value network is developed to obtain values given the agent’s joint state (including the agent’s information, the nearby agents’ observable information, and the locations of the nearby GBSs). A signal-to-interference-plus-noise ratio (SINR)-prediction neural network is also designed, using accumulated SINR measurements obtained when interacting with the cellular network, to map the GBSs’ locations into the SINR levels in order to predict the UAV’s SINR. Numerical results show that with the value network and SINR-prediction network, real-time navigation for multi-UAVs can be efficiently performed in various environments with high success rate.

In this paper, a three-tier heterogeneous network (HetNet) is considered, where access points (APs), small-cell base stations (SBSs) and macrocell BSs (MBSs) transmit in terahertz (THz), millimeter wave (mmWave), microwave frequency bands, respectively. Distinguishing features of transmission in each frequency band are taken into account, including the blockage model, path loss, beamforming and small-scale fading. Path loss based association criterion is considered for user equipments (UEs). By using tools from stochastic geometry, the complementary cumulative distribution function (CCDF) of the received signal power, the Laplace transform of the aggregate interference, and the SINR coverage probability are investigated, and general expressions are obtained. Finally, numerical results show that making the THz APs more densely distributed can enhance the received signal power but decrease the SINR coverage probability.

Unmanned aerial vehicles (UAVs) are expected to be an integral part of wireless networks, and determining collision-free trajectories in multi-UAV non-cooperative scenarios is a challenging task. In this paper, we consider a path planning optimization problem to maximize the collected data from multiple Internet of Things (IoT) nodes under realistic constraints. The considered multi-UAV non-cooperative scenarios involve random number of other UAVs in addition to the typical UAV, and UAVs do not communicate with each other. We translate the problem into an Markov decision process (MDP). Dueling double deep Q-network (D3QN) is proposed to learn the decision making policy for the typical UAV, without any prior knowledge of the environment (e.g., channel propagation model and locations of the obstacles) and other UAVs (e.g., their missions, movements, and policies). Numerical results demonstrate that real-time navigation can be efficiently performed with high success rate, high data collection rate, and low collision rate.

Unmanned aerial vehicles (UAVs) are expected to be an integral part of wireless networks, and determining collision-free trajectories for multiple UAVs while satisfying requirements of connectivity with ground base stations (GBSs) is a challenging task. In this paper, we first reformulate the multi-UAV trajectory optimization problem with collision avoidance and wireless connectivity constraints as a sequential decision making problem in the discrete time domain. We, then, propose a decentralized deep reinforcement learning approach to solve the problem. More specifically, a value network is developed to encode the expected time to destination given the agent’s joint state (including the agent’s information, the nearby agents’ observable information, and the locations of the nearby GBSs). An SINR-prediction network is also designed, using accumulated SINR measurements obtained when interacting with the cellular network, to map the GBSs’ locations into the SINR levels in order to predict the UAV’s SINR. Numerical results show that with the value network and SINR-prediction network, real-time navigation for multi-UAVs can be efficiently performed in various environments with high success rate.

In this paper, a novel synchronous Q-learning method is proposed for solving discrete-time linear quadratic regulator (LQR) problems. To begin with, the Bellman equation corresponding to the optimal Q-function is reformulated into a consistency equation on the parameters of the optimal Q-function and the optimal controller. Then an actor-critic structure is introduced to learn the optimal Q-function and the optimal controller online in real time by using the state samples generated by the behavior policy. Particularly, the proposed synchronous Q-learning scheme simultaneously updates the Q-function approximation and the optimal controller approximation, rather than iterating between policy evaluation and policy improvement. The proposed control scheme is proved to be uniformly ultimately bounded (UUB) under appropriate learning rates, provided that certain persistence of excitation (PE) conditions are satisfied. Besides, the PE conditions can be easily met by injecting appropriate exploration noise into the behavior policy without causing any excitation noise bias. Finally, one simulation example is provided to verify the effectiveness of the proposed synchronous Q-learning method.

In this paper, we consider simultaneous information and energy transfer (SWIPT) in unmanned aerial vehicle (UAV)-assisted cellular Internet of Things (IoT) networks, in which the user equipment (UE) locations are modeled as Thomas cluster processes. A realistic air-to-ground communication model is incorporated into the analysis. In particular, different line of sight (LOS) and non-LOS (NLOS) path loss models are considered for the links from the UAVs to UEs and ground base stations (GBSs) to UEs. Three dimensional (3D) antenna patterns are adopted, e.g., a doughnut-shaped antenna radiation pattern is considered for UAVs and a combination of horizontal and vertical antenna pattern is utilized for GBSs. In addition, we employ the power splitting technique in the SWIPT scenario, which allows the UEs to harvest energy and decode information simultaneously using the same received signal. Association probability and energy coverage probability of the UAVs and GBSs are determined. Moreover, an analysis of the successful transmission probability which jointly addresses the energy and signal-to-interference-plus-noise ratio (SINR) coverages is provided. Finally, performance is further investigated via numerical results.

Beamforming is one of the key techniques in millimeter wave (mmWave) multi-input multi-output (MIMO) communications. Designing appropriate beamforming not only improves the quality and strength of the received signal, but also can help reduce the interference, consequently enhancing the data rate. In this paper, we propose a distributed multi-agent double deep Q-learning algorithm for beamforming in mmWave MIMO networks, where multiple base stations (BSs) can automatically and dynamically adjust their beams to serve multiple highly-mobile user equipments (UEs). In the analysis, largest received power association criterion is considered for UEs, and a realistic channel model is taken into account. Simulation results demonstrate that the proposed learning-based algorithm can achieve comparable performance with respect to exhaustive search while operating at much lower complexity.

In this paper, we jointly consider the downlink simultaneous wireless information and power transfer (SWIPT) and uplink information transmission in unmanned aerial vehicle (UAV)-assisted millimeter wave (mmWave) cellular networks, in which the user equipment (UE) locations are modeled using Poisson cluster processes (e.g., Thomas cluster processes or Matérn cluster processes). Distinguishing features of mmWave communications, such as different path loss models for line-of-sight (LOS) and non-LOS (NLOS) links and directional transmissions are taken into account. In the downlink phase, the association probability, and energy coverages of different tier UAVs and ground base stations (GBSs) are investigated. Moreover, we define a successful transmission probability to jointly present the energy and signal-to-interference-plus-noise ratio (SINR) coverages and provide general expressions. In the uplink phase, we consider the scenario that each UAV receives information from its own cluster member UEs. We determine the Laplace transform of the interference components and characterize the uplink SINR coverage. In addition, we formulate the average uplink throughput, with the goal to identify the optimal time division multiplexing between the donwlink and uplink phases. Through numerical results we investigate the impact of key system parameters on the performance. We show that the network performance is improved when the cluster size becomes smaller. In addition, we analyze the optimal height of UAVs, optimal power splitting value and optimal time division multiplexing that maximizes the network performance.

A K-tier heterogeneous mmWave uplink cellular network with user-centric small cell deployments is considered in this paper. In particular, the user equipments (UEs) are assumed to be clustered around small-cell base stations (BSs) according to the Thomas cluster process. Practical line-of-sight (LOS) and none-line-of-sight (NLOS) models are adopted with different parameters for different tiers. In this setting, we consider a coupled association probability model, and derive the probability that the BSs of each tier are associated with UEs from each cluster. Following the identification of the association probabilities, we characterize the Laplace transforms of the inter-cell interferences in closed-form. Using tools from stochastic geometry, we provide a general expression of the SINR coverage probability in each tier. Via numerical and simulation results, we confirm the analysis and the derived expressions, and investigate the impact of important system parameters.

A K-tier heterogeneous downlink millimeter wave (mmWave) cellular network with user-centric small cell deployments is studied in this paper. In particular, we consider a heterogeneous network model with user equipments (UEs) being deployed according to a Poisson Cluster Process (PCP), i.e., Thomas cluster process, where the UEs are clustered around the base stations (BSs) and the distances between UEs and the BS are modeled as Gaussian distributed. In addition, distinguishing features of mmWave communications including directional beamforming and a sophisticated path loss model incorporating both line-of-sight (LOS) and non-line-of-sight (NLOS) transmissions, are taken into account. In this paper, the complementary cumulative distribution function (CCDF) and probability density function (PDF) of the path loss are provided. Also, using tools from stochastic geometry, we derive a general expression of the signal-to-interference-plus-noise ratio (SINR) coverage probability. Our results demonstrate that coverage probability can be improved by decreasing the size of UE clusters around BSs, and interference has noticeable influence on the coverage performance of our model.

Hadoop is one of the most important Big Data processing and storage systems. In recent years, a lot of efforts have been put to enhance Hadoop's performance from networking perspectives. However, there are limited tools that can help researchers to verify their networking algorithm design in terms of Hadoop's performance. This paper proposes Doopnet which is a framework and toolset for creating Hadoop clusters in a virtualized environment and for monitoring/analysing of Hadoop's networking characteristics under different network configurations. Doopnet enables users to automatically set up a Hadoop cluster over Docker containers running inside Mininet. The Hadoop traffic is collected inside the containers and virtual switches through network flow monitors. The users can easily modify network topologies or configurations through Mininet, observe the networking behaviour through network flow monitors, and analyse the effects of different network settings on Hadoop's performance. Examples are presented to demonstrate how to setup the Doopnet testbed and analyse Hadoop traffic.