Reinforcement-learning-(RL)-based advanced persistent threats (APTs) defense schemes choose the scan interval to enhance the detection accuracy, and the data protection level can be further improved by optimizing the repair rate corresponding to methods, such as malicious files removal, security software version update and passwords reset, each requiring different computational resources, such as CPUs. In this article, we propose an RL-based APT defense scheme to optimize both the continuous scan interval of metering data and repair rate to mitigate the potential loss for meter data management systems (MDMSs) in smart grids with large-scale meters. Based on the size of the metering data stored at MDMS and the compromised data, the data tag granularity and the number of CPUs for repairing, neural networks extract the state feature, address the quantization error of defense policy, and update the weights based on the APT defense experiences and the shared weights of neighboring MDMSs to improve the defense performance as a weighted sum of the defense duration, detection accuracy and data protection level. The computational complexity of the proposed scheme and the performance bounds according to the Nash Equilibrium of the game between the MDMS and the APT attacker are provided. Simulation results based on three MDMSs that receive the metering data from 50–150 smart meters and an APT attacker with the selected attack interval up to 5 s show the performance gain over the benchmark based on the optimal control theory and Q-learning in large-scale smart grids.

Collaborative perception in vehicular networks enables the connected autonomous vehicle (CAV) to gather sensing data, such as feature maps of light detection and ranging (LiDAR) point clouds, from neighboring CAVs to achieve higher perception accuracy, which has performance degradation against data fabrication attacks that share falsified sensing data with random probability. In this paper, we exploit the spatial consistency check to detect the potentially manipulated regions in LiDAR point clouds and measure the inconsistency degree of the received sensing data based on the number of conflict regions, which is the basis for determining the falsified sensing data if the inconsistency degree exceeds the threshold of the hypothesis test. The reinforcement learning (RL)-based collaborative vehicular perception scheme against data fabrication attacks is further proposed to choose CAVs based on the inconsistency degrees, the data quality measured by the confidence scores, the channel gains and the CAV reputations, which enhances the utility as the weighted sum of perception accuracy, speed and minimum latency requirement for data transmission. In addition, the multi-layer perceptron-based neural networks extract the perception features of sensing data from historical experiences, such as the data quality of received feature maps, as well as compress the RL state that linearly increases with the network scales and the spatial granularity of LiDAR point clouds for faster learning. Experimental results based on 10 CAVs equipped with LiDAR sensors and NVIDIA computational units to detect 20 vehicles against data fabrication attacks show that our proposed scheme outperforms the benchmarks in terms of perception accuracy and speed.

Collaborative perception of connected autonomous vehicles (CAVs) that offload the sensing data, such as the feature map extracted from light detection and ranging (LiDAR) point clouds, to an edge device such as the roadside unit (RSU) to detect traffic objects has severe performance degradation due to the offloading latency and packet loss rate (PLR) under jamming and interference. In this paper, we propose an edge-assisted reinforcement learning (RL)-based collaborative perception scheme for CAVs to enhance the accuracy and speed against jamming and interference in LiDAR-based object detection. Based on the spatial confidence score of the feature map, the data size, the channel gains, the received jamming power and interference level, this scheme chooses the critical regions of the feature map, radio channel and transmit power with the hierarchical structure to enhance the learning efficiency. The risk level of the selected policy evaluates the time asynchronization and information loss of the shared feature map using the multi-level risk function based on multiple thresholds of the offloading latency and PLR, with assigning different penalties to mitigate the selection of high-risk policies that degrade perception performance. The upper performance bound in terms of the perception accuracy, latency and utility is provided based on the Stackelberg equilibrium of the game between the jammer and CAVs. Experimental results based on the Robosense RS-LiDAR-16 sensors and the Raspberry Pi to detect 10 vehicles in an$8.5\times 4\times 3.5$m3area show the performance gain with 22.4% higher perception accuracy and 41.3% less latency compared with the benchmark against a smart jammer.

Reinforcement learning (RL)-based collaborative perception in vehicular networks chooses the sub-frame of radio channel resources for connected autonomous vehicles (CAVs) to exchange sensing data to enhance the perception performance, but leads to inaccurate detection in the light detection and ranging (LiDAR)-based object detection due to the asynchronous scan period of the LiDAR point clouds. This paper proposes a RL-based collaborative perception scheme to choose the transmit power and sub-frame to share the feature maps extracted from point clouds. Based on the estimated packet timestamp, the network topology and the channel gains among CAVs, this scheme enhances the perception accuracy and latency against path-loss and interference. The collaborative risk in the policy distribution is formulated as a weighted sum of the perception latency and packet loss rate to avoid the time asynchronization and information loss of the feature map exchange. The performance bound of the perception accuracy and latency is provided based on a Nash equilibrium of the cooperative game among CAVs. Simulation results based on five CAVs show the performance gain of the perception accuracy and latency over the benchmarks.

Both the radio spectrum features such as power spectral density (PSD) and the communication performance such as packet loss rate (PLR) can be exploited to detect jamming attacks, with the resulting detection results used to enhance the reliability of wireless communications. In this paper, we propose a reinforcement learning (RL)-based jamming detection scheme based on the channel energy, the received signal strength indi-cator of each packet, the channel gains, PLR and transmission latency of mobile devices, in which the test threshold and the number of PSD bins are optimized by access point to enhance the utility as a weighted function of the detection speed and accuracy. The detection results are exploited for mobile devices to choose the transmit power and channel to reduce the PLR and transmission latency. Experimental results based on the universal software radio peripheral and Raspberry Pi to detect four jamming types including constant, sweeping, random and smart jamming show that our proposed schemes improve the detection accuracy and speed, as well as the communication performance.

The ultra-high speed (e.g., typically moving with 10–20 Mach) of the hypersonic vehicle (HSV) causes a plasma sheath, which severely degrades the communication performance and results in communication blackouts. In this paper, we propose a deep reinforcement learning (RL)-based HSV reliable communications scheme against jamming, which enables the HSV to select the carrier frequency and transmit power according to the signal quality, the estimated voltage standing wave ratio, flight altitude, flight speed, and angle of attack. Specifically, we design a deep two-level hierarchical structure to compress the high-dimensional state and action space, with the added advantage of leveraging transfer learning to reduce initial exploration and expedite the optimization process. To optimize the learning speed, the dueling architecture is implemented in the deep network to measure the state value and the advantage function of the policies. In contrast to the benchmark, the simulation results indicate that the proposed scheme yields a significant reduction in both bit error rate and transmit power.

Digital twin systems (DTs) are susceptible to active eavesdroppers engaging in wiretapping and jamming activities, aimed at increasing the physical layer's transmit power to steal additional virtual information. In this paper, we propose a deep reinforcement learning-based friendly jamming method for intratwin communications in DTs that enable the friendly jammer to optimize jamming frequency, power and the jamming duration against active eavesdropping. A safe and hierarchical architecture is designed that utilizes information such as the channel state of the device-server and the hostile jamming strength or wiretap channel of the active eavesdropper to improve anti-eavesdropping performance and secrecy rate. We apply the proposed friendly jamming method using universal software radio peripherals and assess its performance through experimentation. The experimental results illustrate that the proposed strategies significantly enhance the DTs secrecy rate in cross-layer transmission, and reduce the eavesdropping data rate and the physical layer energy consumption compared to existing friendly jamming methods.

The jamming and interference in vehicular ad hoc networks (VANETs) depend on the channel states of vehicles from the ambient radio transmitter, which in turn result from the topologies and radio features. In this paper, we propose an environment-aware reinforcement learning (RL)-based VANET communication scheme against jamming and interference that applies the post decision state algorithm to optimize the power allocation and channel selection without relying on the jamming attack model. This scheme exploits the environment information in the state formulation due to the traffic density and their locations reflect the interference level, as well as the location of transmission vehicle combined with building structure and heights indicate the channel gain and shadowing. The proposed post decision state-based RL method employs the estimated future communication distances of the moving vehicles to accelerate the learning process. We provide the performance bounds of the energy consumption, bit error rate (BER), and utility based on a Nash equilibrium. Simulation results show that the proposed scheme significantly reduces the BER with less energy consumption compared with the benchmark.

Online video becomes a significant service in daily life, and it usually adopts a caching and playing mechanism. Due to high mobility and changeable topology, challenges of video downloading still exist in vehicular networks, especially in areas where the roadside units (RSUs) are not fully covered. The flexible deployment of the unmanned aerial vehicle (UAVs) compensate for the lack of RSU coverage, and thus this paper considers that a cyclic flight UAV to assist RSUs in providing video download services for vehicles. With the help of UAV, seamless communication coverage and stable transmission links ensure better service quality for vehicles. In addition, we propose a model-free algorithm based on a deep Q network to find the optimal UAV decision policy to achieve the minimized stalling time. Finally, the simulation results are given to demonstrate that the proposed solution can effectively maintain a high-quality user experience.