This paper studies the anti-jamming channel selection problem in the unmanned aerial vehicle (UAV) communication scenario using machine learning. Recently, deep reinforcement learning (DRL) based anti-jamming approaches have drawn much attention, but most of them require lots of computing resources and power supply for training, which is impractical for the hardware-limited UAVs. What's more, the high complexity of DRL-based algorithms weakens their online learning ability, failing to rapidly adapt to the changing jamming environment. To be applicable to the hardware-limited UAVs, we propose a lightweight reinforcement learning algorithm based on the idea of spectrum state abstraction. We first assign similar spectrum states to clusters using the DRL and clustering algorithms. A state clustering network is deployed in the UAV to convert the large and redundant state space into a small number of state clusters. Based on the clustered states, the UAV uses a simple tabular Q-learning algorithm to online find the optimal anti-jamming policy. The simulation results show that, compared with the conventional DRL approach, the proposed algorithm can efficiently find the optimal anti-jamming policy and fast adapt to the change of jamming pattern in the complicated and dynamic jamming environment.

Sensor deployment and spectrum data completion are two critical stages in radio map construction, which are intricately linked and jointly influence both efficiency and performance. However, existing research often addresses these two issues separately, neglecting their joint optimization. In this study, we introduce two heuristic algorithms to tackle the joint optimization problem of these issues. These algorithms utilize a novel approach that combines deterministic search and fine-grained stochastic optimization, guided by the principle of greedy optimization. Experimental results demonstrate distinct advantages of the proposed algorithms compared to the baselines. Moreover, transferability verification confirms the applicability of the proposed algorithms across various scenarios and their ability for knowledge transfer.

Abstract Although unmanned aerial vehicle (UAV) relay can provide auxiliary communication due to its flexible mobility, it is vulnerable to jamming attacks. This paper considers the UAV relay anti‐jamming communication issue under the threat of a malicious jammer with beam‐forming jamming capability. To prevent the relay link from deteriorating, UAV trajectory adjustment and transmission mode switching between half‐duplex and full‐duplex are two available schemes, while they will incur the additional flying costs and continuous mode switching, respectively. To balance the trade‐off between trajectory optimization and mode selection, this paper investigates the joint trajectory optimization and mode selection anti‐jamming approach. First, an anti‐jamming utility considering the cost‐efficient and end‐to‐end capacity gains is designed. Second, to model the bounded rationality of both the UAV relay and the jammer due to the adversarial context, a biased Stackelberg game to analyse the competitive system interactions is proposed. Moreover, the existence of Stackelberg equilibrium (SE) in the problem is proved. Finally, a joint mode selection and trajectory optimization (JMSTO) algorithm based on the multi‐armed bandit is proposed to obtain the SE. It is further demonstrated that the JMSTO algorithm has a logarithmic regret. The results show that our proposed JMSTO algorithm is superior to non‐joint optimization methods.

In this paper, we study the problem of network reconfiguration when unmanned aerial vehicle (UAV) swarms suffer damage. Multiple UAVs are divided into several groups to perform various tasks. Each master UAV is connected to the ground control station and provides network services for small UAVs that perform various tasks, ensuring that the information of small UAVs can be transmitted back in a timely manner. When master UAVs are destroyed due to factors such as jamming or attacks, the associated small UAVs must select new master UAVs for network service and cooperate with other small UAVs to execute tasks. Based on the heterogeneity and relevance of tasks, we model and analyze the task relationship among different UAVs. Since both master UAVs and small UAVs have respective optimization objectives in the network reconfiguration process, we construct a many-to-one bilateral matching market to model the interaction between master UAVs and small UAVs. To realize an efficient solution for UAV network reconfiguration in complex environments, we propose a distributed matching algorithm and prove that the algorithm can converge to two-sided stable matching. Simulation results indicate that the proposed algorithm can significantly improve the task completion degree of the network compared with three other algorithms.

Here the intelligent jammer issue is studied. With the rapid development of cognitive radio technology, current cognitive terminals can adaptively or intelligently switch channel by spectrum sensing and decision‐making. Most of the traditional jamming methods, such as swept jamming and comb jamming, generally work in a relatively fixed pattern, which are not able to effectively jam the terminals empowered with cognition and spectrum decision‐making capability. In view of this problem, the authors propose an intelligent jamming decision‐making system based on reinforcement learning. First, in order to jam a pair of transmitter and receiver with adaptive frequency hopping capability, a jammer with spectrum sensing, offline training and learning scheme is proposed. Second, a reinforcement learning‐based algorithm for jamming decision‐making is proposed and simulated. A special feature of the proposed scheme is that considering the reward is difficult to obtain in the actual communication system, a virtual jamming decision‐making method is used to enable the jammer to learn and jam efficiently without the user's prior information. Finally, the proposed jamming model and algorithm are implemented and verified on Universal software radio peripheral testbed.

Abstract In heterogeneous wireless networks, network selection algorithms provide the user with the optimum network access choice. The optimal network is evaluated according to network parameters. Considering that the network parameters are dynamic and unavailable for the user in realistic heterogeneous wireless network environments, most existing network selection algorithms cannot work effectively. Learning‐based algorithms can address the problem of uncertain network parameters, while they commonly need considerable network handoff, resulting in unbearable handoff cost. In order to tackle the uncertainty of network parameters, we formulate the network selection problem as a multi‐armed bandit problem. Moreover, two online learning‐based network selection algorithms with a special consideration on reducing network handoff cost are proposed. By updating in a block manner, both algorithms achieve optimal logarithmic‐order regret and limited network handoff cost. The simulation indicates that the two algorithms can significantly reduce the network handoff cost and improve the transmission performance compared with existing algorithms, simultaneously. Copyright © 2014 John Wiley & Sons, Ltd.

Radio resource management (RRM) is crucial for improving resource utilization in heterogeneous wireless networks. Existing work attempts to exploit the network diversity to gain throughput improvement for users, which, however, neglects the impact of user demand on RRM. Armed with the idea that the ultimate goal of communications is to serve users with personalized demand, we introduce another dimension of potential performance gain, user demand diversity gain. This gain derives from the elaborate matching between user demand and radio resource, which can not be directly attained in existing throughput-centric optimization due to users' blindness in maximizing throughput. Aiming at obtaining this gain, we propose the user demand-centric optimization, where users seek to maximize quality of experience (QoE), instead of throughput. This shift enables us to propose a novel game formulation, QoE game. We derive the condition on the existence of the QoE equilibrium, validate the user demand diversity gain and propose a distributed QoE equilibrium learning algorithm. Finally, a cloud assisted learning framework is proposed to accommodate the learning algorithm with significantly reduced cost. Simulation results validate the existence of user demand diversity gain and the effectiveness of the proposed learning algorithm in improving the system efficiency and QoE fairness.

In opportunistic channel access, the user needs to make real time decisions on when and which channel to access with uncertainty. Assuming perfect channel statistics, several studies have applied optimal stopping theory to derive control strategy for sequential sensing/probing based opportunistically accessing (s-SPA), exploiting temporary opportunities among multiple channels. Meanwhile, numerous multi-arm bandit (MAB)-based approaches have been proposed for online learning of channel selection in periodical sensing/accessing system, however, these schemes fail to exploit the opportunistic diversity in short term. In this paper, we investigate online learning of optimal control in s-SPA systems, where both statistics learning and temporary opportunity utilization are jointly considered. An effective and efficient online policy, so called IE-OSP, is proposed, which theoretically guarantees system converges to the optimal s -SPA strategy with bounded probability. Experimental results further show that, the regret of IE-OSP is almost in optimal logarithmic increasing rate over time, and is sub-linear with the increasing number of channels. Compared with existing solutions, our proposed algorithm achieves 25 ~ 30% throughput gain in typical scenarios.

Current research on Internet of Things (IoT) mainly focuses on how to enable general objects to see, hear, and smell the physical world for themselves, and make them connected to share the observations. In this paper, we argue that only connected is not enough, beyond that, general objects should have the capability to learn, think, and understand both physical and social worlds by themselves. This practical need impels us to develop a new paradigm, named cognitive Internet of Things (CIoT), to empower the current IoT with a “brain” for high-level intelligence. Specifically, we first present a comprehensive definition for CIoT, primarily inspired by the effectiveness of human cognition. Then, we propose an operational framework of CIoT, which mainly characterizes the interactions among five fundamental cognitive tasks: perception-action cycle, massive data analytics, semantic derivation and knowledge discovery, intelligent decision-making, and on-demand service provisioning. Furthermore, we provide a systematic tutorial on key enabling techniques involved in the cognitive tasks. In addition, we also discuss the design of proper performance metrics on evaluating the enabling techniques. Last but not the least, we present the research challenges and open issues ahead. Building on the present work and potentially fruitful future studies, CIoT has the capability to bridge the physical world (with objects, resources, etc.) and the social world (with human demand, social behavior, etc.), and enhance smart resource allocation, automatic network operation, and intelligent service provisioning.