Beam tracking methods are instrumental for efficient use of the multi-gigahertz bandwidth available at mmWave frequencies. In this paper, we propose a Multi-armed Bandit (MAB) based Reinforcement Learning (RL) algorithm to periodically select transmitter-receiver beam pairs so as to maximize the average spectral efficiency. Contrary to a traditional Bayesian MAB-based approach, the MAB algorithm proposed by us can track a user as it moves across multiple correlation distances. The algorithm keeps track of the received signal strength to detect a change in the channel correlation and adjusts its strategy to adapt to the new channel conditions. We derive an upper bound on the regret of the proposed algorithm. The proposed algorithm is evaluated on channel data generated using the open-source simulator NYUSIM and is observed to outperform existing algorithms, thus removing the requirement of repeated initial access procedures.

In today's wireless networks, a variety of Radio Access Technologies (RATs) are present. However, each RAT being controlled individually leads to suboptimal utilization of network resources. Due to the remarkable growth of data traffic, interworking among different RATs is becoming necessary to overcome the problem of suboptimal resource utilization. Users can be offloaded from one RAT to another based on loads of different networks, channel conditions and priority of users. We consider the optimal RAT selection problem in a Fifth Generation (5G) New Radio (NR)-Wireless Fidelity (WiFi) network where we aim to maximize the total system throughput subject to constraints on the blocking probability of high priority users and the offloading probability of low priority users. The problem is formulated as a Constrained Markov Decision Process (CMDP). We reduce the effective dimensionality of the action space by eliminating the provably suboptimal actions. We propose low-complexity online heuristics for RAT selection which can operate without the knowledge regarding the statistics of system dynamics. Network Simulator-3 (ns-3) simulations reveal that the proposed algorithms outperform traditional RAT selection algorithms under realistic network scenarios including user mobility.

In an offload-capable Long Term Evolution (LTE)Wireless Fidelity (WiFi) Heterogeneous Network (HetNet), we consider the problem of maximization of the total system throughput under voice user blocking probability constraint. The optimal policy is threshold in nature. However, computation of optimal policy requires the knowledge of the statistics of system dynamics, viz., arrival processes of voice and data users, which may be difficult to obtain in reality. Motivated by the Post-Decision State (PDS) framework to learn the optimal policy under unknown statistics of system dynamics, we propose, in this paper, an online Radio Access Technology (RAT) selection algorithm using Relative Value Iteration Algorithm (RVIA). However, the convergence speed of this algorithm can be further improved if the underlying threshold structure of the optimal policy can be exploited. To this end, we propose a novel structureaware online RAT selection algorithm which reduces the feasible policy space, thereby offering lesser storage and computational complexity and faster convergence. This algorithm provides a novel framework for designing online learning algorithms for other problems and hence is of independent interest. We prove that both the algorithms converge to the optimal policy. Simulation results demonstrate that the proposed algorithms converge faster than a traditional scheme. Also, the proposed schemes perform better than other benchmark algorithms under realistic network scenarios.

Today's wireless networks consist of a multitude of Radio Access Technologies (RATs), each being controlled individually, leading to suboptimal utilization of network resources. However, the unprecedented growth of data traffic is creating the need for an efficient inter-working of various RATs to circumvent the problem of suboptimal utilization of resources. Application of Software Defined Networking (SDN) principles enables the control and management of various RATs in a unified way. In this paper, we specifically focus on the inter-working between Long Term Evolution (LTE) and Wireless Fidelity (WiFi). We propose an SDN based architecture for a network comprising LTE Base Stations (BSs) and WiFi Access Points (APs). Users can be offloaded from one RAT to another based on different criteria, viz., user priority and channel state of users. We consider the problem of optimal RAT selection to maximize the total system throughput subject to constraints on the blocking probability of high priority users and the offloading probability of high priority users and formulate it as a Constrained Markov Decision Process (CMDP). We propose a low-complexity RAT selection algorithm which does not require the knowledge of the statistics of system dynamics. To conduct experiments, we develop a Network Simulator-3 (ns-3) based evaluation platform in accordance with the SDN principles. Experimental results demonstrate that the proposed algorithm provides a near-optimal performance.

Dual Connectivity (DC) is a technique proposed to address the problem of increased handovers in heterogeneous networks. In DC, a foreground User Equipment (UE) with multiple transceivers has a possibility to connect to a Macro eNodeB (MeNB) and a Small cell eNodeB (SeNB) simultaneously. In downlink split bearer architecture of DC, a data radio bearer at MeNB gets divided into two; one part is forwarded to the SeNB through a non-ideal backhaul link to the UE, and the other part is forwarded by the MeNB. This may lead to an increase in the total delay at the UE since different packets corresponding to a single transmission may incur varying amounts of delays in the two different paths. Since the resources in the MeNB are shared by background legacy users and foreground users, DC may increase the blocking probability of background users. Moreover, single connectivity to the small cell may increase the blocking probability of foreground users. Therefore, we target to minimize the average delay of the system subject to a constraint on the blocking probability of background and foreground users. The optimal policy is computed and observed to contain a threshold structure. The variation of average system delay is studied for changes in different system parameters.

In a Heterogeneous Network (HetNet) comprising of multiple Radio Access Technologies (RATs), a user can be associated with a particular RAT and can be steered to other RATs in a seamless manner. To handle the rapid growth of data traffic, offloading of mobile data to Wireless Fidelity (WiFi) has been proposed in a Long Term Evolution (LTE) based HetNet. In this paper, we consider an optimal RAT selection problem in an offload-capable LTE-WiFi system with an objective of maximizing the total system throughput subject to a constraint on the voice user blocking probability. An on-line algorithm for optimal RAT selection is proposed based on a Relative Value Iteration Algorithm (RVIA) centric Q-learning approach. The proposed algorithm can be implemented without any explicit knowledge of arrival processes of voice and data users. Simulation results are presented to exhibit the convergence behavior of the proposed scheme to the optimal policy.

As a part of Fourth Generation (4G) wireless communication system, a user can associate with any radio access technology (RAT) when there are multiple RATs available and can move seamlessly among them. To handle the explosive growth of traffic in cellular network, the idea of mobile data traffic offloading to Wireless Fidelity (WiFi) has been proposed. In this paper, we focus on mobile data offload assisted optimal association problem in a heterogeneous network. The problem has been formulated under the framework of constrained Markov decision process (CMDP). Value iteration and gradient descent algorithm has been used to determine the optimal policy. We aim to maximize the expected average per-user throughput of the system subject to a constraint on blocking probability of voice users. In our model, we consider the possibility of mobile data user offload from one RAT to another during association or departure of a user. Optimal policy has been found to follow a threshold structure. Numerical results are presented to show how the average per-user throughput of the system varies under different load conditions.