In narrowband Internet of Things, the cell can be divided into up to three coverage enhancement (CE) levels, each associated with a narrowband physical random access channel (NPRACH) that has a CE level-specific configuration. Providing resources to NPRACHs increases the success rate of the random access procedure but detracts resources from the uplink carrier for other transmissions. To effectively address this tradeoff we propose to adjust the NPRACH parameters along with the power thresholds that determine the CE levels, which allows to control at the same time the traffic distribution between CE levels and the resources allocated to each CE level. Since the traffic is dynamic and random, reinforcement learning (RL) is a suitable approach for finding an optimal control policy, but its inherent sample inefficiency is a drawback for online learning in an operational network. To overcome this issue, we propose a new model-based RL algorithm that achieves high efficiency even in the early stages of learning.

Network slicing is a key feature of 5G and beyond networks, allowing the deployment of separate logical networks (network slices), sharing a common underlying physical infrastructure, and characterized by distinct descriptors and behaviors. The dynamic allocation of physical network resources among coexisting slices should address a challenging trade-off: to use resources efficiently while assigning each slice sufficient resources to meet its service level agreement (SLA). We consider the allocation of time-frequency resources from a new perspective: to design a control algorithm capable of learning over the operating network, while keeping the SLA violation rate under an acceptable level during the learning process. For this purpose, traditional model-free reinforcement learning (RL) methods present several drawbacks: low sample efficiency, extensive exploration of the policy space, and inability to discriminate between conflicting objectives, causing inefficient use of the resources and/or frequent SLA violations during the learning process. To overcome these limitations, we propose a model-based RL approach built upon a novel modeling strategy that comprises a kernel-based classifier and a self-assessment mechanism. In numerical experiments, our proposal, referred to as kernel-based RL, clearly outperforms state-of-the-art RL algorithms in terms of SLA fulfillment, resource efficiency, and computational overhead.

The virtualization of radio access networks (vRAN) is the last milestone in the NFV revolution. However, the complex dependencies between computing and radio resources make vRAN resource control particularly daunting. We present vrAIn, a dynamic resource orchestrator for vRANs based on deep reinforcement learning. First, we use an autoencoder to project high-dimensional context data (traffic and channel quality patterns) into a latent representation. Then, we use a deep deterministic policy gradient (DDPG) algorithm based on an actor-critic neural network structure and a classifier to map contexts into resource control decisions. We have evaluated vrAIn experimentally, using an open-source LTE stack over different platforms, and via simulations over a production RAN. Our results show that: (i) vrAIn provides savings in computing capacity of up to 30% over CPU-agnostic methods; (ii) it improves the probability of meeting QoS targets by 25% over static policies; (iii) upon computing capacity under-provisioning, vrAIn improves throughput by 25% over state-of-the-art schemes; and (iv) it performs close to an optimal offline oracle. To our knowledge, this is the first work that thoroughly studies the computational behavior of vRANs and the first approach to a model-free solution that does not need to assume any particular platform or context.

The virtualization of radio access networks (vRAN) is the last milestone in the NFV revolution. However, the complex dependencies between computing and radio resources make vRAN resource control particularly daunting. We present vrAIn, a dynamic resource controller for vRANs based on deep reinforcement learning. First, we use an autoencoder to project high-dimensional context data (traffic and signal quality patterns) into a latent representation. Then, we use a deep deterministic policy gradient (DDPG) algorithm based on an actor-critic neural network structure and a classifier to map (encoded) contexts into resource control decisions. We have implemented vrAIn using an open-source LTE stack over different platforms. Our results show that vrAIn successfully derives appropriate compute and radio control actions irrespective of the platform and context: (i) it provides savings in computational capacity of up to 30% over CPU-unaware methods; (ii) it improves the probability of meeting QoS targets by 25% over static allocation policies using similar CPU resources in average; (iii) upon CPU capacity shortage, it improves throughput performance by 25% over state-of-the-art schemes; and (iv) it performs close to optimal policies resulting from an offline oracle. To the best of our knowledge, this is the first work that thoroughly studies the computational behavior of vRANs, and the first approach to a model-free solution that does not need to assume any particular vRAN platform or system conditions.

While the application of the NFV paradigm into the network is proceeding full steam ahead, there is still one last mile- stone to be achieved in this context: the virtualization of the radio access network (vRAN). Due to the very complex de- pendency between the radio conditions and the computing resources needed to provide the baseband processing func- tionality, attaining an efficient resource control is particularly challenging. In this demonstration, we will showcase vrAIn, a vRAN dynamic resource controller that employs deep re- inforcement learning to perform resource assignment deci- sions. vrAIn, which is implemented using an open-source LTE stack over a Linux platform, can achieve substantial sav- ings in the used CPU resources while maintaining the target QoS for the attached terminals and maximize throughput when there is a deficit of computational capacity.

In heterogeneous cellular networks (HetNets), switching OFF small cells under low user traffic periods has been proved to be an effective energy saving strategy. However, this strategy has strong interactions with interference coordination (IC) mechanisms, making it convenient to address both tasks simultaneously. The motivation of this paper is to develop a self-optimization algorithm capable of jointly controlling energy saving and IC mechanisms using an online learning approach. Our proposal is based on a contextual bandit formulation that, among other challenges, implies discovering the most energy-efficient control actions while satisfying a predefined level of Quality of Service (QoS) for the users. We propose a two-level framework comprising a global controller, in charge of a group of macro cells, and multiple local controllers, one per macro cell. The global controller implements a novel algorithm, referred to as the Bayesian Response Estimation and Threshold Search (BRETS), that is capable of learning, for each control action, its feasibility boundaries in terms of QoS and its energy consumption as a function of the aggregated user traffic. The algorithm comes with a bound on its expected convergence time. The local controllers translate the control actions learned by the global controller into local decisions. Our numerical results show that BRETS is only 1% less efficient than an ideal oracle policy, clearly outperforming other benchmark algorithms.

This paper addresses the joint problem of energy saving and interference coordination in heterogeneous networks (HetNets) using a contextual bandit formulation. We propose a semi-distributed scheme consisting of a learning agent and local controllers. The learning agent comprises a neural network (NN) classifier and a Multi-Armed Bandit (MAB) algorithm. The NN classifier is dynamically trained to choose a subset of configurations (i.e., feasible configurations in terms of QoS) based on the context information (network state). Then, the MAB algorithm picks one control (i.e., global configuration parameters) among those selected by the NN classifier, with the aim of improving the energy efficiency. These global configurations are interpreted by the local controllers on each network sector. This scheme allows the learning agent to progressively learn the best policy by observing the network state and the performance of the chosen configurations in terms of energy consumption and QoS. Our numerical results show an energy saving close to 20% with respect to a default policy and an improvement of 13% with respect to addressing energy saving and interference coordination separately.

This paper focuses on interference coordination between the small cell and macro cell tiers of a wireless access network. We propose a new perspective based on a modelfree learning strategy, not requiring any previous knowledge about the network (e.g., topology, interference graph, scheduling algorithms). Our approach is based on a stochastic optimization algorithm known as Response Surface Methodology, that we use to learn the optimal parameter configuration during network operation (online learning) adapting to changes on network conditions (e.g., traffic, user positions). The result is a simple, effective and flexible mechanism that outperforms previous proposals. As a case study we apply our scheme to the dynamic adjustment of LTE-A eICIC parameters (CRE bias and ABS ratio).

This paper focuses on interference coordination between the small cell and macro cell tiers of a wireless access network. We present a self-optimization mechanism for LTE-A eICIC parameters (CRE bias and ABS ratio) following a novel approach based on a model-free learning strategy, not requiring any previous knowledge about the network (e.g., topology, interference graph, and scheduling algorithms). Our proposal is built upon a stochastic optimization algorithm known as response surface methodology (RSM), that we use to find efficient eICIC configurations during network operation (online learning), adapting to changing network conditions, such as traffic or user distribution. The objective consists of optimizing a performance metric for which, in general, mathematical expression is unavailable. In particular, we consider the fifth percentile throughput defined by the 3GPP. By means of RSM, our mechanism obtains local approximations of the objective function to perform steepest ascent iterations with an adjustable level of statistical accuracy. The algorithm can be extended to account for stochastic constraints, allowing the network to optimize one performance metric while maintaining other metrics above a desired level.

Cooperative Spectrum Sharing (CSS) is an appealing approach for primary users (PUs) to share spectrum with secondary users (SUs) because it increases the transmission range or rate of the PUs. Most previous works are focused on developing complex algorithms which may not be fast enough for real-time variations such as channel availability and/or assume perfect information about the network. Instead, we develop a learning mechanism for a PU to enable CSS in a strongly incomplete information scenario with low computational overhead. Our mechanism is based on a Markovian variant of multi-armed bandits (MABs) called superprocess, enhanced with the concept of Upper Confidence Bound (UCB) from stochastic MABs. By means of Monte-Carlo evaluations we show that, despite its low computational overhead, it converges to a low regret solution outperforming baseline approaches such as epsilon-greedy. This algorithm can be extended to include more sophisticated features while maintaining its desirable properties such as low computational overhead and fast speed of convergence.

As cellular network technology evolves, the operators deploy new generation networks while maintaining their legacy networks, since not all users upgrade their terminals at the same pace. Therefore, the spectrum associated to these legacy networks becomes gradually underused. By means of cognitive radio techniques, the operator can allow its new generation terminals to reuse this spectrum. We propose a semi-decentralized scheme in which the operator guides the secondary access by broadcasting some operational parameters of the access strategy. The mechanism dynamically learns the optimal parameters by means of a response surface methodology (RSM), implying a very small signaling overhead. Our results show a notable capacity improvement compared to the classical approaches that exploit either spatial or temporal opportunities.

In Opportunistic Spectrum Access (OSA), conventional secondary users' (SU) radios can not transmit and detect primary user (PU) activity simultaneously, and therefore SU and PU transmissions may overlap, causing harmful interference at PU receivers. The extended approach of periodically interrupting SU transmission to perform spectrum sensing mitigates but does not completely solve the problem. Our proposal consists of a collision detection mechanism simultaneous to reception at the SU receiver (Background Detection), which aborts the ongoing secondary transmission if a collision is detected. While some previous works assumed the existence of a collision detection scheme, little work has been done in proposing a specific one for OSA systems, capable of operating during SU reception. Our proposal exploits local characterization of the PU activity, power levels and packet errors at each time slot. Based on these data, we develop a Maximum A Posteriori (MAP) estimator as the basis for the collision detection system. We describe the general methodology to build this estimator, and study the system's robustness against parameter misestimations. The proposed scheme can be incorporated to existing OSA protocols. Our results show a notable throughput improvement for the SUs while assuring a lower collision probability with PU transmissions in most cases, even under severe estimation inaccuracies.

Cooperative Spectrum Sharing (CSS) is an appealing approach for primary users (PUs) to share spectrum with secondary users (SUs) because it increases the transmission range or rate of the PUs. Most previous works are focused on developing complex algorithms which may not be fast enough for real-time variations such as in channel availability. Instead, we develop a learning mechanism for a PU to enable CSS in a strongly incomplete information scenario with low computational overhead. We model the learning mechanism of the PU to discover which SU to interact with and what offer to make to it with a combination of a Multi-Armed Bandit (MAB) and a Markov Decision Process (MDP). By means of Monte-Carlo simulations we show that, despite its low computational overhead, our proposed mechanism converges to the optimal solution and significantly outperforms the ε-greedy heuristic. This algorithm can be extended to include more sophisticated features while maintaining its desirable properties such as the fast speed of convergence.

This paper introduces a smart assistant for professional volleyball training based on machine-learning techniques (SAETA). SAETA addresses two main aspects of elite sports coaching: 1) technical-tactical effort control, which aims at controlling exercise effort and fatigue levels and 2) exercise quality training, which complements the former by analyzing the execution of player movements. SAETA relies on a sensing infrastructure that monitors both players and their environment, and produces real-time data that is analyzed by different modules of a decision engine. Technical-tactical effort control is based on a dynamic programming model, which selects the best activity and rest durations in interval training, with the goal of maximizing effort while preventing fatigue. Exercise quality control consists of two stages. In the first stage, movements are detected by means of a k-nearest neighbors classifier and in the second stage, movement intensity is classified according to recent statistical data from the player being analyzed. These analyses are reported to coaches and players in real-time. SAETA has been developed in close collaboration with the Universidad Católica San Antonio de Murcia volleyball team, which competes in the Spanish women's premier league. Data gathered during training sessions has provided a knowledge base for the algorithms developed, and has been used for the validation of results.

We consider a spectrum leasing system in which secondary networks offer offload services to a primary network (PN) in exchange of temporary access to the PN's spectrum. When the SANs collude and coordinate their prices, forming a cartel, the PN experiences cartel overcharge, which in our scenario implies lower transmission rates for the serviced PUs. To protect the spectrum owner's interests and possibly enforce market regulation, we propose an intervention framework in which an intervention device or manager (possibly with the authorization and/or supervision of an external regulatory agency) counteracts cartel formation. This framework exploits the specific features that make wireless systems different from conventional markets, enabling the manager to modify the set of achievable outcomes. The intervention capability is limited, so the objective is to design an intervention rule which is maximizes the PN transmission rate within the given constraints.

We consider a scenario in which a licensed wireless operator or primary network (PN) coexists with an ad-hoc, cognitive, secondary network. To minimize harmful interference on the primary users (PUs), secondary users (SUs) sense the PU activity on the licensed spectrum band before transmitting in spectrum opportunities. We study a bandwidth reservation (BR) scheme by which the PN keeps a set of adjacent channels free of PU transmissions. These reserved channels only accommodate PU traffic when all the non-reserved channels are used, and the SUs only occupy available channels within the reserved spectrum. Intuitively, this strategy reduces collision probability and simplifies the design of opportunistic spectrum access (OSA) mechanisms. However, from the point of view of the PN, BR entails a tradeoff between the benefits of an improved coexistence with SUs, and the capacity reduction associated to having fewer options for PU channel allocation. The main objective of this paper is to determine when BR improves the overall PN performance under SU activity. The SUs are characterized by a hardware limited radio, imperfect spectrum sensing, bayesian estimation of PU activity and multichannel access. Because PU capacity is the central issue, we assume a PN capable of exploiting all the available bandwidth at every moment. By means of a Markov-reward model, we compute the expected PU capacity with and without BR, considering propagation effects, interference and random locations. The results show that, in a non-congested PN with SU activity, the interference reduction capability of BR increases the overall capacity of the PN compared to not using BR.

We consider a spectrum leasing system in which secondary networks offer offload services to a primary network (PN) in exchange for temporary access to the PN's spectrum. When the coverage areas of several secondary access nodes (SANs) overlap, they compete for primary users (PUs), which benefits the PN, except when the SANs collude and coordinate their prices, forming a cartel. As a result, the PN obtains lower transmission rates for the serviced PUs. Our coalitional game analysis shows that stable cartels always exist and can form easily. To protect the spectrum owner's interests and enforce market regulation, we propose an intervention framework in which an intervention manager counteracts cartel formation. The specific features that make wireless systems different from conventional markets enable the manager to modify the set of achievable outcomes. The intervention capability is limited; thus, the objective is to design an intervention rule maximizing the PN transmission rate within the given constraints. Importantly, the intervention can solely act as a threat or a warning that does not need to be executed in practice. To reduce the computational effort, we also propose a low-complexity intervention rule that performs similarly to the optimal one in terms of assurable PN rate increment and outperforms other effective approaches.

This paper presents a novel system for effort control in team-sport training, where each component is associated with a particularized program. The system relies on a decision engine that analyzes training data and provides instructions to athletes, and on a sensing infrastructure which monitors athletes' biometrics and movements and gathers environmental data such as humidity or temperature. The prototype has been developed using IMOTE2 and MicaZ sensors under TinyOS and BioHarness biometric belts. The decision technique estimates the ratios of each effort regime, based on best-fit polynomial approximations using historical data from previous training sessions. The complete system results in a virtual coach, operating in real time and automatically. This methodology has been piloted in an experiment with the UCAM Volleyball Murcia team, top of the Spanish national women's volleyball league. Data obtained during training sessions have provided a knowledge base for the algorithms developed and allowed for validation of the results.

In dense radio frequency identification (RFID) systems, reader-to-reader interference may appear among readers operating nearby. When the anticollision algorithm is Frame Slotted Aloha (FSA), a usual solution is to multiplex the frames of interfering readers by means of a central controller. The scenario studied considers RFID tags attached to objects carried by conveyor belts or vehicles so that each tag crosses, in constant motion, one reader's interrogation region. The problem addressed consists of determining the length of the multiplexed FSA frames such that each object is successfully identified, with high probability, before exiting its interrogation region. In this paper, we develop a mathematical model, based on dynamic system theory, allowing us to obtain the configurations attaining the desired reliability objective, to show that some configurations present multiple equilibrium points and should be avoided, and to define the operating boundaries of the system. The model can also incorporate fading and capture effects.

This paper proposes a scheduling algorithm for non-real-time traffic in Vehicle to Infrastructure (V2I) networks. The scheduler operates at the link layer and uses the contention-free, poll-based access mode of the 802.11e standard. Our proposal exploits two specific features of V2I that are not present in other wireless systems. First, the fact that vehicles are constrained to a predefined area (the road segment under radio coverage) which can be characterized in terms of packet error ratio (PER). Second, that it is possible to know the vehicle speed, either accurately or approximately. These characteristics allow us to model the data transmission process in V2I as a dynamical system. However, the unpredictable arrivals of new vehicles make it difficult to directly apply dynamic programming to obtain the optimum scheduler. We overcome this limitation by developing a tractable formulation of the problem, similar to a classical linear quadratic regulator design problem but with the significant difference that constraints are included. We propose a novel algorithmic solution that can be efficiently computed and can easily incorporate quality of service differentiation. The scheduler is compared to three other feasible schemes, assessing the impact of the system parameters in the performance. Numerical results show that the proposed scheme is more beneficial when the average PER values have notable variations along the covered area.

Automatic identification based on radio frequency identification (RFID) is progressively being introduced into industrial environments, enabling new applications and processes. In the context of communications, RFID rely mostly on Frame Slotted Aloha (FSA) anticollision protocols. Their goal is to reduce the time required to detect all the tags within range (identification time). Using FSA, the maximum identification rate is achieved when the number of contending tags equals the number of contention slots available in the frame. Therefore, the reader must estimate the number of contenders and allocate that number of slots for the next frame. This paper introduces the new MFML-DFSA anticollision protocol. It estimates the number of contenders by means of a maximum-likelihood estimator, which uses the statistical information from several frames (multiframe estimation) to improve the accuracy of the estimate. Based on this expected number of tags, the algorithm determines the best frame length for the next reading frame, taking into account the constraints of the EPCglobal Class-1 Gen-2 standard. The MFML-DFSA algorithm is compared with previous proposals and found to outperform these in terms of (lower) average identification time and computational cost, which makes it suitable for implementation in commercial RFID readers.

In cooperative automatic repeat request (C-ARQ) protocols, one or more nodes can act as relays, collaborating in the frame retransmission process between a sender and a destination node. In the framework of a broadband mobile access network, we consider a relay enhanced cell where the sender represents the base station, and an undefined number of relays can cooperate with a single user (destination). In this paper we present a Markov model complemented with a reward model to analyze the throughput performance and the efficiency in the bandwidth utilization at the base station, highlighting a clear tradeoff between them. Both performance metrics are balanced by means of a multi-objective optimization algorithm resulting in a retransmission strategy for the access node that notably improves the bandwidth efficiency while maintaining the throughput very close to its maximum. The benefits of the proposed approach are evaluated in two scenarios. In the first one, the relays are part of the network infrastructure. In the second one, the relays are cooperative users.

TCP performance over 3G links can be enhanced by means of active queue management (AQM) techniques at the downlink buffer of the mobile network link layer (RLC). In this paper we describe slope based discard (SBD), a novel deterministic buffer management technique whose discarding policy is based on the observation of the filling rate of the buffer. By means of extensive simulation experiments we compare the performance of TCP over three different RLC buffer management schemes: drop-tail (no AQM), SBD and random early detection (RED). The simulation scenarios are configured to evaluate how the end-to-end goodput and delay are influenced by the number of multiplexed flows, the TCP flavor, the buffer size and the wireless channel error rate.

This paper introduces Slope Based Discard (SBD), a novel Active Queue Management (AQM) scheme that improves TCP performance over 3G wireless links. Its main objective is to prevent buffer overflow at the downlink buffer in the Radio Network Controller, where the flows of a single user are multiplexed. In contrast to Random Early Detection (RED), SBD is a deterministic algorithm, and its discarding policy is based on the observation of the filling rate of the buffer. By means of extensive simulation experiments, we disclose the influence of each SBD parameter on TCP goodput and latency. Compared to RED, which parameter configuration is also addresed, SBD shows similar goodput figures but lower delays. We describe other characteristics that make SBD a better choice than RED for 3G links, like its greater capability to prevent buffer overflow, and its easier implementation.