Using environmental sensory data can enhance communications beam training and reduce its overhead compared to conventional methods. However, the availability of fresh sensory data during inference may be limited due to sensing constraints or sensor failures, necessitating a realistic model for multimodal sensing. This paper proposes a joint multimodal sensing and beam prediction framework that operates under a constraint on the average sensing rate, i.e., how often fresh sensory data should be obtained. The proposed method combines deep reinforcement learning, i.e., a deep Q-network (DQN), with a neural network (NN)-based beam predictor. The DQN determines the sensing decisions, while the NN predicts the best beam from the codebook. To capture the effect of limited fresh data during inference, the age of information (AoI) is incorporated into the training of both the DQN and the beam predictor. Lyapunov optimization is employed to design a reward function that enforces the average sensing constraint. Simulation results on a real-world dataset show that AoI-aware training improves top-1 and top-3 inference accuracy by 44.16% and 52.96%, respectively, under a strict sensing constraint. The performance gain, however, diminishes as the sensing constraint is relaxed.

We consider a pull-based status update system consisting of a finite-state Markov source, an energy-harvesting-enabled transceiver, and a sink. Both data and command channels are error-prone. The sink is interested in real-time tracking of the source and sends commands for updates to the transceiver. However, due to noise in the command channel, the transceiver may experiencefalse alarmsormiss-detections. We study the problem of minimizing the long-term time average of a (generic) distortion subject to a constraint on the average number of commands. Due to the pull-based nature of the system, the controller (i.e., sink) has only partial observability of the source state and the available energy at the transceiver. Therefore, we model the problem as a constrained partially observable Markov decision process (C-POMDP), which is then cast as a constrained belief-MDP (C-belief-MDP) problem. The infinite belief-state space makes solving the C-belief-MDP difficult. Thus, we effectively truncate the belief-state space and formulate a finite-state C-belief-MDP problem. We develop a stationary randomized policy (SRP) to solve the problem by employing Lagrangian relaxation, the relative value iteration algorithm (RVIA), and bisection search. Due to the curse of dimensionality in deriving the SRP, we also develop a low-complexity policy applying the drift-plus-penalty method, which transforms the C-belief-MDP problem into a sequence of per-slot optimization problems. Simulation results show the effectiveness of the proposed policies and their superiority compared to baseline policies.

We address the problem of real-time remote tracking of a partially observable Markov source in an energy harvesting system with an unreliable communication channel. We consider both sampling and transmission costs. Different from most prior studies that assume the source is fully observable, the sampling cost renders the source partially observable. The goal is to jointly optimize sampling and transmission policies for two semantic-aware metrics: i) a general distortion measure and ii) the age of incorrect information (AoII). We formulate a stochastic control problem. To solve the problem for each metric, we cast a partially observable Markov decision process (POMDP), which is transformed into a belief MDP. Then, for both AoII under the perfect channel setup and distortion, we express the belief as a function of the age of information (AoI). This expression enables us to effectively truncate the corresponding belief space and formulate a finite-state MDP problem, which is solved using the relative value iteration algorithm. For the AoII metric in the general setup, a deep reinforcement learning policy is proposed. Simulation results show the effectiveness of the derived policies and, in particular, reveal a non-monotonic switching-type structure of the real-time optimal policy with respect to AoI.

We study the problem of remote tracking in an energy-harvesting enabled status update system consisting of an information source, a sampler, a transmitter, and a monitor. The information source is modeled as a finite-state Markov chain. The sampler samples the source, and the transmitter transmits the taken samples to the monitor. We consider both sampling and transmission costs, and thus, the source is not fully observable. The primary objective is to determine the optimal joint sampling and transmission policies based on a goal-oriented metric, defined by a generic distortion function. We first formulate a stochastic optimization problem and cast it into a partially observable Markov decision process (POMDP) problem. Subsequently, we employ the notion of belief state and characterize the belief space through the age of information (AoI) to convert the problem into a finite-state MDP problem, which is then solved via the relative value iteration algorithm. We also explore different estimation strategies at the monitor and examine their impact on the system performance. The simulation results show the effectiveness of the derived policy and reveal that, depending on the source dynamic, the choice of estimation strategy itself can significantly influence the overall performance.

We consider a multi-source relaying system where independent sources randomly generate status update packets which are sent to the destination with the aid of a relay through unreliable links. We develop transmission scheduling policies to minimize the weighted sum average age of information (AoI) subject to transmission capacity and long-run average resource constraints. We formulate a stochastic control optimization problem and solve it using a constrained Markov decision process (CMDP) approach and a drift-plus-penalty method. The CMDP problem is solved by transforming it into an MDP problem using the Lagrangian relaxation method. We theoretically analyze the structure of optimal policies for the MDP problem and subsequently propose a structure-aware algorithm that returns a practical near-optimal policy. Using the drift-plus-penalty method, we devise a near-optimal low-complexity policy that performs the scheduling decisions dynamically. We also develop a model-free deep reinforcement learning policy for which the Lyapunov optimization theory and a dueling double deep Q-network are employed. The complexities of the proposed policies are analyzed. Simulation results are provided to assess the performance of our policies and validate the theoretical results. The results show up to 91% performance improvement compared to a baseline policy.

In this paper, we propose a novel joint resource allocation and cooperative caching scheme for power-domain non-orthogonal multiple access (PD-NOMA)-based heterogeneous networks (HetNets). In our scheme, the requested content is fetched directly from the edge if it is cached in the storage of one of the base stations (BSs), and otherwise is fetched via the backhaul. Our scheme consists of two phases: 1) Caching phase where the contents are saved in the storage of the BSs; and 2) Delivery phase where the requested contents are delivered to users. We formulate a novel optimization problem over radio resources and content placement variables. We aim to minimize the network cost subject to quality-of-service (QoS), caching, subcarrier assignment, and power allocation constraints. By exploiting advanced optimization methods, such as alternative search method (ASM), Hungarian algorithm, successive convex approximation (SCA), we obtain an efficient sub-optimal solution of the optimization problem. Numerical results illustrate that our ergodic caching policy via the proposed resource management algorithm can achieve a considerable reduction on the total cost on average compared to the most popular caching and random caching policy. Moreover, our cooperative NOMA scheme outperforms orthogonal multiple access (OMA) in terms of the delivery cost in general with an acceptable complexity increase.

In this paper, we study a Deep Reinforcement Learning (DRL) based framework for an online end-user service provisioning in a Network Function Virtualization (NFV)-enabled network. We formulate an optimization problem aiming to minimize the cost of network resource utilization. The main challenge is provisioning the online service requests by fulfilling their Quality of Service (QoS) under limited resource availability. Moreover, fulfilling the stochastic service requests in a large network is another challenge that is evaluated in this paper. To solve the formulated optimization problem in an efficient and intelligent manner, we propose a Deep Q-Network for Adaptive Resource allocation (DQN-AR) in NFV-enabled network for function placement and dynamic routing which considers the available network resources as DQN states. Moreover, the service’s characteristics, including the service life time and number of the arrival requests, are modeled by the Uniform and Exponential distribution, respectively. In addition, we evaluate the computational complexity of the proposed method. Numerical results carried out for different ranges of parameters reveal the effectiveness of our framework. In specific, the obtained results show that the average number of admitted requests of the network increases by 7 up to 14% and the network utilization cost decreases by 5 and 20%.

In this paper, we propose a Zero-Touch, deep reinforcement learning (DRL)-based Proactive Failure Recovery framework called ZT-PFR for stateful network function virtualization (NFV)-enabled networks. To this end, we formulate a resource-efficient optimization problem minimizing the network cost function including resource cost and wrong decision penalty. As a solution, we propose state-of-the-art DRL-based methods such as soft-actor-critic (SAC) and proximal-policy-optimization (PPO). In addition, to train and test our DRL agents, we propose a novel impending-failure model. Moreover, to keep network status information at an acceptable freshness level for appropriate decision-making, we apply the concept of age of information to strike a balance between the event and scheduling based monitoring. Several key systems and DRL algorithm design insights for ZT-PFR are drawn from our analysis and simulation results. For example, we use a hybrid neural network, consisting long short-term memory layers in the DRL agents structure, to capture impending-failures time dependency.

We consider a multi-source relaying system where the sources independently and randomly generate status update packets which are sent to the destination with the aid of a buffer-aided relay through unreliable links. We formulate a stochastic optimization problem aiming to minimize the sum average age of information (AAoI) of sources under per-slot transmission capacity constraints and a long-run average resource constraint. To solve the problem, we recast it as a constrained Markov decision process (CMDP) problem and adopt the Lagrangian method. We analyze the structure of an optimal policy for the resulting MDP problem that possesses a switching-type structure. We propose an algorithm that obtains a stationary deterministic near-optimal policy, establishing a benchmark for the system. Simulation results show the effectiveness of our algorithm compared to benchmark algorithms.

Network slicing is a crucial part of the 5G networks that communication service providers (CSPs) seek to deploy. By exploiting three main enabling technologies, namely, software-defined networking (SDN), network function virtualization (NFV), and network slicing, communication services can be served to the end-users in an efficient, scalable, and flexible manner. To adopt these technologies, what is highly important is how to allocate the resources and admit the customers of the CSPs based on the predefined criteria and available resources. In this regard, we propose a novel joint resource and admission management algorithm for slice-enabled networks. In the proposed algorithm, our target is to minimize the network cost of the CSP subject to the slice requests received from the tenants corresponding to the virtual machines and virtual links constraints. Our performance evaluation of the proposed method shows its efficiency in managing CSP’s resources.