Effective recognition of jamming in a communication system is essential to maintain the integrity of the electromagnetic spectrum space. In this paper, a novel feature-enhanced open-set jamming pattern recognition method (FOSR) is proposed. First, an in-phase and quadrature (I/Q) data feature enhancement module is designed based on a complex-valued autoencoder to capture the interaction features between the I and Q channels. Then, a jamming feature extraction module is designed to extract jamming characteristics for known patterns by integrating the raw I/Q data with their interaction features. Subsequently, an adaptive threshold open-set classification module is proposed to recognize both known and unknown patterns. Finally, to address the domain shift problem, we extend FOSR with a domain adaptation (DA) module based on distribution alignment and classifier calibration, referred to as FOSR-DA. Simulation results show that the proposed method achieves superior recognition accuracy and exhibits strong robustness when dealing with the domain shift problem.

The deployment of unmanned aerial vehicles (UAVs) to assist in measurement collection for radio map construction has significant potential. In this work, we investigate the UAV-assisted radio map updating system, where the UAV has to collect informative measurements to improve radio map accuracy and reach the destination within the constraint of limited onboard energy. We apply Ordinary Kriging to construct the radio map and use Kriging variance as a metric to evaluate the accuracy of the map. We then formulate a finite-horizon Markov Decision Process (MDP) that optimizes the UAVs trajectory, aiming to maximize the total reduction in Kriging variance under the system's constraints. The MDP is challenging due to its sparse reward and large, continuous state space. To address this, we propose an AT-DQN algorithm that utilizes reward shaping and combines Agent Transformer (AT) with Dueling DQN for effective trajectory learning. Finally, through numerical experiments, we verify the efficiency of the proposed algorithm in both radio map updating and trajectory optimization.

Spectrum management for the coexistence of multiple unmanned aerial vehicle (UAV) networks is a challenging issue, considering both space and frequency reuse. To address this issue, we propose a joint bandwidth and spectrum usage zone (SUZ) flexible allocation scheme, leveraging interference graph. We formulate a joint spectrum bandwidth allocation and SUZs adjustment problem to maximize the system utility, which is a binary nonlinear programming (BNLP) problem. Then we decompose it into two subproblems: the high-priority UAV networks subproblem and the low-priority UAV networks subproblem. The high-priority subproblem is solved using a graph coloring method based on the interference graph, whereas the low-priority subproblem is addressed through a sequential one-step block coordinate descent (SOBCD) approach by constructing a spectrum assignment hypergraph. Simulation results demonstrate that the total utility with the proposed scheme outperforms benchmark schemes, and there exists an optimal SUZ grid adjustment to maximize the total utility.

In this paper, we propose a deep learning (DL)-based task-driven spectrum prediction framework, named DeepSPred. The DeepSPred comprises a feature encoder and a task predictor, where the encoder extracts spectrum usage pattern features, and the predictor configures different networks according to the task requirements to predict future spectrum. Based on the DeepSPred, we first propose a novel 3D spectrum prediction method combining a flow processing strategy with 3D vision Transformer (ViT, i.e., Swin) and a pyramid to serve possible applications such as spectrum monitoring task, named 3D-SwinSTB. 3D-SwinSTB unique3D Patch Merging ViT-to-3D ViT Patch Expandingand pyramid designs help the model accurately learn the potential correlation of the evolution of the spectrogram over time. Then, we propose a novel spectrum occupancy rate (SOR) method by redesigning a predictor consisting exclusively of 3D convolutional and linear layers to serve possible applications such as dynamic spectrum access (DSA) task, named 3D-SwinLinear. Unlike the 3D-SwinSTB output spectrogram, 3D-SwinLinear projects the spectrogram directly as the SOR. Finally, we employ transfer learning (TL) to ensure the applicability of our two methods to diverse spectrum services. The results show that our 3D-SwinSTB outperforms recent benchmarks by more than 5%, while our 3D-SwinLinear achieves a 90% accuracy, with a performance improvement exceeding 10%.

The radio map, which describes spatial signal strength and network coverage information, is crucial in modern wireless systems for network planning and resource management. Fine-grained radio maps rely on measurements collected by sparsely deployed spectrum sensors in the area of interest. However, due to physical limitations and security considerations, these measurements may exhibit non-uniform distribution and be entirely absent in certain inaccessible areas, making it challenging for accurate radio map reconstruction. Thus, in this work, considering the issues of non-uniform sampling and inaccessible areas, we propose a deep completion partial convolution network for radio map reconstruction. This approach captures the spatial characteristics by separating the missing measurements from sampled ones and does not require prior knowledge of emitters. We evaluate our method using a simulated dataset for campus environments and demonstrate its effectiveness over several baselines for reconstructing radio maps.

In this paper, a dense Internet of Things (IoT) monitoring system is considered in which a large number of IoT devices contend for channel access so as to transmit timely status updates to the corresponding receivers using a carrier sense multiple access (CSMA) scheme. Under two packet management schemes with and without preemption in service, the closed-form expressions of the average age of information (AoI) and the average peak AoI of each device is characterized. It is shown that the scheme with preemption in service always leads to a smaller average AoI and a smaller average peak AoI, compared to the scheme without preemption in service. Then, a distributed noncooperative medium access control game is formulated in which each device optimizes its waiting rate so as to minimize its average AoI or average peak AoI under an average energy cost constraint on channel sensing and packet transmitting. To overcome the challenges of solving this game for an ultra-dense IoT, a mean-field game (MFG) approach is proposed to study the asymptotic performance of each device for the system in the large population regime. The accuracy of the MFG is analyzed, and the existence, uniqueness, and convergence of the mean-field equilibrium (MFE) are investigated. Simulation results show that the proposed MFG is accurate even for a small number of devices; and the proposed CSMA-type scheme under the MFG analysis outperforms three baseline schemes with fixed and dynamic waiting rates. Moreover, it is observed that the average AoI and the average peak AoI under the MFE do not necessarily decrease with the arrival rate.

In this paper, a dense Internet of Things (IoT) monitoring system is studied in which a large number of devices contend for transmitting timely status packets to their corresponding receivers over wireless noisy channels, using a carrier sense multiple access (CSMA) scheme. When each device completes one transmission, due to possible transmission failure, two cases with and without transmission feedback to each device must be considered. Particularly, for the case with no feedback, the device uses policy (I): It will go to an idle state and release the channel regardless of the outcome of the transmission. For the case with perfect feedback, if the transmission succeeds, the device will go to an idle state, otherwise it uses either policy (W), i.e., it will go to a waiting state and re-contend for channel access; or it uses policy (S), i.e., it will stay at a service state and occupy this channel to attempt another transmission. For those three policies, the closed-form expressions of the average age of information (AoI) of each device are characterized under schemes with and without preemption in service. It is shown that, for each policy, the scheme with preemption in service always achieves a smaller average AoI, compared with the scheme without preemption. Then, a mean-field approximation approach with guaranteed accuracy is developed to analyze the asymptotic performance for the considered system with an infinite number of devices and the effects of the system parameters on the average AoI are characterized. Simulation results show that the proposed mean-field approximation is accurate even for a small number of devices. The results also show that olicy (S) achieves the smallest average AoI compared with policies (I) and (W), and the average AoI does not always decrease with the arrival rate for all three policies.

In this paper, a dense Internet of Things (IoT) monitoring system is studied in which a large number of devices contend for transmitting timely status packets to their corresponding receivers over wireless noisy channels, using a carrier sense multiple access (CSMA) scheme. When each device completes one transmission, due to possible transmission failure, two cases with and without transmission feedback must be considered. Particularly, for the case without feedback, the device uses policy (I): It goes to an idle state and releases the channel regardless of the outcome of the transmission. For the case with perfect feedback, if the transmission succeeds, the device goes to an idle state, otherwise it uses either policy (W), i.e., it goes to a waiting state and re-contends for channel access; or it uses policy (S), i.e., it stays at a service state and occupies this channel to attempt another transmission. For those three policies under schemes with and without preemption in service, the closed- form expressions of the average age of information (AoI) are characterized. It is shown that, for each policy, the scheme with preemption in service always achieves a smaller average AoI, compared with the scheme without preemption. Then, a mean-field approximation approach with guaranteed accuracy is developed to analyze the asymptotic performance for the considered system with an infinite number of devices. Simulation results show that the proposed mean-field approximation is accurate even for a small number of devices and policy (S) achieves the smallest average AoI among the three policies.

In this paper, a lifelong learning problem is studied for an Internet of Things (IoT) system. In the considered model, each IoT device aims to balance its information freshness and energy consumption tradeoff by controlling its computational resource allocation at each time slot under dynamic environments. An unmanned aerial vehicle (UAV) is deployed as a flying base station so as to enable the IoT devices to adapt to novel environments. To this end, a new lifelong reinforcement learning algorithm, used by the UAV, is proposed in order to adapt the operation of the devices at each visit by the UAV. By using the experience from previously visited devices and environments, the UAV can help devices adapt faster to future states of their environment. To do so, a knowledge base shared by all devices is maintained at the UAV. Simulation results show that the proposed algorithm can converge 25% to 50% faster than a policy gradient baseline algorithm that optimizes each device’s decision making problem in isolation.

In this paper, a novel centralized resource allocation scheme is proposed to enable a wireless base station to adapt to heterogeneous Internet of Things (IoT) environments and manage scarce communication resources to ensure timely delivery of IoT device data. In the considered system, the timeliness of information is determined using non-linear age of information (AoI) metrics that can naturally quantify the freshness of information. To capture the inherent heterogeneity of the IoT system, non-linear aging functions are proposed and designed specifically for IoT devices having different types of messages. To minimize AoI, the proposed centralized scheme allocates the limited communication resources considering AoI and enables the base station to learn the device types. Furthermore, the proposed centralized resource management scheme with different activation probabilities, outage probabilities, and heterogeneity levels is analyzed in terms of the average instantaneous AoI. Simulation results show that the proposed centralized allocation scheme effectively decreases the average instantaneous AoI in a massive IoT with high outage probability and high heterogeneity.

In this paper, a dense Internet of Things (IoT) monitoring system for computational intensive applications is studied in which a large number of devices with computing capability pre-process the collected raw status information into update packets and contend for transmitting them to the corresponding receivers, using a carrier sense multiple access (CSMA) scheme. Depending on whether the pre-processing operation completes when each device senses a channel, two policies are considered: pre-process-then-Sense policy (PtS) and preprocessing-while-Sensing policy (PwS). Particularly, under policy PtS, each device must complete the pre-processing operation before sensing a channel; while under policy PwS, it performs the pre-processing operation and senses a channel concurrently. Here, for policy PwS, if the pre-processing operation is incomplete while a sensed channel is available to be used, then each device will still occupy the channel by sending dummy bits. For both policies, the closed-form expressions of the average age of information (AoI) are characterized. Then, a mean-field approximation framework with guaranteed accuracy is developed to study the asymptotic performance for the considered system in the large population regime. Simulation results validate the analytical results and show that the proposed mean-field approximation under policy PtS is accurate even for a small number of devices. It is also observed that policy PtS achieves a smaller average AoI than policy PwS, revealing that it is unnecessary for each device to occupy the channel before the pre-processing operation completes.

Resource management in Internet-of-Things (IoT) systems is a major challenge due to the massive scale and heterogeneity of the IoT system. For instance, most IoT applications require timely delivery of collected information, which is a key challenge for the IoT. In this article, novel centralized and distributed resource allocation schemes are proposed to enable IoT devices to share limited communication resources and to transmit IoT messages in a timely manner. In the considered system, the timeliness of information is captured using nonlinear Age-of-Information (AoI) metrics that can naturally quantify the freshness of information. To model the inherent heterogeneity of the IoT system, the nonlinear aging functions are defined in terms of IoT device types and message content. To minimize AoI, the proposed resource management schemes allocate the limited communication resources considering AoI. In particular, the proposed centralized scheme enables the base station to learn the device types and to determine aging functions. Moreover, the proposed distributed scheme enables the devices to share the limited communication resources based on available information on other devices and their AoI. The convergence of the proposed distributed scheme is proved, and the effectiveness in reducing the AoI with partial information is analyzed. Furthermore, the proposed resource management schemes with different number of devices, activation probabilities, and outage probabilities are analyzed in terms of the average instantaneous AoI. Simulation results show that the proposed centralized scheme achieves significantly lower average instantaneous AoI when compared to simple centralized allocation without learning, while the proposed distributed scheme achieves significantly lower average instantaneous AoI when compared to random allocation. The results also show that the proposed centralized scheme outperforms the proposed distributed scheme in almost all cases, but the distributed approach is more viable for a massive IoT.

In this paper, a real-time Internet of Things (IoT) monitoring system is considered in which multiple IoT devices must transmit timely updates on the status information of a common underlying physical process to a common destination. In particular, a real-world IoT scenario is considered in which multiple (partially) observed status information by different IoT devices are required at the destination, so that the real-time status of the physical process can be properly re-constructed. By taking into account such correlated status information at the IoT devices, the problem of IoT device scheduling is studied in order to jointly minimize the average age of information (AoI) at the destination and the average energy cost at the IoT devices. Particularly, two types of IoT devices are considered: Type-I devices whose status updates randomly arrive and type-II devices whose status updates can be generated-at-will with an associated sampling cost. This stochastic problem is formulated as an infinite horizon average cost Markov decision process (MDP). The optimal scheduling policy is shown to be threshold-based with respect to the AoI at the destination, and the threshold is non-increasing with the channel condition of each device. For a special case in which all devices are type-II, the original MDP can be reduced to an MDP with much smaller state and action spaces. The optimal policy is further shown to have a similar threshold-based structure and the threshold is non-decreasing with an energy cost function of the devices. Simulation results illustrate the structure of the optimal policy and show the effectiveness of the optimal policy compared with a myopic baseline policy.

Minimization of the expected value of age of information (AoI) is a risk-neutral approach, and it thus cannot capture rare, yet critical, events with potentially large AoI. In order to capture the effect of these events, in this paper, the notion of conditional value-at-risk (CVaR) is proposed as an effective coherent risk measure that is suitable for minimization of AoI for real-time IoT status updates. In the considered monitoring system, an IoT device monitors a physical process and sends the status updates to a remote receiver with an updating cost. The optimal status update process is designed to jointly minimize the AoI at the receiver, the CVaR of the AoI at the receiver, and the energy cost. This stochastic optimization problem is formulated as an infinite horizon discounted risk-aware Markov decision process (MDP), which is computationally intractable due to the time inconsistency of the CVaR. By exploiting the special properties of coherent risk measures, the risk-aware MDP is reduced to a standard MDP with an augmented state space, for which we derive the optimal stationary policy using dynamic programming. In particular, the optimal history-dependent policy of the risk-aware MDP is shown to depend on the history only through the augmented system states and can be readily constructed using the optimal stationary policy of the augmented MDP. The proposed solution is shown to be computationally tractable and able to minimize the AoI in real-time IoT monitoring systems in a risk-aware manner.

In this paper, a real-time Internet of Things (IoT) monitoring system is considered in which the IoT devices are scheduled to sample associated underlying physical processes and send the status updates to a common destination. In a real-world IoT, due to the possibly different dynamics of each physical process, the sizes of the status updates for different devices are often different and each status update typically requires multiple transmission slots. By taking into account such multi-time slot transmissions with non-uniform sizes of the status updates under noisy channels, the problem of joint device scheduling and status sampling is studied in order to minimize the average age of information (AoI) at the destination. This stochastic problem is formulated as an infinite horizon average cost Markov decision process (MDP). The monotonicity of the value function of the MDP is characterized and then used to show that the optimal scheduling and sampling policy is threshold-based with respect to the AoI at each device. To overcome the curse of dimensionality, a low-complexity suboptimal policy is proposed through a semi-randomized base policy and linear approximated value functions. The proposed suboptimal policy is shown to exhibit a similar structure to the optimal policy, which provides a structural base for its effective performance. A structure-aware algorithm is then developed to obtain the suboptimal policy. The analytical results are further extended to the IoT monitoring system with random status update arrivals, for which, the optimal scheduling and sampling policy is also shown to be threshold-based with the AoI at each device. Simulation results illustrate the structures of the optimal policy and show a near-optimal AoI performance resulting from the proposed suboptimal solution approach.

In this paper, a real-time Internet of Things (IoT) monitoring system is considered in which the IoT devices are scheduled to sample underlying physical processes and send the status updates to a common destination. In a real-world IoT, due to the possibly different dynamics of each physical process, the sizes of the status updates for different devices are often different and each status update typically requires multiple transmission slots. By taking into account such multi-time slot transmissions with non niform sizes of esas da es under noisy channels, the problem of joint device scheduling and status sampling is studied in order to minimize the average age of information (AoI) at the destination. This stochastic problem is formulated as an infinite horizon average cost Markov decision process (MDP). The monotonicity of the value function of the MDP is characterized and then used to show that the optimal scheduling and sampling policy is threshold-based with respect to the AoI at each device. To overcome the curse of dimensionality, a low-complexity suboptimal policy is proposed through a semi-randomized base policy and linear approximated value functions. The proposed suboptimal policy is shown to exhibit a similar structure to the optimal policy, which provides a structural base for its effective performance. A structure-aware algorithm is then developed to obtain the suboptimal policy. Simulation results illustrate the structures of the optimal policy and show a near-optimal AoI performance of the proposed suboptimal policy.

The effective operation of time-critical Internet of things (IoT) applications requires real-time reporting of fresh status information of underlying physical processes. In this paper, a real-time IoT monitoring system is considered, in which the IoT devices sample a physical process with a sampling cost and send the status packet to a given destination with an updating cost. This joint status sampling and updating process is designed to minimize the average age of information (AoI) at the destination node under an average energy cost constraint at each device. This stochastic problem is formulated as an infinite horizon average cost constrained Markov decision process (CMDP) and transformed into an unconstrained Markov decision process (MDP) using a Lagrangian method. For the single IoT device case, the optimal policy for the CMDP is shown to be a randomized mixture of two deterministic policies for the unconstrained MDP, which is of threshold type. This reveals a fundamental tradeoff between the average AoI at the destination and the sampling and updating costs. Then, a structure-aware optimal algorithm to obtain the optimal policy of the CMDP is proposed and the impact of the wireless channel dynamics is studied while demonstrating that channels having a larger mean channel gain and less scattering can achieve better AoI performance. For the case of multiple IoT devices, a low-complexity semi-distributed suboptimal policy is proposed with the updating control at the destination and the sampling control at each IoT device. Then, an online learning algorithm is developed to obtain this policy, which can be implemented at each IoT device and requires only the local knowledge and small signaling from the destination. The proposed learning algorithm is shown to converge almost surely to the suboptimal policy. Simulation results show the structural properties of the optimal policy for the single IoT device case; and show that the proposed policy for multiple IoT devices outperforms a zero-wait baseline policy, with average AoI reductions reaching up to 33%.

The effective operation of time-critical Internet of things (IoT) applications requires real-time reporting of fresh status information of underlying physical processes. In this paper, a real-time IoT monitoring system is considered, in which an IoT device samples a physical process with a sampling cost and sends the status packet to a given destination with an updating cost. The optimal status sampling and updating process is designed to minimize the average age of information (AoI) at the destination under an average energy cost constraint at the device. This stochastic optimization problem is formulated as an infinite horizon average cost constrained Markov decision process (CMDP). Using a Lagrangian method, the CMDP is transformed into an unconstrained Markov decision process (MDP), where the optimal policy for the CMDP is a randomized mixture of two deterministic policies for the unconstrained MDP. It is shown that the optimal policy for the unconstrained MDP is of threshold type with respect to the AoI state of the device and the AoI state of the destination. This reveals a fundamental tradeoff between the average AoI of the destination and the sampling and updating costs. Then, a structure-aware algorithm is proposed to obtain the optimal policy for the CMDP. Finally, the impact of the wireless channel dynamics on the system performance is studied while demonstrating that channels having a large mean channel gain and less scattering can achieve better AoI performance.

In this paper, we investigate cooperative caching for spectrum access in cognitive radio networks. By cooperative caching, we mean that the unlicensed secondary base station (SBS) can cache certain primary contents to serve primary users, in exchange for the opportunities to access the licensed spectrum. We consider the joint optimization of caching and scheduling of the SBS to maximize the weighted average number of satisfied secondary requests under the average available time constraint and the cache capacity constraint. This problem is a mixed-integer bilinear programming, which is challenging in general. By exploring the special structure of the problem, we first show that the optimal caching satisfies a cache-split structure and the optimal scheduling satisfies a rate-ratio structure. Then, based on these optimality properties, we transform the original problem into a simplified joint cache splitting and SU partitioning optimization problem, and propose an efficient algorithm to solve it optimally. Moreover, we investigate the impacts of the primary and secondary content popularity distributions on the system performance. Numerical results verify the theoretical analysis and provide some counter-intuitive insights.

Caching and multicasting at base stations are two promising approaches to support massive content delivery over wireless networks. However, existing scheduling designs do not fully exploit the advantages of the two approaches. In this paper, we consider the optimal dynamic multicast scheduling to jointly minimize the average delay, power, and fetching costs for cache-enabled content-centric wireless networks. We formulate this stochastic optimization problem as an infinite horizon average cost Markov decision process (MDP).By usingrelative value iterationand special structures of the request queue dynamics, we analyze the properties of the value function and the state-action cost function of the MDP for both the uniform and nonuniform channel cases. Based on these properties, we show that the optimal policy, which is adaptive to the request queue state, has a switch structure in the uniform case and a partial switch structure in the nonuniform case. Moreover, in the uniform case with two contents, we show that the switch curve is monotonically non-decreasing. Motivated by the switch structures of the optimal policy, we propose a low-complexity suboptimal policy, which exhibits similar switch structures to the optimal policy, and design a low-complexity algorithm to compute this policy.

Caching in wireless device-to-device (D2D) networks can be utilized to offload data traffic during peak times. However, the design of incentive mechanisms is challenging due to the heterogeneous preference and selfish nature of user terminals (UTs). In this paper, we propose an incentive mechanism in which the base station (BS) rewards those UTs that share contents with others using D2D communication. We study the cost minimization problem for the BS and the utility maximization problem for each UT. In particular, the BS determines the rewarding policy to minimize his total cost, while each UT aims to maximize his utility by choosing his caching policy. We formulate the conflict among UTs and the tension between the BS and the UTs as a Stackelberg game. We show the existence of the equilibrium and propose an iterative gradient algorithm (IGA) to obtain the Stackelberg Equilibrium. Extensive simulations are carried out to evaluate the performance of the proposed caching scheme and comparisons are drawn with several baseline caching schemes with no incentives. Numerical results show that the caching scheme under our incentive mechanism outperforms other schemes in terms of the BS serving cost and the utilities of the UTs.

Caching at small base stations (SBSs) has demonstrated significant benefits in alleviating the backhaul requirement in heterogeneous cellular networks (HetNets). While many existing works focus on what contents to cache at each SBS, an equally important problem is what contents to deliver so as to satisfy dynamic user demands given the cache status. In this paper, we study optimal content delivery in cache-enabled HetNets by considering the inherent multicast capability of wireless medium. We consider stochastic content multicast scheduling to jointly minimize the average network delay and power costs under a multiple access constraint. We establish a content-centric request queue model and formulate this stochastic optimization problem as an infinite horizon average cost Markov decision process (MDP). By using relative value iteration and special properties of the request queue dynamics, we characterize some properties of the value function of the MDP. Based on these properties, we show that the optimal multicast scheduling policy is of threshold type. Then, we propose a structure-aware optimal algorithm to obtain the optimal policy. We also propose a low-complexity suboptimal policy, which possesses similar structural properties to the optimal policy, and develop a low-complexity algorithm to obtain this policy.

Caching and multicasting at base stations are two promising approaches to support massive content delivery over wireless networks. However, existing scheduling designs do not make full use of the advantages of the two approaches. In this paper, we consider the optimal dynamic multicast scheduling to jointly minimize the average delay, power and fetching costs for cache-enabled content-centric wireless networks. We formulate this stochastic optimization problem as an infinite horizon average cost Markov decision process (MDP). It is well-known to be a difficult problem and there generally only exist numerical solutions. By using relative value iteration algorithm and the special structures of the request queue dynamics, we analyze the properties of the value function and the state-action cost function of the MDP for both the uniform and nonuniform channel cases. Based on these properties, we show that the optimal policy, which is adaptive to the request queue state, has a switch structure in the uniform case and a partial switch structure in the nonuniform case. Moreover, in the uniform case with two contents, we show that the switch curve is monotonically non-decreasing. The optimality properties obtained in this paper can provide design insights for practical networks.

Optimal queueing control of multi-hop networks remains a challenging problem even in the simplest scenarios. In this paper, we consider a two-hop half-duplex relaying system with random channel connectivity. The relay is equipped with a finite buffer. We focus on stochastic link selection and transmission rate control to maximize the average system throughput subject to a half-duplex constraint. We formulate this stochastic optimization problem as an infinite horizon average cost Markov decision process (MDP), which is well-known to be a difficult problem. By using sample-path analysis and exploiting the specific problem structure, we first obtain an equivalent Bellman equation with reduced state and action spaces. By using relative value iteration algorithm, we analyze the properties of the value function of the MDP. Then, we show that the optimal policy has a threshold-based structure by characterizing the supermodularity in the optimal control. Based the threshold-based structure and Markov chain theory, we further simplify the original complex stochastic optimization problem to a static optimization problem over a small discrete feasible set and propose a simple algorithm to solve the static optimization problem. Furthermore, we obtain the closed-form optimal threshold for the symmetric case. The analytical results obtained in this paper also provide design insights for two-hop relaying systems with multiple relays equipped with finite relay buffers.

Most existing works about scheduling and resource allocation for orthogonal frequency division multiplexing (OFDM) based two-way relay networks have focused on immediate relay forwarding. In this paper, we consider relay buffering in delay-tolerant networks. The relay node is aided by two buffers and one for each user, so that it can adaptively decide when to buffer the received packets or to forward them according to the instantaneous channel and queue conditions. We formulate the joint optimization of subcarrier assignment, transmission mode selection (direct or relay mode), and relay strategy selection (buffering or forwarding), for maximizing the long-term average throughput. An efficient dual-based algorithm is proposed to characterize the optimal policy. Simulation results show that relay buffering can significantly enhance the long-term throughput in OFDM bidirectional transmission systems.