The quality of experience (QoE) delivered by video conferencing systems is significantly influenced by accurately estimating the time-varying available bandwidth between the sender and receiver. Bandwidth estimation for real-time communications remains an open challenge due to rapidly evolving network architectures, increasingly complex protocol stacks, and the difficulty of defining QoE metrics that reliably improve user experience. In this work, we propose a deployed, human-in-the-loop, data-driven framework for bandwidth estimation to address these challenges. Our approach begins with training objective QoE reward models derived from subjective user evaluations to measure audio and video quality in real-time video conferencing systems. Subsequently, we collect roughly 1M network traces with objective QoE rewards from real-world Microsoft Teams calls to curate a bandwidth estimation training dataset. We then introduce a novel distributional offline reinforcement learning (RL) algorithm to train a neural-network-based bandwidth estimator aimed at improving QoE for users. Our real-world A/B test demonstrates that the proposed approach reduces the subjective poor call ratio by 11.41% compared to the baseline bandwidth estimator. Furthermore, the proposed offline RL algorithm is benchmarked on D4RL tasks to demonstrate its generalization beyond bandwidth estimation.

Real-time video applications require accurate bandwidth estimation (BWE) to maintain user experience across varying network conditions. However, increasing network heterogeneity challenges general-purpose BWE algorithms, necessitating solutions that adapt to end-user environments. While widely adopted, heuristic-based methods are difficult to individualize without extensive domain expertise. Conversely, online reinforcement learning (RL) offers ease of customization but neglects prior domain expertise and suffers from sample inefficiency. Thus, we present Merlin, an imitation learning-based solution that replaces the manual parameter tuning of heuristic-based methods with data-driven updates to streamline end-user personalization. Our key insight is that transforming heuristic-based BWE algorithms into neural networks facilitates data-driven personalization. Merlin utilizes Behavioral Cloning to efficiently learn from offline telemetry logs, capturing heuristic policies without live network interactions. The cloned policy can then be seamlessly tailored to end user network conditions through online finetuning. In real intercontinental videoconferencing calls, Merlin matches our heuristic's policy with no statistically significant differences in user quality of experience (QoE). Finetuning Merlin's control policy to end-user environments enables QoE improvements of up to 7.8 % compared to the heuristic policy. Lastly, our IL-based design performs competitively with current state-of-the-art online RL techniques but converges with 80 % fewer videoconferencing samples, facilitating practical end-user personalization.

The quality of experience (QoE) delivered by video conferencing systems to end users depends in part on correctly estimating the capacity of the bottleneck link between the sender and the receiver over time. Bandwidth estimation for real-time communications (RTC) remains a significant challenge, primarily due to the continuously evolving heterogeneous network architectures and technologies. From the first bandwidth estimation challenge which was hosted at ACM MMSys 2021, we learned that bandwidth estimation models trained with reinforcement learning (RL) in simulations to maximize network-based reward functions may not be optimal in reality due to the sim-to-real gap and the difficulty of aligning network-based rewards with user-perceived QoE. This grand challenge aims to advance bandwidth estimation model design by aligning reward maximization with user-perceived QoE optimization using offline RL and a real-world dataset with objective rewards which have high correlations with subjective audio/video quality in Microsoft Teams. All models submitted to the grand challenge underwent initial evaluation on our emulation platform. For a comprehensive evaluation under diverse network conditions with temporal fluctuations, top models were further evaluated on our geographically distributed testbed by using each model to conduct 600 calls within a 12-day period. The winning model is shown to deliver comparable performance to the top behavior policy in the released dataset. By leveraging real-world data and integrating objective audio/video quality scores as rewards, offline RL can therefore facilitate the development of competitive bandwidth estimators for RTC.

The canonical solution methodology for finite constrained Markov decision processes (CMDPs), where the objective is to maximize the expected infinite-horizon discounted rewards subject to the expected infinite-horizon discounted costs' constraints, is based on convex linear programming (LP). In this brief, we first prove that the optimization objective in the dual linear program of a finite CMDP is a piecewise linear convex (PWLC) function with respect to the Lagrange penalty multipliers. Next, we propose a novel, provably optimal, two-level gradient-aware search (GAS) algorithm which exploits the PWLC structure to find the optimal state-value function and Lagrange penalty multipliers of a finite CMDP. The proposed algorithm is applied in two stochastic control problems with constraints for performance comparison with binary search (BS), Lagrangian primal-dual optimization (PDO), and LP. Compared with the benchmark algorithms, it is shown that the proposed GAS algorithm converges to the optimal solution quickly without any hyperparameter tuning. In addition, the convergence speed of the proposed algorithm is not sensitive to the initialization of the Lagrange multipliers.

In this paper, we investigate reinforcement learning (RL) in multi-fidelity environments and enhance the performance of the agent using cross-correlated data.We introduce a multifidelity estimator based on control variates to reduce the variance in state-action value function estimation.By employing this estimator, we develop a multifidelity Monte Carlo RL (MFMCRL) algorithm that boosts agent learning in high-fidelity settings.Our experiments show that, given a finite highfidelity sample budget, the MFMCRL agent outperforms an RL agent relying solely on high-fidelity interactions for policy optimization.

Non-orthogonal multiple access (NOMA) is a key technology to enable massive machine type communications (mMTC) in 5G networks and beyond. In this paper, NOMA is applied to improve the random access efficiency in high-density spatially-distributed multi-cell wireless IoT networks, where IoT devices contend for accessing the shared wireless channel using an adaptive$p$-persistent slotted Aloha protocol. To enable a capacity-optimal network, a novel formulation of random channel access management is proposed, in which the transmission probability of each IoT device is tuned to maximize the geometric mean of users’ expected capacity. It is shown that the network optimization objective is high dimensional and mathematically intractable, yet it admits favourable mathematical properties that enable the design of efficient data-driven algorithmic solutions which do not require a priori knowledge of the channel model or network topology. A centralized model-based algorithm and a scalable distributed model-free algorithm, are proposed to optimally tune the transmission probabilities of IoT devices and attain the maximum capacity. The convergence of the proposed algorithms to the optimal solution is further established based on convex optimization and game-theoretic analysis. Extensive simulations demonstrate the merits of the novel formulation and the efficacy of the proposed algorithms.

In this paper, we apply the Non-Orthogonal Multiple Access (NOMA) technique to improve the massive channel access of a wireless IoT network where solar-powered Unmanned Aerial Vehicles (UAVs) relay data from IoT devices to remote servers. Specifically, IoT devices contend for accessing the shared wireless channel using an adaptive p-persistent slotted Aloha protocol; and the solar-powered UAVs adopt Successive Interference Cancellation (SIC) to decode multiple received data from IoT devices to improve access efficiency. To enable an energy-sustainable capacity-optimal network, we study the joint problem of dynamic multi-UAV altitude control and multi-cell wireless channel access management of IoT devices as a stochastic control problem with multiple energy constraints. We first formulate this problem as a Constrained Markov Decision Process (CMDP), and propose an online model-free Constrained Deep Reinforcement Learning (CDRL) algorithm based on Lagrangian primal-dual policy optimization to solve the CMDP. Extensive simulations demonstrate that our proposed algorithm learns a cooperative policy in which the altitude of UAVs and channel access probability of IoT devices are dynamically controlled to attain the maximal long-term network capacity while ensuring energy sustainability of UAVs, outperforming baseline schemes. The proposed CDRL agent can be trained on a small network, yet the learned policy can efficiently manage networks with a massive number of IoT devices and varying initial states, which can amortize the cost of training the CDRL agent.

Quantum computing is a computational paradigm with the potential to outperform classical methods for a variety of problems. Proposed recently, the Quantum Approximate Optimization Algorithm (QAOA) is considered as one of the leading candidates for demonstrating quantum advantage in the near term. QAOA is a variational hybrid quantum-classical algorithm for approximately solving combinatorial optimization problems. The quality of the solution obtained by QAOA for a given problem instance depends on the performance of the classical optimizer used to optimize the variational parameters. In this paper, we formulate the problem of finding optimal QAOA parameters as a learning task in which the knowledge gained from solving training instances can be leveraged to find high-quality solutions for unseen test instances. To this end, we develop two machine-learning-based approaches. Our first approach adopts a reinforcement learning (RL) framework to learn a policy network to optimize QAOA circuits. Our second approach adopts a kernel density estimation (KDE) technique to learn a generative model of optimal QAOA parameters. In both approaches, the training procedure is performed on small-sized problem instances that can be simulated on a classical computer; yet the learned RL policy and the generative model can be used to efficiently solve larger problems. Extensive simulations using the IBM Qiskit Aer quantum circuit simulator demonstrate that our proposed RL- and KDE-based approaches reduce the optimality gap by factors up to 30.15 when compared with other commonly used off-the-shelf optimizers.

Non-orthogonal multiple access (NOMA) has been considered as a promising solution for improving the spectrum efficiency of next-generation wireless networks. In this paper, the performance of a p-persistent slotted ALOHA system in support of NOMA transmissions is investigated. Specifically, wireless users can choose to use high or low power for data transmissions with certain probabilities. To achieve the maximum network throughput, an analytical framework is developed to analyze the successful transmission probability of NOMA and long term average throughput of users involved in the non-orthogonal transmissions. The feasible region of the maximum number of concurrent users using high and low power to ensure successful NOMA transmissions are quantified. Based on analysis, an algorithm is proposed to find the optimal transmission probabilities for users to choose high and low power to achieve the maximum system throughput. In addition, the impact of power settings on the network performance is further investigated. Simulations are conducted to validate the analysis.

Energy harvesting is a promising solution to enable energy sustainable operation of IoT devices. Especially for under-water IoT network as it is difficult and costly for underwater IoT devices to replace the battery. Unlike traditional power supply, energy harvesting from green sources is a random process and is dependent on the charging environment, which poses new challenges for provisioning quality of services of IoT networks. Due to the high cost for low-powered IoT devices to update its energy status with the scheduler, distributed transmission protocol is more desirable for the IoT networks. In this work, we consider an IoT network where IoT devices use adaptive p-persistent ALOHA for data transmissions. Each IoT device can contend for channel access only when it is ready, i.e., it has a data for transmission and it harvests enough energy for communications. Due to stochastic energy harvesting and random access, the number of ready devices in the network may vary. As such, an analytical framework is first developed using a discrete Markov model to analyze the average number of ready devices. Next, an optimization problem is formulated to maximize the system throughput by tuning the transmission probability. Given that the wireless environment is unknown at different IoT devices, e.g., total number of contending devices, data arrival rates of other IoT devices, a multi-agent reinforcement learning algorithm is introduced for each device to autonomously tune the transmission probability in a distributed manner. In addition, game theory is applied to design the reward function to ensure an equilibrium and to closely approach the optimal parameter setting. Numerical results show that the proposed learning algorithm can greatly improve the throughput performance comparing with other algorithms.

In this article, we study an energy sustainable Internet-of-Underwater Things (IoUT) network with tidal energy harvesting. Specifically, an analytical model is first developed to analyze the performance of the IoUT network, characterizing the stochastic nature of energy harvesting and traffic demands of IoUT nodes, and the salient features of acoustic communication channels. It is found that the spatial uncertainty resulting from underwater acoustic communication may cause a severe fairness issue. As such, an optimization problem is formulated to maximize the network throughput under fairness constraints, by tuning the random access parameters of each node. Given the global network information, including the number of nodes, energy harvesting rates, communication distances, etc., the optimization problem can be efficiently solved with the Branch and Bound (BnB) method. Considering a realistic network where the network information may not be available at the IoUT nodes, we further propose a multiagent reinforcement learning approach for each node to autonomously adapt the random access parameter based on the interactions with the dynamic network environment. The numerical results show that the proposed learning algorithm greatly improves the throughput performance compared with the existing solutions, and approaches the derived theoretical bound.

In this paper, we analytically study the performance of opportunistic multi-channel bonding protocol supporting delay-sensitive multimedia services. We consider a multi-channel system shared by IEEE 802.11ac users who can transmit over multiple channels and legacy users who can only transmit over one single channel. By analyzing the channel bonding behavior of IEEE 802.11ac users and the random access of legacy users, bonding probability and successful bonding probability of IEEE 802.11ac users can be derived. Furthermore, the access delays of both legacy and 802.11ac users are analyzed. According to the analytical results, the network capacity which quantifies the maximum number of multimedia flows that can be supported with guaranteed delay is then presented. Additionally, the impacts of different parameters such as traffic data rate on the network capacity are investigated. Our analytical results show that channel bonding is favorable when the secondary channels are underutilized. But channel bonding should be disabled when there are already intense contentions from legacy users. Based on the analytical results, we propose a heuristic bonding policy which can provide important guidelines to control the number of flows to satisfy the QoS requirement and achieve the maximum network capacity. Extensive simulations have been conducted to validate the analytical results.

Radio frequency (RF) energy harvesting is a promising technology that enables self-sustainable wireless Internet of Things (IoT) networks. In this article, we analyze the energy harvesting performance of a Wi-Fi-based IoT network, where a large number of IoT devices are connected via Wi-Fi for both data communication and energy transfer. Applying probability theory and statistical geometry, we first develop an analytical model to study the energy sustainability of wireless IoT devices with Wi-Fi charging, which operate in an active/charging mode and access the channel using carrier sensing multiple access with collision avoidance (CSMA/CA) protocol. Based on the analysis, we derive the necessary and sufficient conditions for the AP beaconing frequency and the charging period of IoT devices to ensure that a network with a general random topology is long-term energy sustainable. It is shown that transmission collisions due to random access result in too much energy consumption that makes it difficult to achieve energy sustainability of IoT devices. To maximize the total network throughput while ensuring long-term energy sustainability of Wi-Fi IoT devices, a distributed energy-sustainable throughput-optimal algorithm is proposed for user charging period selection. Extensive simulations using NS-3 validate the analysis and demonstrate the efficiency of the proposed algorithm.

To support bandwidth-intensive services such as virtual reality video applications, next generation WLANs will allow users to transmit over multiple channels for high data rate transmissions. In this paper, an analytical model is developed to study the performance of the dynamic channel bonding in IEEE 802.11ac, and non-contiguous channel aggregation in IEEE 802.11ax, with coexisting legacy single channel users. By modeling the transmissions of single channel and multi-channel users with and without channel bonding as a two-level renewal process, the bonding probability of multi-channel users, along with the throughput of different users in each channel, are derived. Our analysis shows that multi-channel users can boost their throughput at the cost of degraded throughput of legacy users. Furthermore, it is shown that 802.11ax provides higher spectrum utilization compared to 802.11ac, while 802.11ac provides a friendlier coexistence with single channel users. Based on the analysis, a heuristic algorithm for primary channel selection is further proposed to maximize the throughput of multi-channel users. Extensive simulations using NS-3 validate the analysis and demonstrate the efficiency of the proposed channel selection algorithm.

Long Term Evolution (LTE) deployment in the unlicensed spectrum is considered a promising solution to overcome spectrum shortage. To ensure fair coexistence among unlicensed users, two Load Based Equipment (LBE) access technologies are introduced in European Telecommunications Standards Institute (ETSI) standard. However, it is not clear whether the two LBE protocols can ensure fair channel access between unlicensed Licensed Assisted Access (LAA) and Wi-Fi users. To this end, renewal theory based analytical models are developed to study the performance of these two LBE random access protocols. Specifically, the throughput performance of Wi-Fi users and LAA users is first derived and compared. Our results show that both options may not achieve throughput fairness among Wi-Fi and LAA users if the key protocol parameters are not fine tuned. Generally option A favors Wi-Fi users, while option B favors LAA users. To improve the fairness performance, channel access parameters in both protocols should be adapted to network conditions in order to achieve the best coexisting performance in terms of both fairness and network throughput. The analysis provides important guidance for the implementation of Listen-before-Talk based access mechanisms in LAA. Extensive simulations using NS-3 are conducted to validate the accuracy of the models.

In this paper, we investigate the access mechanisms of LTE Licensed Assisted Access (LAA) co-existing with Wi-Fi over the unlicensed band. To this end, we first develop an analytical model to study the performance of existing Load Based Equipment (LBE) MAC for Unlicensed Long Term Evolution (U-LTE), identify the fairness issues, and quantify the reservation overhead of the protocol. To maximize the network throughput and ensure fair spectrum sharing of U- LTE and Wi-Fi, we propose a hybrid MAC protocol that combines the best features of LBE MAC and Frame Based Equipment (FBE) MAC. A two-level renewal process-based model is also developed to analyze the throughput performance of the proposed MAC. By jointly optimizing the sleep period and the contention window size of U-LTE, the best co-existing performance in terms of the total network throughput and throughput fairness of U-LTE and Wi-Fi can be achieved, with minimal reservation overhead. Extensive simulations using NS-3 validate the analysis and demonstrate the efficiency of the proposed MAC protocol.

In this paper, an analytical model is developed to study the performance of distributed and opportunistic multichannel bonding in IEEE 802.11ac WLANs, with co-existing legacy IEEE 802.11a/b/g users. By modeling the transmissions of legacy users and ac users with and without channel bonding in each channel as a two-level renewal process, the channel bonding probability of ac users in each secondary channel is derived. Based on the bonding probability, the throughput of legacy users and ac users can be analyzed respectively. Our analysis shows that ac users with bonding capabilities achieve higher throughput at the cost of degraded throughput of legacy users. The overall network throughput also decreases due to the increased contention level imposed by ac users in secondary channels. Based on the analysis, we further propose a channel selection scheme for ac users to select the best primary channel, in order to mitigate the contentions in the network and attain the maximal network throughput. Extensive simulations using NS-3 validate the analysis and demonstrate the efficiency of the proposed channel selection scheme.

An analytical model is developed to evaluate the network performance of an IEEE 802.11ac Wireless Local Area Network (WLAN) in support of delay sensitive video services over multiple channels. Specifically, the channel bonding probability and the channel access delay of wireless users are analyzed, considering the contentions among legacy and ac users in the same channel and across multiple channels. Based on the analysis, the network capacity region, i.e., the maximum number of traffic flows can be supported with the bounded delay performance in a multi-channel WLAN with and without channel bonding, is then derived. Our analysis shows that channel bonding can greatly improve the network capacity when the channel is under-utilized with a small number of legacy users co-existing with the ac users; yet channel bonding is not always favorable and it may degrade the network capacity when the number of legacy users increases due to the increased contentions in the network. The analysis provides important guidance for effective admission control and channel bonding strategies to guarantee the bonded service delay of realtime applications. Extensive simulations validate the analysis.

In this paper, an analytical framework is developed to study the performance of opportunistic channel bonding in IEEE 802.11 WLANs. Specifically, we consider a WLAN operating on multiple channels shared by both legacy users and IEEE 802.11ac users with channel bonding capability. By capturing the opportunistic channel bonding from the IEEE 802.11ac users in the primary channel and the random access of legacy users in the secondary channels, we derive the successful channel bonding probability, and the throughput of both legacy and IEEE 802.11ac users. Our analysis shows that with multi-channel bonding, ac users achieves a higher throughput at the cost of reduced throughput of legacy users in the secondary channels. The channel bonding achieves a higher total network throughput only when there is no legacy user in the secondary channels, and the total throughput decreases when legacy users exist due to the increased contentions in the secondary channels. The analysis provides important guidance for the deployment of multi-channel WLANs where ac users should select a proper primary channel to maximize its bonding opportunity and to attain the maximum throughput. Extensive simulations are conducted to validate the analysis.