Bitrate adaptation (or ABR) plays a crucial role in shaping the user QoE of low-latency live streaming (LLLS) applications. However, the unique characteristics of modern WebRTC-based LLLS systems render traditional ABR paradigms inefficient or even ineffective in real-world scenarios. This motivates us to revisit the bitrate adaptation problem within this complex yet realistic context and propose Salmon, an innovative bitrate adaptation framework designed for WebRTC-based LLLS applications. Salmon pragmatically addresses challenges arising from new QoE objectives, two-stream handovers, user interaction behaviors, and application-specific signal semantics. Deployed on a leading e-commerce LLLS platform, Salmon demonstrates significant performance gains over state-of-the-art algorithms. Notably, in low-bandwidth conditions, Salmon reduces startup delay by 17.7%, stall by 11%, frame jumps by 41.2%, and switching rate by 25.9×.

Low latency live streaming (LLLS) leverages chunked transfer encoding (CTE) to substantially reduce end-to-end latency. However, this paradigm introduces a cascade of challenges for adaptive bitrate (ABR) algorithms: (1) the sending idle periods between chunks in CTE render bandwidth measurement difficult and prone to error; (2) bandwidth prediction in LLLS is an irregular time series forecasting with uncertain future segment size, leading to a circular prediction dependency; (3) stochastic uncertainty within LLLS, such as fluctuating idle time, leads to imprecise buffer evolution and ABR degradation. In this paper, we tackle the issues and present AAR, a novel LLLS framework that comprises 3 key modules: (1) accurate bandwidth measurement that leverages a server-side Flag to identify burst transmission and isolate chunks. We further propose to fuse our two learning and heuristic-based algorithms via confidence estimation; (2) bandwidth prediction via conditional normalizing flow to simultaneously learn joint variable distributions. We further propose a bitrate-aware transformer to capture the intrinsic circular relationships as backbone flow condition; (3) an LLLS tailored ABR with a novel and robust objective to maximize the minimum Quality of Experience (QoE) under uncertainty. We propose two theorems to derive the min solution via download time bounds, and we maximize the QoE via Model Predictive Controller (MPC) with LLLS tailored state evolution. Extensive experiments on real-world network traces demonstrate that AAR significantly outperforms baselines with absolute error reduction by 11%-83% for measurement and up to 17% for prediction. We also improve QoE by up to 102% across all tested network conditions.

Recently, protecting the Intellectual Property (IP) of deep neural networks (DNNs) has attracted attention from researchers. This is because training DNN models can be costly especially when acquiring and labeling training data require domain expertise. DNN watermarking and fingerprinting are two techniques proposed to prevent DNN IP infringement. Although these two techniques achieve high performance on defending against previously proposed DNN stealing attacks, researchers recently show that both of them are ineffective against generative model inversion attacks. Specifically, an adversary inverts training data from well-trained DNNs and uses the inverted data to train DNNs from scratch such that DNN watermarking and fingerprinting are both bypassed. This novel model stealing strategy shows that data inverted from victim models can be effectively exploited by adversaries, which poses a new threat to the IP protection of DNNs. To combat this new threat, one potential solution is to enable defenders to prove ownership on data inverted from models being protected. If the training data of a suspected model, which can be disclosed via the judicial process, are proven to be data inverted from victim models, then IP infringement is detected. This research direction is currently underexplored. In this paper, we fill the gap in the literature to investigate countermeasures against this emerging threat. We propose a simple but effective method, called InverseDataInspector (IDI), to detect whether data are inverted from victim models. Specifically, our method first extracts features from both the inverted data and victim models. These features are then combined and used for training classifiers. Experimental results demonstrate that our method achieves high performance on detecting inverted data and also generalizes to new generative model inversion methods that are not seen when training classifiers.

Long propagation delay that causes throughput degradation of underwater acoustic networks (UWANs) is a critical issue in the medium access control (MAC) protocol design in UWANs. This paper develops a deep reinforcement learning (DRL) based MAC protocol for UWANs, referred to as delayed-reward deep-reinforcement learning multiple access (DR-DLMA), to maximize the network throughput by judiciously utilizing the available time slots resulted from propagation delays or not used by other nodes. In the DR-DLMA design, we first put forth a new DRL algorithm, termed asdelayed-reward deep Q-network (DR-DQN). Then we formulate the multiple access problem in UWANs as a reinforcement learning (RL) problem by defining state, action, and reward in the parlance of RL, and thereby realizing the DR-DLMA protocol. In traditional DRL algorithms, e.g., the original DQN algorithm, the agent can get access to the “reward” from the environment immediately after taking an action. In contrast, in our design, the “reward” (i.e., the ACK packet) is only available after twice the one-way propagation delay after the agent takes an action (i.e., to transmit a data packet). The essence of DR-DQN is to incorporate the propagation delay into the DRL framework and modify the DRL algorithm accordingly. In addition, in order to reduce the cost of online training deep neural network (DNN), we provide a nimble training mechanism for DR-DQN. The optimal network throughputs in various cases are given as a benchmark. Simulation results show that our DR-DLMA protocol with nimble training mechanism can: (i) find the optimal transmission strategy when coexisting with other protocols in a heterogeneous environment; (ii) outperform state-of-the-art MAC protocols (e.g., slotted FAMA and DOTS) in a homogeneous environment; and (iii) greatly reduce energy consumption and run-time compared with DR-DLMA with traditional DNN training mechanism.

This paper investigates a futuristic spectrum sharing paradigm for heterogeneous wireless networks with imperfect channels. In the heterogeneous networks, multiple wireless networks adopt different medium access control (MAC) protocols to share a common wireless spectrum and each network is unaware of the MACs of others. This paper aims to design a distributed deep reinforcement learning (DRL) based MAC protocol for a particular network, and the objective of this network is to achieve a global$\alpha$-fairness objective. In the conventional DRL framework, feedback/reward given to the agent is always correctly received, so that the agent can optimize its strategy based on the received reward. In our wireless application where the channels are noisy, the feedback/reward (i.e., the ACK packet) may be lost due to channel noise and interference. Without correct feedback, the agent (i.e., the network user) may fail to find a good solution. Moreover, in the distributed protocol, each agent makes decisions on its own. It is a challenge to guarantee that the multiple agents will make coherent decisions and work together to achieve the same objective, particularly in the face of imperfect feedback channels. To tackle the challenge, we put forth (i) a feedback recovery mechanism to recover missing feedback information, and (ii) a two-stage action selection mechanism to aid coherent decision making to reduce transmission collisions among the agents. Extensive simulation results demonstrate the effectiveness of these two mechanisms. Last but not least, we believe that the feedback recovery mechanism and the two-stage action selection mechanism can also be used in general distributed multi-agent reinforcement learning problems in which feedback information on rewards can be corrupted.

This paper investigates a new medium access control (MAC) protocol for multi-channel heterogeneous networks (HetNets) based on deep reinforcement learning (DRL), referred to as multi-channel deep-reinforcement learning multiple access (MC-DLMA). Specifically, we consider a HetNet where different radio networks adopt different MAC protocols to transmit data packets to a common access point on different wireless channels. Three key challenges for the MC-DLMA node are (i) no environmental knowledge is known in advance; (ii) the channels in HetNets are allocated to nodes using different MAC protocols; (iii) the capacities of different channels may be different. The main goal of MC-DLMA is to find an optimal access policy to transmit on those pre-allocated channels and expedite more efficient spectrum utilization. Due to the complex temporal correlation of spectrum states in HetNets, the traditional DRL technique, e.g., original deep Q-network (DQN) algorithm, is no longer applicable to our problem. In our MC-DLMA design, an advanced class of recurrent neural network, termed as Gated Recurrent Unit (GRU), is embedded into the original DQN technique to aggregate observations over time and reason the underlying temporal feature in multi-channel HetNets. Furthermore, we analytically give the optimal spectrum access patterns and derive the optimal throughputs in various HetNet scenarios. With judicious definitions of the state, action, and reward function in the parlance of the DRL framework, simulation results show that MC-DLMA can (i) find the optimal spectrum access strategies in various HetNets, (ii) outperform the random access policy, the whittle index policy, and the original DQN, (iii) perform cooperative transmission in a fully distributed manner in the presence of multiple agents, and (iv) adapt well to the environmental changes.

This paper investigates a new class of carrier-sense multiple access (CSMA) protocols that employ deep reinforcement learning (DRL) techniques, referred to as carrier-sense deep-reinforcement learning multiple access (CS-DLMA). The goal of CS-DLMA is to enable efficient and equitable spectrum sharing among a group of co-located heterogeneous wireless networks. Existing CSMA protocols, such as the medium access control (MAC) protocol of WiFi, are designed for a homogeneous network in which all nodes adopt the same protocol. Such protocols suffer from severe performance degradation in a heterogeneous environment where there are nodes adopting other MAC protocols. CS-DLMA aims to circumvent this problem by making use of DRL. In particular, this paper adopts α-fairness as the general objective of CS-DLMA. With α-fairness, CS-DLMA can achieve a range of different objectives (e.g., maximizing sum throughput, achieving proportional fairness, or achieving max-min fairness) when coexisting with other MACs by changing the value of α. A salient feature of CS-DLMA is that it can achieve these objectives without knowing the coexisting MACs through a learning process based on DRL. The underpinning DRL technique in CS-DLMA is deep Q-network (DQN). However, the conventional DQN algorithms are not suitable for CS-DLMA due to their uniform time-step assumption. In CSMA protocols, time steps are non-uniform in that the time duration required for carrier sensing is smaller than the duration of data transmission. This paper introduces a non-uniform time-step formulation of DQN to address this issue. Our simulation results show that CS-DLMA can achieve the general α-fairness objective when coexisting with TDMA, ALOHA, and WiFi protocols by adjusting its own transmission strategy. Interestingly, we also find that CS-DLMA is more Pareto efficient than other CSMA protocols, e.g., p-persistent CSMA, when coexisting with WiFi. Although this paper focuses on the use of our non-uniform time-step DQN formulation in wireless networking, we believe this new DQN formulation can also find use in other domains.

This paper considers the problem of efficient spectrum utilization in heterogeneous wireless networks (HetNets), wherein different radio networks adopt different medium access control (MAC) protocols to transmit data packets to a common access point on different wireless channels. To allow emerging radio nodes to transmit on those pre-allocated channels and to expedite more efficient spectrum utilization, we exploit the advanced deep reinforcement learning technique to develop a new generation of MAC protocols, referred to as multi-channel deep-reinforcement learning multiple access (MC-DLMA). The emerging radio nodes that adopt MC-DLMA can make full use of the underutilized spectrum resource and maximize the sum throughput of the overall HetNet by learning the transmission patterns of the existing radio nodes. For benchmarking, we derive the optimal throughputs analytically and demonstrate that MC-DLMA can achieve the near-optimal results. Moreover, compared with other baselines (e.g., the Whittle Index policy and the random access policy), our MC-DLMA can significantly improve the sum throughput of the HetNet in various scenarios.

This paper investigates a deep reinforcement learning (DRL)-based MAC protocol for heterogeneous wireless networking, referred to as a Deep-reinforcement Learning Multiple Access (DLMA). Specifically, we consider the scenario of a number of networks operating different MAC protocols trying to access the time slots of a common wireless medium. A key challenge in our problem formulation is that we assume our DLMA network does not know the operating principles of the MACs of the other networks-i.e., DLMA does not know how the other MACs make decisions on when to transmit and when not to. The goal of DLMA is to be able to learn an optimal channel access strategy to achieve a certain pre-specified global objective. Possible objectives include maximizing the sum throughput and maximizing α-fairness among all networks. The underpinning learning process of DLMA is based on DRL. With proper definitions of the state space, action space, and rewards in DRL, we show that DLMA can easily maximize the sum throughput by judiciously selecting certain time slots to transmit. Maximizing general α-fairness, however, is beyond the means of the conventional reinforcement learning (RL) framework. We put forth a new multi-dimensional RL framework that enables DLMA to maximize general α-fairness. Our extensive simulation results show that DLMA can maximize sum throughput or achieve proportional fairness (two special classes of α-fairness) when coexisting with TDMA and ALOHA MAC protocols without knowing they are TDMA or ALOHA. Importantly, we show the merit of incorporating the use of neural networks into the RL framework (i.e., why DRL and not just traditional RL): specifically, the use of DRL allows DLMA (i) to learn the optimal strategy with much faster speed and (ii) to be more robust in that it can still learn a near-optimal strategy even when the parameters in the RL framework are not optimally set.

This paper investigates the use of deep reinforcement learning (DRL) in the design of a "universal" MAC protocol referred to as Deep-reinforcement Learning Multiple Access (DLMA). The design framework is partially inspired by the vision of DARPA SC2, a 3-year competition whereby competitors are to come up with a clean-slate design that "best share spectrum with any network(s), in any environment, without prior knowledge, leveraging on machine-learning technique". While the scope of DARPA SC2 is broad and involves the redesign of PHY, MAC, and Network layers, this paper's focus is narrower and only involves the MAC design. In particular, we consider the problem of sharing time slots among a multiple of time-slotted networks that adopt different MAC protocols. One of the MAC protocols is DLMA. The other two are TDMA and ALOHA. The DRL agents of DLMA do not know that the other two MAC protocols are TDMA and ALOHA. Yet, by a series of observations of the environment, its own actions, and the rewards - in accordance with the DRL algorithmic framework - a DRL agent can learn the optimal MAC strategy for harmonious co-existence with TDMA and ALOHA nodes. In particular, the use of neural networks in DRL (as opposed to traditional reinforcement learning) allows for fast convergence to optimal solutions and robustness against perturbation in hyper- parameter settings, two essential properties for practical deployment of DLMA in real wireless networks.