The restless multi-armed bandit (RMAB) framework is a popular model with applications across a wide variety of fields. However, its solution is hindered by the exponentially growing state space (with respect to the number of arms) and the combinatorial action space, making traditional reinforcement learning methods infeasible for large-scale instances. In this paper, we propose GINO-Q, a three-timescale stochastic approximation algorithm designed to learn an asymptotically optimal index policy for RMABs. GINO-Q mitigates the curse of dimensionality by decomposing the RMAB into a series of subproblems, each with the same dimension as a single arm, ensuring that complexity increases linearly with the number of arms. Unlike recently developed Whittle-index-based algorithms, GINO-Q does not require RMABs to be indexable, enhancing its flexibility and applicability. Our experimental results demonstrate that GINO-Q consistently learns near-optimal policies, even for non-indexable RMABs where Whittle-index-based algorithms perform poorly, and it converges significantly faster than existing baselines.

The impact of communication on decision-making systems has been extensively studied under the assumption of dedicated communication channels. We instead consider communicating through actions, where the message is embedded into the actions of an agent which interacts with the environment in a Markov decision process (MDP) framework. We conceptualize the MDP environment as a finite-state channel (FSC), where the actions of the agent serve as the channel input, while the states of the MDP observed by another agent (i.e., receiver) serve as the channel output. Here, we treat the environment as a communication channel over which the agent communicates through its actions, while at the same time, trying to maximize its reward. We first characterize the optimal information theoretic trade-off between the average reward and the rate of reliable communication in the infinite-horizon regime. Then, we propose a novel framework to design a joint control/coding policy, termed Act2Comm, which seamlessly embeds messages into actions. From a communication perspective, Act2Comm functions as a learning-based channel coding scheme for non-differentiable FSCs under input-output constraints. From a control standpoint, Act2Comm learns an MDP policy that incorporates communication capabilities, though at the cost of some control performance. Overall, Act2Comm effectively balances the dual objectives of control and communication in this environment. Experimental results validate Act2Comm's capability to enable reliable communication while maintaining a certain level of control performance.

We introduce and validate the lottery codec hypothesis, which states that untrained subnetworks within randomly initialized networks can serve as synthesis networks for overfitted image compression, achieving rate-distortion (RD) performance comparable to trained networks. This hypothesis leads to a new paradigm for image compression by encoding image statistics into the network substructure. Building on this hypothesis, we propose LotteryCodec, which overfits a binary mask to an individual image, leveraging an over-parameterized and randomly initialized network shared by the encoder and the decoder. To address over-parameterization challenges and streamline subnetwork search, we develop a rewind modulation mechanism that improves the RD performance. LotteryCodec outperforms VTM and sets a new state-of-the-art in single-image compression. LotteryCodec also enables adaptive decoding complexity through adjustable mask ratios, offering flexible compression solutions for diverse device constraints and application requirements.

We consider a Markov decision process (MDP) in which actions prescribed by the controller are executed by a separate actuator, which may behave adversarially. At each time step, the controller selects and transmits an action to the actuator; however, the actuator may deviate from the intended action to degrade the control reward. Given that the controller observes only the sequence of visited states, we investigate whether the actuator can covertly deviate from the controller's policy to minimize its reward without being detected. We establish conditions for covert adversarial behavior over an infinite time horizon and formulate an optimization problem to determine the optimal adversarial policy under these conditions. Additionally, we derive the asymptotic error exponents for detection in two scenarios: (1) a binary hypothesis testing framework, where the actuator either follows the prescribed policy or a known adversarial strategy, and (2) a composite hypothesis testing framework, where the actuator may employ any stationary policy. For the latter case, we also propose an optimization problem to maximize the adversary's performance.

This paper focuses on the information freshness of finite-state Markov sources, using the uncertainty of information (UoI) as the performance metric. Measured by Shannon’s entropy, UoI can capture not only the transition dynamics of the Markov source but also the different evolutions of information quality caused by the different values of the last observation. We consider an information update system with$M$finite-state Markov sources transmitting information to a remote monitor via$m$communication channels ($1\le m < M$). At each time, only$m$Markov sources can be selected to transmit their latest information to the remote monitor. Our goal is to explore the optimal scheduling policy to minimize the sum-UoI of the Markov sources. The problem is formulated as a restless multi-armed bandit (RMAB). We relax the RMAB and then decouple the relaxed problem into$M$single bandit problems. Importantly, analyzing the single bandit problem provides useful properties with which the relaxed problem reduces to maximizing a concave and piecewise linear function, allowing us to develop a gradient method to solve the relaxed problem and obtain its optimal policy. By rounding up the optimal policy for the relaxed problem, we obtain an index policy for the original RMAB problem. Notably, the proposed index policy is universal in the sense that it applies to general RMABs with bounded cost functions. Moreover, we show that our policy is asymptotically optimal as$m$and$M$tend to$\infty $with$m/M$fixed. In non-asymptotic cases, numerical results demonstrate that our index policy is near-optimal and performs as well as the celebrated Whittle index policy in the problems that are Whittle-indexable. Unlike the Whittle index policy, our index policy does not require “indexability”; the indices can be computed regardless of indexability in the Whittle’s sense. Thus, our index policy is a promising alternative method for the class of RMABs of concern: it can be used when the Whittle index policy is not viable and it performs as well as the Whittle index policy even when the Whittle index policy is viable.

Most IoT systems use random access protocols for wireless communication. This paper considers an IoT node that generates periodic traffic to be delivered to a destination over a random access network; each packet of the node is expected to be delivered before its deadline. We say that the node is in a stressed period if, within a time interval, its successive packets miss their deadlines. In many systems, the worst-case performance is significantly affected by stressed periods. Characterizing the stochastic properties of stressed periods is thus of fundamental importance. In this paper, we use a fluid flow model to approximate the evolution of the buffer occupancy (i.e., backlog) at the transmitting node. We derive a relationship between buffer occupancy and delay and formally define a stressed period via a time interval in which the buffer occupancy exceeds a certain threshold. With this model, we analyze the dynamics of the buffer occupancy evolution and obtain the probability distributions of stressed period duration and delay. Real network experiments show that our model can well approximate the distributions of stressed period duration and delay in practical WiFi networks. The theoretical results of this paper can be used to analyze the robustness and worst-case performance of IoT monitoring and control systems built on random access networks.

This paper proposes using the uncertainty of information (UoI), measured by Shannon’s entropy, as a metric for information freshness. We consider a system in which a central monitor observes M binary Markov processes through m communication channels (m

Remote state estimation over a wireless network is of significant importance in many industrial applications, such as condition monitoring. In these cases, sensors deliver their data to remote estimators through wireless channels, which makes communication reliability a core issue. In this article, we propose an error-aware design to carry out network scheduling and channel allocation according to estimation error covariance and channel quality, with the aim of minimizing the total estimation error covariance. We develop multidimensional conflict graphs to model the interference and conflicts, and on this basis, a two-phase heuristic algorithm is further proposed to adaptively assign slots and channels at each superframe. Theoretical analysis and extensive simulations are given to show the effectiveness of our error-aware design in preventing the estimation error covariance from diverging, and hence able to improve the accuracy of remote estimation and monitoring.

WirelessHART is one of the most widely used communication standards in industrial wireless networks. In order to meet the stringent real-time requirements in industrial applications, WirelessHART incorporates many designs including the time slotted channel hopping mechanism that enables dynamic time scheduling and channel allocation. In this paper, we study the problem of joint transmission scheduling and channel allocation aiming to minimize the end-to-end delay of multiple flows in multihop WirelessHART networks. We propose a new network model based on a multidimensional scheduling space spanned by flow-link-channel-slot tuples. A multidimensional conflict graph is then established to depict the conflict relationships among the tuples. Based on this, the original delay minimization problem is formulated as an integer program, which however is difficult to solve due to its significantly large scale. To this end, we develop an iterative hop-wise scheduling algorithm by transforming the original problem into a series of maximum weighted independent set problems. We derive theoretical analysis on the schedulability and the performance bound of the proposed algorithm. In addition, we show that our results can be easily extended to accommodate more general scenarios. Finally, extensive simulation results are provided to demonstrate the effectiveness of the algorithm.