We propose a privacy-preserving uplink over-the-air computation (AirComp) method, termed FLORAS, for single-input single-output (SISO) wireless federated learning (FL) systems. From the perspective of communication designs, FLORAS eliminates the requirement of channel state information at the transmitters (CSIT) by leveraging the properties of orthogonal sequences. From the privacy perspective, we prove that FLORAS offers bothitem-levelandclient-leveldifferential privacy (DP) guarantees. Moreover, by properly adjusting the system parameters, FLORAS can flexibly achieve different DP levels at no additional cost. A new FL convergence bound is derived which, combined with the privacy guarantees, allows for a smooth tradeoff between the achieved convergence rate and differential privacy levels. Experimental results demonstrate the advantages of FLORAS compared with the baseline AirComp method, and validate that the analytical results can guide the design of privacy-preserving FL with different tradeoff requirements on the model convergence and privacy levels.

Reinforcement Learning from Human Feedback (RLHF) has emerged as a pivotal technique for aligning artificial intelligence systems with human values, achieving remarkable success in fine-tuning large language models. However, existing RLHF frameworks often assume that human preferences are relatively homogeneous and can be captured by a single, unified reward model. This assumption overlooks the inherent diversity and heterogeneity across individuals, limiting the adaptability of RLHF to personalized scenarios and risking misalignments that can diminish user satisfaction and trust in AI systems. In this paper, we address these challenges by introducing Low-Rank Adaptation (LoRA) into the personalized RLHF framework. We apply LoRA in the parameter space of the aggregation of all personalized reward functions, thereby enabling efficient learning of personalized reward models from potentially limited local datasets. Our approach exploits potential shared structures among the local ground-truth reward models while allowing for individual adaptation, without relying on restrictive assumptions about shared representations as in prior works. We further establish sample complexity guarantees for our method. Theoretical analysis demonstrates the effectiveness of the proposed approach in capturing both shared and individual-specific structures within heterogeneous human preferences, addressing the dual challenge of personalization requirements and practical data constraints. Experimental results on real-world datasets corroborate the efficiency of our algorithm in the personalized RLHF setting.

Transformers have shown potential in solving wireless communication problems, particularly via in-context learning (ICL), where models adapt to new tasks through prompts without requiring model updates. However, prior ICL-based Transformer models rely on deep architectures with many layers to achieve satisfactory performance, resulting in substantial storage and computational costs. In this work, we propose CHain Of thOught Symbol dEtection (CHOOSE), a CoT-enhanced shallow Transformer framework for wireless symbol detection. By introducing autoregressive latent reasoning steps within the hidden space, CHOOSE significantly improves the reasoning capacity of shallow models (1-2 layers) without increasing model depth. This design enables lightweight Transformers to achieve detection performance comparable to much deeper models, making them well-suited for deployment on resource-constrained mobile devices. Experimental results demonstrate that our approach outperforms conventional shallow Transformers and achieves performance comparable to that of deep Transformers, while maintaining storage and computational efficiency. This represents a promising direction for implementing Transformer-based algorithms in wireless receivers with limited computational resources.

Pre-trained Transformers, through in-context learning (ICL), have demonstrated exceptional capabilities to adapt to new tasks using example prompts without model update. Transformer-based wireless receivers, where prompts consist of the pilot data in the form of transmitted and received signal pairs, have shown high estimation accuracy when pilot data are abundant. However, pilot information is often costly and limited in practice. In this work, we propose the DEcision Feedback INContExt Detection (DEFINED) solution as a new wireless receiver design, which bypasses channel estimation and directly performs symbol detection using the (sometimes extremely) limited pilot data. The key innovation in DEFINED is the proposed decision feedback mechanism in ICL, where we sequentially incorporate the detected symbols into the prompts to improve the detections for subsequent symbols. Extensive experiments across a broad range of wireless communication settings demonstrate that DEFINED achieves significant performance improvements, in some cases only needing a single pilot pair.

Large Language Models (LLMs) rely on the contextual information embedded in examples/demonstrations to perform in-context learning (ICL). To mitigate the risk of LLMs potentially leaking private information contained in examples in the prompt, we introduce a novel data-adaptive differentially private algorithm called **AdaDPSyn** to generate synthetic examples from the private dataset and then use these synthetic examples to perform ICL. The objective of AdaDPSyn is to adaptively adjust the noise level in the data synthesis mechanism according to the inherent statistical properties of the data, thereby preserving high ICL accuracy while maintaining formal differential privacy guarantees. A key innovation in AdaDPSyn is the *Precision-Focused Iterative Radius Reduction* technique, which dynamically refines the aggregation radius - the scope of data grouping for noise addition - based on patterns observed in data clustering, thereby minimizing the amount of additive noise. We conduct extensive experiments on standard benchmarks and compare AdaDPSyn with DP few-shot generation algorithm (Tang et al., 2023). The experiments demonstrate that AdaDPSyn not only outperforms DP few-shot generation, but also maintains high accuracy levels close to those of non-private baselines, providing an effective solution for ICL with privacy protection.

An intriguing property of the Transformer is its ability to perform in-context learning (ICL), where the Transformer can solve different inference tasks without parameter updating based on the contextual information provided by the corresponding input-output demonstration pairs. It has been theoretically proved that ICL is enabled by the capability of Transformers to perform gradient-descent algorithms (Von Oswald et al., 2023a; Bai et al., 2024). This work takes a step further and shows that Transformers can perform learning-to-optimize (L2O) algorithms. Specifically, for the ICL sparse recovery (formulated as LASSO) tasks, we show that a K-layer Transformer can perform an L2O algorithm with a provable convergence rate linear in K. This provides a new perspective explaining the superior ICL capability of Transformers, even with only a few layers, which cannot be achieved by the standard gradient-descent algorithms. Moreover, unlike the conventional L2O algorithms that require the measurement matrix involved in training to match that in testing, the trained Transformer is able to solve sparse recovery problems generated with different measurement matrices. Besides, Transformers as an L2O algorithm can leverage structural information embedded in the training tasks to accelerate its convergence during ICL, and generalize across different lengths of demonstration pairs, where conventional L2O algorithms typically struggle or fail. Such theoretical findings are supported by our experimental results.

Language recognition tasks are fundamental in natural language processing (NLP) and have been widely used to benchmark the performance of large language models (LLMs). These tasks also play a crucial role in explaining the working mechanisms of transformers. In this work, we focus on two representative tasks in the category of regular language recognition, known as 'even pairs' and 'parity check', the aim of which is to determine whether the occurrences of certain subsequences in a given sequence are even. Our goal is to explore how a one-layer transformer, consisting of an attention layer followed by a linear layer, learns to solve these tasks by theoretically analyzing its training dynamics under gradient descent. While even pairs can be solved directly by a one-layer transformer, parity check need to be solved by integrating Chain-of-Thought (CoT), either into the inference stage of a transformer well-trained for the even pairs task, or into the training of a one-layer transformer. For both problems, our analysis shows that the joint training of attention and linear layers exhibits two distinct phases. In the first phase, the attention layer grows rapidly, mapping data sequences into separable vectors. In the second phase, the attention layer becomes stable, while the linear layer grows logarithmically and approaches in direction to a max-margin hyperplane that correctly separates the attention layer outputs into positive and negative samples, and the loss decreases at a rate of $O(1/t)$. Our experiments validate those theoretical results.

Although transformers have demonstrated impressive capabilities for in-context learning (ICL) in practice, theoretical understanding of the underlying mechanism that allows transformers to perform ICL is still in its infancy. This work aims to theoretically study the training dynamics of transformers for in-context classification tasks. We demonstrate that, for in-context classification of Gaussian mixtures under certain assumptions, a single-layer transformer trained via gradient descent converges to a globally optimal model at a linear rate. We further quantify the impact of the training and testing prompt lengths on the ICL inference error of the trained transformer. We show that when the lengths of training and testing prompts are sufficiently large, the prediction of the trained transformer approaches the ground truth distribution of the labels. Experimental results corroborate the theoretical findings.

The radio frequency (RF) spectrum is essential for wireless communication but is becoming increasingly limited due to the rapid growth in device usage. Real-time spectrum sensing facilitates dynamic spectrum sharing, but conventional methods face significant challenges, including the high power consumption of analog-to-digital converters (ADCs) and the computational demands of Fast Fourier Transforms (FFTs). To address these limitations, prior work introduced a frequencydomain analog signal processor. This processor includes a digitally tunable narrow-bandpass filter implemented with programmable dispersion-engineered elements and a scalable pathsharing delayed signal combiner. However, the naive spectrum sweeping method employed in this design remains highly timeand energy-intensive. In this work, we improve the spectrum sensing efficiency of the analog signal processor by co-designing a sensing matrix generation method with a decoder-based transformer for in-context spectrum recovery. Specifically, we introduce a novel algorithm for sensing matrix generation that leverages the hardware design of the analog signal processor. We show that the generated sensing matrices can be interpreted as part of the well-designed prompts for a transformer with specifically designed parameter matrices to solve the sparse spectrum sensing problem efficiently through its in-context learning capability. To characterize the efficiency of the in-context learningenabled spectrum sensing approach, we provide rigorous theoretical guarantees on the in-context spectrum sensing and evaluate the performances through empirical results. Compared to baseline approaches, our method achieves significant improvements in accuracy.

Large language models (LLMs) exhibit impressive in‑context learning (ICL) capabilities, yet the quality of their predictions is fundamentally limited by the few costly labeled demonstrations that can fit into a prompt. Meanwhile, there exist vast and continuously growing amounts of unlabeled data that may be closely related to the ICL task. How to utilize such unlabeled data to provably enhance the performance of ICL thus becomes an emerging fundamental question. In this work, we propose a novel augmented ICL framework, in which the prompt includes a small set of labeled examples alongside a block of unlabeled inputs. We focus on the multi-class linear classification setting and demonstrate that, with chain-of-thought (CoT) prompting, a multi-layer transformer can effectively emulate an expectation–maximization (EM) algorithm. This enables the transformer to implicitly extract useful information from both labeled and unlabeled data, leading to provable improvements in ICL accuracy. Moreover, we show that such a transformer can be trained via teacher forcing, with its parameters converging to the desired solution at a linear rate. Experiments demonstrate that the augmented ICL framework consistently outperforms conventional few-shot ICL, providing empirical support for our theoretical findings. To the best of our knowledge, this is the first theoretical study on the impact of unlabeled data on the ICL performance of transformers.

Reinforcement Learning from Human Feedback (RLHF) has emerged as a key technique for post‑training large language models. Despite its empirical success, the theoretical understanding of RLHF is still limited, as learning the KL-regularized target with only preference feedback poses additional challenges compared with canonical RL. Existing works mostly study the reward-based Bradley-Terry (BT) preference model, and extend classical designs utilizing optimism or pessimism. This work, instead, considers the general preference model (whose practical relevance has been observed recently) and obtains performance guarantees with major, order-wise improvements over existing ones. Surprisingly, these results are derived from algorithms that directly use empirical estimates (i.e., greedy sampling), as opposed to constructing optimistic or pessimistic estimates in previous works. This insight has a deep root in the unique structural property of the optimal policy class under the KL-regularized target, and we further specialize it to the BT model, highlighting the surprising sufficiency of greedy sampling in RLHF.

This paper investigates a hybrid learning framework for reinforcement learning (RL) in which the agent can leverage both an offline dataset and online interactions to learn the optimal policy. We present a unified algorithm and analysis and show that augmenting confidence-based online RL algorithms with the offline dataset outperforms any pure online or offline algorithm alone and achieves state-of-the-art results under two learning metrics, i.e., sub-optimality gap and online learning regret. Specifically, we show that our algorithm achieves a sub-optimality gap $\tilde{O}( \sqrt{1/(N_0/ \mathtt{C}(\pi^\star| \rho)+N_1} ) )$, where $\mathtt{C}(\pi^\star|\rho)$ is a new concentrability coefficient, $N_0$ and $N_1$ are the numbers of offline and online samples, respectively. For regret minimization, we show that it achieves a constant $\tilde{O}( \sqrt{N_1/(N_0/\mathtt{C}(\pi^{-}|\rho)+N_1)} )$ speed-up compared to pure online learning, where $\mathtt{C}(\pi^-|\rho)$ is the concentrability coefficient over all sub-optimal policies. Our results also reveal an interesting separation on the desired coverage properties of the offline dataset for sub-optimality gap minimization and regret minimization. We further validate our theoretical findings in several experiments in special RL models such as linear contextual bandits and Markov decision processes (MDPs).

Distributionally robust offline reinforcement learning (RL) aims to find a policy that performs the best under the worst environment within an uncertainty set using an offline dataset collected from a nominal model. While recent advances in robust RL focus on Markov decision processes (MDPs), robust non-Markovian RL is limited to planning problem where the transitions in the uncertainty set are known. In this paper, we study the learning problem of robust offline non-Markovian RL. Specifically, when the nominal model admits a low-rank structure, we propose a new algorithm, featuring a novel dataset distillation and a lower confidence bound (LCB) design for robust values under different types of the uncertainty set. We also derive new dual forms for these robust values in non-Markovian RL, making our algorithm more amenable to practical implementation. By further introducing a novel type-I concentrability coefficient tailored for offline low-rank non-Markovian decision processes, we prove that our algorithm can find an$\epsilon $-optimal robust policy using$O(1/\epsilon ^{2})$offline samples. Moreover, we extend our algorithm to the case when the nominal model does not have specific structure. With a new type-II concentrability coefficient, the extended algorithm also enjoys polynomial sample efficiency under all different types of the uncertainty set.

In multi-task reinforcement learning (RL) under Markov decision processes (MDPs), the presence of shared latent structures among multiple MDPs has been shown to yield significant benefits to the sample efficiency compared to single-task RL. In this paper, we investigate whether such a benefit can extend to more general sequential decision making problems such as predictive state representations (PSRs). The main challenge here is that the large and complex model space makes it hard to identify what types of common latent structure of multi-task PSRs can reduce the model complexity and improve sample efficiency. To this end, we posit a joint model class for tasks and use the notion of $\eta$-bracketing number to quantify its complexity; this number also serves as a general metric to capture the similarity of tasks and thus determines the benefit of multi-task over single-task RL. We first study upstream multi-task learning over PSRs, in which all tasks share the same observation and action spaces. We propose a provably efficient algorithm UMT-PSR for finding near-optimal policies for all PSRs, and demonstrate that the advantage of multi-task learning manifests if the joint model class of PSRs has a smaller $\eta$-bracketing number compared to that of individual single-task learning. We further investigate downstream learning, in which the agent needs to learn a new target task that shares some commonalities with the upstream tasks via a similarity constraint. By exploiting the learned PSRs from the upstream, we develop a sample-efficient algorithm that provably finds a near-optimal policy. Upon specialization to some examples with small $\eta$-bracketing numbers, our results further highlight the benefit compared to directly learning a single-task PSR.

The general sequential decision-making problem, which includes Markov decision processes (MDPs) and partially observable MDPs (POMDPs) as special cases, aims at maximizing a cumulative reward by making a sequence of decisions based on a history of observations and actions over time. Recent studies have shown that the sequential decision-making problem is statistically learnable if it admits a low-rank structure modeled by predictive state representations (PSRs). Despite these advancements, existing approaches typically involve oracles or steps that are computationally intractable. On the other hand, the upper confidence bound (UCB) based approaches, which have served successfully as computationally efficient methods in bandits and MDPs, have not been investigated for more general PSRs, due to the difficulty of optimistic bonus design in these more challenging settings. This paper proposes the first known UCB-type approach for PSRs, featuring a novel bonus term that upper bounds the total variation distance between the estimated and true models. We further characterize the sample complexity bounds for our designed UCB-type algorithms for both online and offline PSRs. In contrast to existing approaches for PSRs, our UCB-type algorithms enjoy computational tractability, last-iterate guaranteed near-optimal policy, and guaranteed model accuracy.

In this paper, we consider federated reinforcement learning for tabular episodic Markov Decision Processes (MDP) where, under the coordination of a central server, multiple agents collaboratively explore the environment and learn an optimal policy without sharing their raw data. While linear speedup in the number of agents has been achieved for some metrics, such as convergence rate and sample complexity, in similar settings, it is unclear whether it is possible to design a *model-free* algorithm to achieve linear *regret* speedup with low communication cost. We propose two federated Q-Learning algorithms termed as FedQ-Hoeffding and FedQ-Bernstein, respectively, and show that the corresponding total regrets achieve a linear speedup compared with their single-agent counterparts, while the communication cost scales logarithmically in the total number of time steps $T$. Those results rely on an event-triggered synchronization mechanism between the agents and the server, a novel step size selection when the server aggregates the local estimates of the state-action values to form the global estimates, and a set of new concentration inequalities to bound the sum of non-martingale differences. This is the first work showing that linear regret speedup and logarithmic communication cost can be achieved by model-free algorithms in federated reinforcement learning.

The problem of two-player zero-sum Markov games has recently attracted increasing interests in theoretical studies of multi-agent reinforcement learning (RL). In particular, for finite-horizon episodic Markov decision processes (MDPs), it has been shown that model-based algorithms can find an $\epsilon$-optimal Nash Equilibrium (NE) with the sample complexity of $O(H^3SAB/\epsilon^2)$, which is optimal in the dependence of the horizon $H$ and the number of states $S$ (where $A$ and $B$ denote the number of actions of the two players, respectively). However, none of the existing model-free algorithms can achieve such an optimality. In this work, we propose a model-free stage-based algorithm and show that it achieves the same sample complexity as the best model-based algorithm, and hence for the first time demonstrate that model-free algorithms can enjoy the same optimality in the $H$ dependence as model-based algorithms. The main improvement of the dependency on $H$ arises by leveraging the popular variance reduction technique based on the reference-advantage decomposition previously used only for single-agent RL. However, such a technique relies on a critical monotonicity property of the value function, which does not hold in Markov games due to the update of the policy via the coarse correlated equilibrium (CCE) oracle. Thus, to extend such a technique to Markov games, our algorithm features a key novel design of updating the reference value functions as the pair of optimistic and pessimistic value functions whose value difference is the smallest in the history in order to achieve the desired improvement in the sample efficiency.

Federated representation learning (FRL) is a popular personalized federated learning (FL) framework where clients work together to train a common representation while retaining their personalized heads. Existing studies, however, largely focus on the over-parameterized regime. In this paper, we make the initial efforts to investigate FRL in the under-parameterized regime, where the FL model is insufficient to express the variations in all ground-truth models. We propose a novel FRL algorithm FLUTE, and theoretically characterize its sample complexity and convergence rate for linear models in the under-parameterized regime. To the best of our knowledge, this is the first FRL algorithm with provable performance guarantees in this regime. FLUTE features a data-independent random initialization and a carefully designed objective function that aids the distillation of subspace spanned by the global optimal representation from the misaligned local representations. On the technical side, we bridge low-rank matrix approximation techniques with the FL analysis, which may be of broad interest. We also extend FLUTE beyond linear representations. Experimental results demonstrate that FLUTE outperforms state-of-the-art FRL solutions in both synthetic and real-world tasks.

Function approximation has experienced significant success in the field of reinforcement learning (RL). Despite a handful of progress on developing theory for Nonstationary RL with function approximation under structural assumptions, existing work for nonstationary RL with general function approximation is still limited. In this work, we propose a UCB-type of algorithm LSVI-Nonstationary following the popular least-square-value-iteration (LSVI) framework. LSVI-Nonstationary features the restart mechanism and a new design of bonus term to handle nonstationarity, and performs no worse than the existing confidence-set based algorithm SW-OPEA in [1], which has been shown to outperform the existing algorithms for nonstationary linear and tabular MDPs in the small variation budget setting.

We study the problems of differentially private federated online prediction from experts against both *stochastic adversaries* and *oblivious adversaries*. We aim to minimize the average regret on $m$ clients working in parallel over time horizon $T$ with explicit differential privacy (DP) guarantees. With stochastic adversaries, we propose a **Fed-DP-OPE-Stoch** algorithm that achieves $\sqrt{m}$-fold speed-up of the per-client regret compared to the single-player counterparts under both pure DP and approximate DP constraints, while maintaining logarithmic communication costs. With oblivious adversaries, we establish non-trivial lower bounds indicating that *collaboration among clients does not lead to regret speed-up with general oblivious adversaries*. We then consider a special case of the oblivious adversaries setting, where there exists a low-loss expert. We design a new algorithm **Fed-SVT** and show that it achieves an $m$-fold regret speed-up under both pure DP and approximate DP constraints over the single-player counterparts. Our lower bound indicates that Fed-SVT is nearly optimal up to logarithmic factors. Experiments demonstrate the effectiveness of our proposed algorithms. To the best of our knowledge, this is the first work examining the differentially private online prediction from experts in the federated setting.

Transformers have achieved extraordinary success in modern machine learning due to their excellent ability to handle sequential data, especially in next-token prediction (NTP) tasks. However, the theoretical understanding of their performance in NTP is limited, with existing studies focusing mainly on asymptotic performance. This paper provides a fine-grained non-asymptotic analysis of the training dynamics of a one-layer transformer consisting of a self-attention module followed by a feed-forward layer. We first characterize the essential structural properties of training datasets for NTP using a mathematical framework based on partial orders. Then, we design a two-stage training algorithm, where the pre-processing stage for training the feed-forward layer and the main stage for training the attention layer exhibit fast convergence performance. Specifically, both layers converge sub-linearly to the direction of their corresponding max-margin solutions. We also show that the cross-entropy loss enjoys a linear convergence rate. Furthermore, we show that the trained transformer presents non-trivial prediction ability with dataset shift, which sheds light on the remarkable generalization performance of transformers. Our analysis technique involves the development of novel properties on the attention gradient and further in-depth analysis of how these properties contribute to the convergence of the training process. Our experiments further validate our theoretical findings.

The remarkable instruction-following capability of large language models (LLMs) has sparked a growing interest in automatically finding good prompts, i.e., prompt optimization. Most existing works follow the scheme of selecting from a pre-generated pool of candidate prompts. However, these designs mainly focus on the generation strategy, while limited attention has been paid to the selection method. Especially, the cost incurred during the selection (e.g., accessing LLM and evaluating the responses) is rarely explicitly considered. To overcome this limitation, this work provides a principled framework, TRIPLE, to efficiently perform prompt selection under an explicit budget constraint. TRIPLE is built on a novel connection established between prompt optimization and fixed-budget best arm identification (BAI-FB) in multi-armed bandits (MAB); thus, it is capable of leveraging the rich toolbox from BAI-FB systematically and also incorporating unique characteristics of prompt optimization. Extensive experiments on multiple well-adopted tasks using various LLMs demonstrate the remarkable performance improvement of TRIPLE over baselines while satisfying the limited budget constraints. As an extension, variants of TRIPLE are proposed to efficiently select examples for few-shot prompts, also achieving superior empirical performance.

The in-context learning (ICL) capability of pre-trained models based on the transformer architecture has received growing interest in recent years. While theoretical understanding has been obtained for ICL in reinforcement learning (RL), the previous results are largely confined to the single-agent setting. This work proposes to further explore the in-context learning capabilities of pre-trained transformer models in competitive multi-agent games, i.e., in-context game-playing (ICGP). Focusing on the classical two-player zero-sum games, theoretical guarantees are provided to demonstrate that pre-trained transformers can provably learn to approximate Nash equilibrium in an in-context manner for both decentralized and centralized learning settings. As a key part of the proof, constructional results are established to demonstrate that the transformer architecture is sufficiently rich to realize celebrated multi-agent game-playing algorithms, in particular, decentralized V-learning and centralized VI-ULCB.

We propose a novel communication design, termed random orthogonalization, for federated learning (FL) in a massive multiple-input and multiple-output (MIMO) wireless system. The key novelty of random orthogonalization comes from the tight coupling of FL and two unique characteristics of massive MIMO – channel hardening and favorable propagation. As a result, random orthogonalization can achieve natural over-the-air model aggregation without requiring transmitter side channel state information (CSI) for the uplink phase of FL, while significantly reducing the channel estimation overhead at the receiver. We extend this principle to the downlink communication phase and develop a simple but highly effective model broadcast method for FL. We also relax the massive MIMO assumption by proposing an enhanced random orthogonalization design for both uplink and downlink FL communications, that does not rely on channel hardening or favorable propagation. Theoretical analyses with respect to both communication and machine learning performance are carried out. In particular, an explicit relationship among the convergence rate, the number of clients, and the number of antennas is established. Experimental results validate the effectiveness and efficiency of random orthogonalization for FL in massive MIMO.

The recent development of reinforcement learning (RL) has boosted the adoption of online RL for wireless radio resource management (RRM). However, online RL algorithms require direct interactions with the environment, which may be undesirable given the potential performance loss due to the unavoidable exploration in RL. In this work, we first explore the use ofofflineRL algorithms in solving the RRM problem. We evaluate several state-of-the-art offline RL algorithms for a practical RRM problem that aims at maximizing a linear combination of total rates and 5-percentile rates via user scheduling. Our findings indicate that the performance of offline RL for the RRM problem is heavily contingent upon the behavior policy deployed for data collection. We propose an innovative offline RL approach utilizing heterogeneous datasets from various behavior policies. This method demonstrates that a strategic mixture of datasets enables near-optimal RL policy generation, even with suboptimal behavior policies. Additionally, we introduce two enhancements: an ensemble-based policy to augment dataset mixture training efficiency, and a novel offline-to-online strategy for seamless adaptation to new environments. Our data mixture approach achieves over 95% efficiency of an online RL agent in the absence of expert data. The ensemble algorithm notably reduces training duration by half compared to the data mixture method. Furthermore, our model, when applied with offline-to-online fine-tuning, surpasses existing benchmarks by approximately 5% in our user scheduling problem.

We propose a novel private-preserving uplink over-the-air computation (AirComp) method, termed FLORAS, for wireless federated learning (FL) systems. From the communication design perspective, FLORAS eliminates the requirement of channel state information at the transmitters (CSIT) by leveraging the properties of orthogonal sequences. From the privacy perspective, we prove that FLORAS can offer pure differential privacy (DP) guarantee, and explicitly characterize the achievable$\epsilon$-DP level as a function of the FLORAS parameter configuration. A novel FL convergence bound is derived which, combined with the pure DP guarantee, allows for a smooth tradeoff between convergence rate and DP guarantee levels. Experiments based on real-world datasets not only corroborate the theoretical findings but also empirically demonstrate the communication and privacy advantages of FLORAS over state-of-the-art AirComp methods.

General function approximation is a powerful tool to handle large state and action spaces in a broad range of reinforcement learning (RL) scenarios. However, theoretical understanding of non-stationary MDPs with general function approximation is still limited. In this paper, we make the first such an attempt. We first propose a new complexity metric called dynamic Bellman Eluder (DBE) dimension for non-stationary MDPs, which subsumes majority of existing tractable RL problems in static MDPs as well as non-stationary MDPs. Based on the proposed complexity metric, we propose a novel confidence-set based model-free algorithm called SW-OPEA, which features a sliding window mechanism and a new confidence set design for non-stationary MDPs. We then establish an upper bound on the dynamic regret for the proposed algorithm, and show that SW-OPEA is provably efficient as long as the variation budget is not significantly large. We further demonstrate via examples of non-stationary linear and tabular MDPs that our algorithm performs better in small variation budget scenario than the existing UCB-type algorithms. To the best of our knowledge, this is the first dynamic regret analysis in non-stationary MDPs with general function approximation.

This paper studies federated linear contextual bandits under the notion of user-level differential privacy (DP). We first introduce a unified federated bandits framework that can accommodate various definitions of DP in the sequential decision-making setting. We then formally introduce user-level central DP (CDP) and local DP (LDP) in the federated bandits framework, and investigate the fundamental trade-offs between the learning regrets and the corresponding DP guarantees in a federated linear contextual bandits model. For CDP, we propose a federated algorithm termed as $\texttt{ROBIN}$ and show that it is near-optimal in terms of the number of clients $M$ and the privacy budget $\varepsilon$ by deriving nearly-matching upper and lower regret bounds when user-level DP is satisfied. For LDP, we obtain several lower bounds, indicating that learning under user-level $(\varepsilon,\delta)$-LDP must suffer a regret blow-up factor at least $\min\{1/\varepsilon,M\}$ or $\min\{1/\sqrt{\varepsilon},\sqrt{M}\}$ under different conditions.

This paper investigates conservative exploration in reinforcement learning where the performance of the learning agent is guaranteed to be above a certain threshold throughout the learning process. It focuses on the tabular episodic Markov Decision Process (MDP) setting that has finite states and actions. With the knowledge of an existing safe baseline policy, an algorithms termed as StepMix is proposed to balance the exploitation and exploration while ensuring that the conservative constraint is never violated in each episode with high probability. StepMix features a unique design of a mixture policy that adaptively and smoothly interpolates between the baseline policy and the optimistic policy. Theoretical analysis shows that StepMix achieves near-optimal regret order as in the constraint-free setting, indicating that obeying the stringent episode-wise conservative constraint does not compromise the learning performance. Besides, a randomization based EpsMix algorithm is also proposed and shown the achieve the same performance as StepMix. The algorithm design and theoretical analysis are further extended to the setting where the baseline policy is not given a priori but must be learned from an offline dataset, and it is proved that similar conservative guarantee and regret can be achieved if the offline dataset is sufficiently large. Experiment results corroborate the theoretical analysis and demonstrate the effectiveness of the proposed conservative exploration strategies.

Existing theoretical studies on offline reinforcement learning (RL) mostly consider a dataset sampled directly from the target task. In practice, however, data often come from several heterogeneous but related sources. Motivated by this gap, this work aims at rigorously understanding offline RL with multiple datasets that are collected from randomly perturbed versions of the target task instead of from itself. An information-theoretic lower bound is derived, which reveals a necessary requirement on the number of involved sources in addition to that on the number of data samples. Then, a novel HetPEVI algorithm is proposed, which simultaneously considers the sample uncertainties from a finite number of data samples per data source and the source uncertainties due to a finite number of available data sources. Theoretical analyses demonstrate that HetPEVI can solve the target task as long as the data sources collectively provide a good data coverage. Moreover, HetPEVI is demonstrated to be optimal up to a polynomial factor of the horizon length. Finally, the study is extended to offline Markov games and offline robust RL, which demonstrates the generality of the proposed designs and theoretical analyses.

In this work, we investigate a general federated multitask learning (FMTL) problem where each task may be performed at multiple clients, and each client may perform multiple tasks. Although the tasks share some common representation (i.e., feature-map) that can help to learn, the distribution of the features in the feature space may vary across different tasks at different clients, which poses a significant challenge to FMTL. While non-independent and identically distributed (non-IID) local datasets at different clients are often considered detrimental to model convergence in federated learning (FL), such statistical heterogeneity in feature space may be beneficial to the generalization performance. In this work, we establish the impact of statistical feature heterogeneity on generalization, through the lens of a multi-task linear regression model. In order to leverage the feature distribution heterogeneity, we propose a novel augmented dataset based approach, and prove that under certain conditions, FMTL on heterogeneous datasets can outperform the homogeneous counterpart in terms of the generalization performance. The theoretical analysis further leads to a simple client weighting method based on optimizing the excess risk upper bound. Experimental results demonstrate that the generalization performance can be improved on a real-world dataset with the proposed method.

Most existing federated multi-armed bandits (FMAB) designs are based on the presumption that clients will implement the new design to collaborate with the server. In reality, however, it may not be possible to modify the client protocols. Motivated by this limitation, this work focuses on clients who always maximize their individual cumulative rewards, and introduces a novel idea of reward teaching, where the server guides the clients towards global optimality through implicit local reward adjustments. Under this framework, the server faces two tightly coupled tasks of bandit learning and target teaching, whose combination is non-trivial and challenging. A novel algorithm, called Teaching-After-Learning (TAL), is proposed, which encourages and discourages clients’ explorations separately. General performance analyses of TAL on regret and cost are first established when the clients’ strategies satisfy certain requirements. To particularize the results, clients with UCB or ε-greedy strategies are then considered, where novel technical approaches are developed to analyze their warm-start behaviors. The obtained guarantees concretely demonstrate that when facing these client strategies, TAL achieves logarithmic regrets while only incurring logarithmic adjustment costs, which is order-optimal w.r.t. a natural lower bound.

Reinforcement learning (RL) under changing environment models many real-world applications via nonstationary Markov Decision Processes (MDPs), and hence gains considerable interest. However, theoretical studies on nonstationary MDPs in the literature have mainly focused on tabular and linear (mixture) MDPs, which do not capture the nature of unknown representation in deep RL. In this paper, we make the first effort to investigate nonstationary RL under episodic low-rank MDPs, where both transition kernels and rewards may vary over time, and the low-rank model contains unknown representation in addition to the linear state embedding function. We first propose a parameter-dependent policy optimization algorithm called PORTAL, and further improve PORTAL to its parameter-free version of Ada-PORTAL, which is able to tune its hyper-parameters adaptively without any prior knowledge of nonstationarity. For both algorithms, we provide upper bounds on the average dynamic suboptimality gap, which show that as long as the nonstationarity is not significantly large, PORTAL and Ada-PORTAL are sample-efficient and can achieve arbitrarily small average dynamic suboptimality gap with polynomial sample complexity.

Rate adaptation is a key mechanism in current IEEE 802.11 networks and next-generation cellular systems. Observing that the operating time scale of rate adaptation is usually much smaller than the user association and scheduling, we study a joint design of wireless user association and scheduling and rate adaptation with different time scales to maximize cumulative system throughput while guaranteeing desired fairness among users. We develop a maximum-weight type user association and scheduling algorithm that combines the virtual queues (tracking the scheduling debt for each user to ensure the desired fairness guarantee) and Upper Confidence Bound (UCB) estimates in its weight measure; each selected user then adopts the UCB algorithm to perform rate adaptation in a smaller time scale. We show that our proposed algorithm yields a cumulative regret growing with the square root of the time horizon up to a logarithmic factor, and achieves zero cumulative fairness violation after a certain number of time frames. We demonstrate the efficiency of the proposed algorithm via simulations using synthetic and realistic data traces.

Catering to the proliferation of Internet of Things devices and distributed machine learning at the edge, we propose an energy harvesting federated learning (EHFL) framework in this paper. The introduction of EH implies that a client’s availability to participate in any FL round cannot be guaranteed, which complicates the theoretical analysis. We derive novel convergence bounds that capture the impact of time-varying device availabilities due to the random EH characteristics of the participating clients, for both parallel and local stochastic gradient descent (SGD) with non-convex loss functions. The results suggest that having a uniform client scheduling that maximizes the minimum number of clients throughout the FL process is desirable, which is further corroborated by the numerical experiments using a real-world FL task and a state-of-the-art EH scheduler.

We propose a novel uplink communication method, coined random orthogonalization, for federated learning (FL) in a massive multiple-input and multiple-output (MIMO) wireless system. The key novelty of random orthogonalization comes from the tight coupling of FL model aggregation and two unique characteristics of massive MIMO – channel hardening and favorable propagation. As a result, random orthogonalization can achieve natural over-the-air model aggregation without requiring transmitter side channel state information, while significantly reducing the channel estimation overhead at the receiver. Theoretical analyses with respect to both communication and machine learning performances are carried out. In particular, an explicit relationship among the convergence rate, the number of clients and the number of antennas is established. Experimental results validate the effectiveness and efficiency of random orthogonalization for FL in massive MIMO.

Motivated by the engineering application of efficient mobility management in ultra-dense wireless networks, we propose a novel cost-aware cascading bandit model with two-level actions. Compared with the standard cascading bandit model with a single-level action, this new model captures the real-world action sequence in mobility management, where the base station not only decides on an ordered neighbor cell list before measurement, but also executes the final handover decision to the target base station. We first analyze the optimal offline policy when the arm statistics are known beforehand. An online learning algorithm coined two-level Cost-aware Cascading UCB (CC-UCB) is then proposed to exploit the structure of the optimal offline policy with estimated arm statistics. Theoretical analysis shows that the cumulative regret under two-level CC-UCB scales logarithmically in time, which coincides with the asymptotic lower bound, thus is order-optimal. Simulation results corroborate the theoretical results and validate the effectiveness of two-level CC-UCB for mobility management.

As representation learning becomes a powerful technique to reduce sample complexity in reinforcement learning (RL) in practice, theoretical understanding of its advantage is still limited. In this paper, we theoretically characterize the benefit of representation learning under the low-rank Markov decision process (MDP) model. We first study multitask low-rank RL (as upstream training), where all tasks share a common representation, and propose a new multitask reward-free algorithm called REFUEL. REFUEL learns both the transition kernel and the near-optimal policy for each task, and outputs a well-learned representation for downstream tasks. Our result demonstrates that multitask representation learning is provably more sample-efficient than learning each task individually, as long as the total number of tasks is above a certain threshold. We then study the downstream RL in both online and offline settings, where the agent is assigned with a new task sharing the same representation as the upstream tasks. For both online and offline settings, we develop a sample-efficient algorithm, and show that it finds a near-optimal policy with the suboptimality gap bounded by the sum of the estimation error of the learned representation in upstream and a vanishing term as the number of downstream samples becomes large. Our downstream results of online and offline RL further capture the benefit of employing the learned representation from upstream as opposed to learning the representation of the low-rank model directly. To the best of our knowledge, this is the first theoretical study that characterizes the benefit of representation learning in exploration-based reward-free multitask RL for both upstream and downstream tasks.

In this paper, we consider a status updating system where updates are generated at a constant rate at$K$sources and sent to the corresponding recipients through a noise-free broadcast channel. We assume that perfect channel state information (CSI) is available at the transmitter before each transmission, and the transmitter is able to utilize the CSI to precode the updates. Our object is to design optimal precoding schemes to minimize the summed averageage of information(AoI) at the recipients. Under various assumptions on the size of each update$B$, the number of transmit antennas$M$, and the number of receive antennas$N$at each user, this paper identifies the corresponding age-optimal precoding and transmission scheduling strategies. Specifically, for the case when$N=1$, a round-robin based updating scheme is shown to be optimal. For the two-user systems with$N>B$or$M\notin [N:2N]$, framed updating schemes are proven to be optimal. For other cases in the two-user systems, a framed alternating updating scheme is proven to be 2-optimal.

A general framework of personalized federated multi-armed bandits (PF-MAB) is proposed, which is a new bandit paradigm analogous to the federated learning (FL) framework in supervised learning and enjoys the features of FL with personalization. Under the PF-MAB framework, a mixed bandit learning problem that flexibly balances generalization and personalization is studied. A lower bound analysis for the mixed model is presented. We then propose the Personalized Federated Upper Confidence Bound (PF-UCB) algorithm, where the exploration length is chosen carefully to achieve the desired balance of learning the local model and supplying global information for the mixed learning objective. Theoretical analysis proves that PF-UCB achieves an O(log(T)) regret regardless of the degree of personalization, and has a similar instance dependency as the lower bound. Experiments using both synthetic and real-world datasets corroborate the theoretical analysis and demonstrate the effectiveness of the proposed algorithm.

This paper presents a novel federated linear contextual bandits model, where individual clients face different $K$-armed stochastic bandits coupled through common global parameters. By leveraging the geometric structure of the linear rewards, a collaborative algorithm called Fed-PE is proposed to cope with the heterogeneity across clients without exchanging local feature vectors or raw data. Fed-PE relies on a novel multi-client G-optimal design, and achieves near-optimal regrets for both disjoint and shared parameter cases with logarithmic communication costs. In addition, a new concept called collinearly-dependent policies is introduced, based on which a tight minimax regret lower bound for the disjoint parameter case is derived. Experiments demonstrate the effectiveness of the proposed algorithms on both synthetic and real-world datasets.

Despite the significant interests and many progresses in decentralized multi-player multi-armed bandits (MP-MAB) problems in recent years, the regret gap to the natural centralized lower bound in the heterogeneous MP-MAB setting remains open. In this paper, we propose BEACON -- Batched Exploration with Adaptive COmmunicatioN -- that closes this gap. BEACON accomplishes this goal with novel contributions in implicit communication and efficient exploration. For the former, we propose a novel adaptive differential communication (ADC) design that significantly improves the implicit communication efficiency. For the latter, a carefully crafted batched exploration scheme is developed to enable incorporation of the combinatorial upper confidence bound (CUCB) principle. We then generalize the existing linear-reward MP-MAB problems, where the system reward is always the sum of individually collected rewards, to a new MP-MAB problem where the system reward is a general (nonlinear) function of individual rewards. We extend BEACON to solve this problem and prove a logarithmic regret. BEACON bridges the algorithm design and regret analysis of combinatorial MAB (CMAB) and MP-MAB, two largely disjointed areas in MAB, and the results in this paper suggest that this previously ignored connection is worth further investigation.

Consider an energy harvesting (EH) sensor that continuously monitors a system and sends time-stamped status update to a destination. The sensor harvests energy from nature and uses it to power its updating operations. The destination keeps track of the system status through the successfully received updates. With the recently introduced information freshness metric “Age of Information” (AoI), our objective is to design optimal online status updating policy to minimize the long-term average AoI at the destination, subject to the energy causality constraint at the sensor. Due to the noisy channel between the sensor and the destination, each transmitted update may be erased with a fixed probability, and the AoI at the destination will be reset to zero only when an update is successfully received. We first consider status updating without feedback available to the sensor and show that the Best-effort Uniform updating (BU) policy is optimal in a broadly defined class of online policies. We then investigate status updating with perfect feedback to the sensor and prove similar optimality of the Best-effort Uniform updating with Retransmission (BUR) policy. In order to prove the optimality of the proposed policies, for each case, we first identify a lower bound on the long-term average AoI among a broad class of online policies, and then construct a sequence of virtual policies to approach the lower bound asymptotically. Since those virtual policies are sub-optimal to the original policy, the original policy is thus optimal.

The decentralized stochastic multi-player multi-armed bandit (MP-MAB) problem, where the collision information is not available to the players, is studied in this paper. Building on the seminal work of Boursier and Perchet (2019), we propose error correction synchronization involving communication (EC-SIC), whose regret is shown to approach that of the centralized stochastic MP-MAB with collision information. By recognizing that the communication phase without collision information corresponds to the Z-channel model in information theory, the proposed EC-SIC algorithm applies optimal error correction coding for the communication of reward statistics. A fixed message length, as opposed to the logarithmically growing one in Boursier and Perchet (2019), also plays a crucial role in controlling the communication loss. Experiments with practical Z-channel codes, such as repetition code, flip code and modified Hamming code, demonstrate the superiority of EC-SIC in both synthetic and real-world datasets.

In this paper, we investigate the impact of context diversity on stochastic linear contextual bandits. As opposed to the previous view that contexts lead to more difficult bandit learning, we show that when the contexts are sufficiently diverse, the learner is able to utilize the information obtained during exploitation to shorten the exploration process, thus achieving reduced regret. We design the LinUCB-d algorithm, and propose a novel approach to analyze its regret performance. The main theoretical result is that under the diverse context assumption, the cumulative expected regret of LinUCB-d is bounded by a constant. As a by-product, our results improve the previous understanding of LinUCB and strengthen its performance guarantee.

In this paper, we consider a status updating system where the transmitter sends status updates of the signal it monitors to the destination through a rate-limited link. We consider the scenario where the status of the monitored signal only changes at discrete time points. The objective is to let the destination be synchronized with the source in a timely manner once a status change happens. What complicates the problem is that the transmission takes multiple time slots due to the link-rate constraint. Thus, the transmitter has to decide to switch or to skip a new update when the status of the monitored signal changes and it has not completed the transmission of the previous one yet. We adopt a metric called “Age of Synchronization” (AoS) to measure the “dissatisfaction” of the destination when it is desynchronized with the source. Then, the objective of this paper is to minimize the time-average AoS by designing optimal transmission policies for the transmitter. We formulate the problem as a Markov decision process (MDP) and prove the multi-threshold structure of the optimal policy. Based on that, we propose a low computational-complexity algorithm for the MDP value iteration. We then evaluate the performance of the multi-threshold policy through simulations and compare it with two baseline policies and the AoI-optimal policy.

In this paper, we consider a status updating system where updates are generated at a constant rate at K sources and sent to the corresponding recipients through a broadcast channel. We assume that perfect channel state information (CSI) is available at the transmitter before each transmission, and the additive noise is negligible at the receivers. Under various assumptions on the number of antennas at the transmitter and the size of updates, our object is to design precoding and transmission scheduling schemes for the minimization of the summed time-average Age of Information (AoI) at the recipients. We show that when the transmitter has a single antenna, precoding is unnecessary, and the optimal policy is to update each recipient in a greedy round-robin fashion. When the transmitter has multiple antennas, updating with round-robin precoding is age-optimal.

An energy-harvesting sensor node that is sending status updates to a destination is considered. The sensor is equipped with a battery of finite size to save its incoming energy, and consumes one unit of energy per status update transmission, which is delivered to the destination instantly over an error-free channel. The setting is online in which the harvested energy is revealed to the sensor causally over time after it arrives, and the goal is to design status update transmission times (policy) such that the long term average age of information (AoI) is minimized. The AoI is defined as the time elapsed since the latest update has reached at the destination. Two energy arrival models are considered: a random battery recharge (RBR) model, and an incremental battery recharge (IBR) model. In both models, energy arrives according to a Poisson process with unit rate, with values that completely fill up the battery in the RBR model, and with values that fill up the battery incrementally in a unit-by-unit fashion in the IBR model. The key approach to characterizing the optimal status update policy for both models is showing the optimality of renewal policies, in which the inter-update times follow a renewal process in a certain manner that depends on the energy arrival model and the battery size. It is then shown that the optimal renewal policy has an energy-dependent threshold structure, in which the sensor sends a status update only if the AoI grows above a certain threshold that depends on the energy available in its battery. For both the random and the incremental battery recharge models, the optimal energy-dependent thresholds are characterized explicitly, i.e., in closed-form, in terms of the optimal long term average AoI. It is also shown that the optimal thresholds are monotonically decreasing in the energy available in the battery, and that the smallest threshold, which comes in effect when the battery is full, is equal to the optimal long term average AoI.

In this paper, we investigate the impact of coding on the Age of Information (AoI) in a two- user broadcast symbol erasure channel with feedback. We assume each update consists of K symbols and the source is able to broadcast one symbol in each time slot. Due to random channel noise, the intended symbol at each user will be erased according to an independent and identically distributed (i.i.d.) Bernoulli process. A user is able to successfully decode an update if it accumulates sufficient information and successfully decodes the K symbols of the update. Assuming perfect feedback information at the source right after the transmission of each symbol, our objective is to design an adaptive coding scheme to achieve small AoI at both users. We propose a novel coding scheme to judiciously combine symbols from different updates together, and analyze the AoI at both users. Compared with a baseline greedy scheme, the proposed adaptive coding scheme improves the AoI at the weak user by orders of magnitude without compromising the AoI at the strong user.

In this paper, we examine a status updating system where updates generated by the source are sent to the monitor through an erasure channel. We assume each update consists of k symbols and the symbol erasure in each time slot follows an independent and identically distributed (i.i.d.) Bernoulli process. We assume rateless coding scheme is adopted at the transmitter and an update can be successfully decoded if k coded symbols are received successfully. We assume perfect feedback available at the source, so that it knows whether a transmitted symbol has been erased instantly. Then, at the beginning of each time slot, the source has the choice to start transmitting a new update, or continue with the transmission of the previous update if it is not delivered yet. We adopt the metric “Age of Information” (AoI) to measure the freshness of information at the destination, where the AoI is defined as the age of the latest decoded update at the destination. Our objective is to design an optimal online transmission scheme to minimize the time-average AoI. The transmission decision is based on the instantaneous AoI, the age of the update being transmitted, as well as the number of successfully delivered symbols of the update. We formulate the problem as a Markov Decision Process (MDP) and identify the monotonic threshold structure of the optimal policy. Numerical results corroborate the structural properties of the optimal solution.

A real-time status updating system is considered, in which an energy harvesting sensor is acquiring measurements regarding some physical phenomenon and sending them to a destination through an erasure channel. The setting is online, in which energy arrives in units according to a Poisson process with unit rate, with arrival times being revealed causally over time. Energy is saved in a unit-sized battery. The sensor is notified by the destination of whether updates were erased via feedback. Updates need to reach the destination successfully in a timely fashion, namely, such that the long term average age of information, defined as the time elapsed since the latest successful update has reached the destination, is minimized. First, it is shown that the optimal status update policy has a renewal structure: successful update times should constitute a renewal process. Then, threshold-greedy policies are investigated: a new update is transmitted, following a successful one, only if the age of information grows above a certain threshold; and if it is erased, then all subsequent update attempts are greedily scheduled whenever energy is available. The optimal threshold-greedy policy is then analytically derived.

In this paper, we investigate the impact of diverse user preference on learning under the stochastic multi-armed bandit (MAB) framework. We aim to show that when the user preferences are sufficiently diverse and each arm is optimal for certain users, the O(log T ) regret incurred by exploring the sub-optimal arms under the standard stochastic MAB setting can be reduced to a constant. Our intuition is that to achieve sub-linear regret, the number of times an optimal arm being pulled should scale linearly in time; when all arms are optimal for certain users and pulled frequently, the estimated arm statistics can quickly converge to their true values, thus reducing the need of exploration dramatically. We cast the problem into a stochastic linear bandits model, where both user preferences and arm states are modeled as independent and identical distributed (i.i.d) d-dimensional random vectors. After receiving a user preference vector at the beginning of each time slot, the learner pulls an arm and receives a reward as the linear product of the preference vector and the arm state vector. We also assume that the state of the pulled arm is revealed to the learner once it is pulled. We propose a Weighted Upper Confidence Bound (W-UCB) algorithm and show that it can achieve a constant regret when the user preferences are sufficiently diverse. The performance of W-UCB under general setups is also completely characterized and validated with synthetic data.

An additive white Gaussian noise energy-harvesting channel with an infinite-sized battery is considered. The energy arrival process is modeled as a sequence of independent and identically distributed random variables. The channel capacity 1/2 log(1 + P) is achievable by the so-called best-effort and save-and-transmit schemes where P denotes the battery recharge rate. This paper analyzes the save-and-transmit scheme whose transmit power is strictly less than P and the best-effort scheme as a special case of save-and-transmit without a saving phase. In the finite blocklength regime, we obtain new nonasymptotic achievable rates for these schemes that approach the capacity with gaps vanishing at rates proportional to 1/√n and ((log n)/n)1/2respectively where n denotes the blocklength. The proof technique involves analyzing the escape probability of a Markov process. When P is sufficiently large, we show that allowing the transmit power to back off from P can improve the performance for save-and-transmit. The results are extended to a block energy arrival model where the length of each energy block L grows sublinearly in n. We show that the save-and-transmit and best-effort schemes achieve coding rates that approach the capacity with gaps vanishing at rates proportional to √(L/n) and (max{log n, L}/n)1/2, respectively.

We consider an energy harvesting sensor that is sending measurement updates regarding some physical phenomenon to a destination. The sensor relies on energy harvested from nature to measure and send its updates, and is equipped with a battery of finite size to collect its harvested energy. The energy harvesting process is Poisson with unit rate, and arrives in amounts that fully recharge the battery. Our setting is online in the sense that the times of energy arrivals are revealed causally to the sensor after the energy is harvested; only the statistics of the arrival process is known a priori. Updates need to be sent in a timely manner to the destination, namely, such that the long term average age of information is minimized over the course of communication. The age of information is defined as the time elapsed since the freshest update has reached the destination. We first show that the optimal scheduling update policy is a renewal policy, and then show that it has a multi threshold structure: the sensor sends an update only if the age of information grows above a certain threshold that depends on the available energy.

We study the optimum scheduling of a multicast system, where a server transmits information to multiple users via multicasting based on requests from the users. To improve energy efficiency, the server can queue and bundle the requests from different users based on the requested contents, and serve all users requesting the same contents in a later one-time transmission. A longer waiting time can increase the average number of users served in each multicasting transmission, thus improve the energy efficiency. The higher energy efficiency is achieved at the cost of the timeliness of the information, which can be measured by using age-of-information (AoI). The goal of this paper is to identify the multicast scheduling strategy that can optimize the tradeoff between energy efficiency and AoI. Using optimum stopping theories, we develop optimum stopping rules that can minimize a cost function expressed as a weighted combination of AoI penalty function and energy efficiency, where the weight coefficient is used to adjust the tradeoff between the two. Specifically, we consider the case that the AoI penalty grows exponentially with time, and show that the optimum scheduling can be formulated as a simple threshold test with a low complexity one-step look ahead stopping rule. The proposed multicast scheduling strategy can achieve the optimum tradeoff between energy efficiency and AoI penalty.

In this paper, we propose a cost-aware cascading bandits model, a new variant of multi-armed bandits with cascading feedback, by considering the random cost of pulling arms. In each step, the learning agent chooses an {\it ordered} list of items and \congr{examines} them sequentially, until certain stopping condition is satisfied. Our objective is then to maximize the expected {\it net reward} in each step, i.e., the reward obtained in each step minus the total cost incurred in examining the items, by deciding the ordered list of items, as well as when to stop examination. We study both the offline and online settings, depending on whether the state and cost statistics of the items are known beforehand. For the offline setting, we show that the Unit Cost Ranking with Threshold 1 (UCR-T1) policy is optimal. For the online setting, we propose a Cost-aware Cascading Upper Confidence Bound (CC-UCB) algorithm, and show that the cumulative regret scales in $O(\log T)$. We also provide a lower bound for all $\alpha$-consistent policies, which scales in $\Omega(\log T)$ and matches our upper bound. The performance of the CC-UCB algorithm is evaluated with both synthetic and real-world data.

We consider an energy harvesting transmitter sending status updates regarding a physical phenomenon it observes to a receiver. Different from the existing literature, we consider a scenario where the status updates carry information about an independent message. The transmitter encodes this message into the timings of the status updates. The receiver needs to extract this encoded information, as well as update the status of the observed phenomenon. The timings of the status updates, therefore, determine both the age of information (AoI) and the message rate (rate). We study the tradeoff between the achievable message rate and the achievable average AoI. We propose several achievable schemes and compare their rate-AoI performances.

In this paper, we consider a status monitoring system where an energy harvesting sensor continuously sends time-stamped status updates to a destination. With a non-zero probability, each update will be corrupted by noise and result in an updating failure. The destination keeps track of the system status through the successfully delivered updates. We assume that there is a perfect feedback channel between the destination and the source, so that the source is aware of the updating failures once they occur. With the feedback information, our objective is to design the optimal online status updating policy to minimize the expected long-term average Age of Information (AoI) at the destination, subject to the energy causality constraint at the sensor. We propose a status updating policy called Best-effort Uniform updating with Retransmission (BUR), under which the source tries to equalize the delay between two successful updates as much as possible, and retransmits an update immediately if the previous transmission fails. We show that the BUR policy achieves the minimum expected long-term average AoI among a broad class of online policies.

An additive white Gaussian noise (AWGN) energy-harvesting (EH) channel is considered where the transmitter is equipped with an infinite-sized battery which stores energy harvested from the environment. The energy arrival process is modeled as a sequence of independent and identically distributed (i.i.d.) random variables. The capacity of this channel is known and is achievable by the so-called best-effort and save-and-transmit schemes. This paper investigates the best-effort scheme in the finite blocklength regime and establishes the first nonasymptotic achievable rate for it. The first-order term of the nonasymptotic achievable rate equals the capacity, and the second-order term is proportional to -√{logn/n}-where n denotes the blocklength. The proof technique involves analyzing the escape probability of a Markov process. In addition, we use this new proof technique to analyze the save-and-transmit and obtain a new non-asymptotic achievable rate for it, whose first-order and second-order terms achieve the capacity and the scaling -1/√n respectively. For all sufficiently large signal-to-noise ratios (SNRs), our new achievable rate outperforms the existing ones.

We study the distributed estimations of a spatially correlated random field with decentralized wireless sensor networks (WSNs). Nodes in the WSN take spatial samples of the random field, then each node estimates the values of arbitrary points on the random field by iteratively exchanging information with each other, without the need of a central controller. The objective is to minimize the time (or number of iterations) required for all nodes in the network to reach a distributed consensus on the estimation result, with mean squared error (MSE) below a certain threshold. We find the sufficient conditions for this optimization problem, and identify the asymptotically optimum solutions when time is large and the MSE threshold is small. Specifically, we propose a distributed iterative estimation algorithm that defines the procedures for both information propagation and information estimation in each iteration. The key parameters of the algorithm, including an edge weight matrix and a sample weight matrix, are designed by following the asymptotically optimum criteria. It is shown that the asymptotically optimum performance can be achieved by distributively projecting the measurement samples into a subspace related to the covariance matrices of data and noise samples. Simulation results show that all nodes in a large network can obtain accurate estimation results with only a few iterations.

In this paper, we consider a scenario where an energy harvesting sensor continuously monitors a system and sends time-stamped status updates to a destination. The destination keeps track of the system status through the received updates. We use the metric Age of Information (AoI), the time that has elapsed since the last received update was generated, to measure the “freshness” of the status information available at the destination. We assume energy arrives randomly at the sensor according to a Poisson process, and each status update consumes one unit of energy. Our objective is to design optimal online status update policies to minimize the long-term average Aol, subject to the energy causality constraint at the sensor. We consider three scenarios, i.e., the battery size is infinite, finite, and one unit only, respectively. For the infinite battery scenario, we adopt a best-effort uniform status update policy and and show that it minimizes the long-term average AoI. For the finite battery scenario, we adopt an energy-aware adaptive status update policy, and prove that it is asymptotically optimal when the battery size goes to infinity. For the last scenario where the battery size is one, we propose a threshold based status update policy. We analytically characterize the long-term average AoI under this policy, and prove it is optimal. Simulation results corroborate the theoretical bounds.

In this paper, we study the optimal energy management policy of an energy harvesting transmitter by taking both battery degradation and finite battery constraints into consideration. We consider a scenario where the sensor is able to harvest energy from the ambient environment and use it to power its transmission. The harvested energy can be used for transmission immediately without entering the equipped battery, or charged into the battery and discharged later for transmission. When the battery is charged or discharged, a cost will be incurred to account for its impact on battery degradation. We impose a long-term average cost constraint on the battery, which is translated to the average number of charge/discharge operations per unit time. At the same time, we assume the capacity of the battery is finite, and the total amount of energy stored in the battery cannot exceed its capacity. Our objective is to develop an online energy management policy to maximize the long-term average throughput of the transmitter under both the battery usage constraint and finite battery constraint. We propose an energy-aware adaptive transmission policy, which is a modified version of the optimal policy for the infinite battery case. Our analysis indicates that the energy-aware adaptive transmission policy is asymptotically optimal when the battery size is sufficiently large. Simulation results corroborate the theoretical analysis.

In this paper, we study the optimum sensing of a time-varying random event with a sensor powered by energy harvesting devices. The system aims at reconstructing a band-unlimited continuous-time random process by using discrete-time samples collected by a sensor. Due to the random nature of the harvested energy, the sensor might not have sufficient energy to perform a sensing operation at a desired time instant. We propose a best- effort Poisson sensing policy. The sensing policy defines a set of random candidate sampling instants following a Poisson point process (PPP) in the time domain. At a given candidate sampling instant, the sensor collects a sample if there is sufficient energy to do so, and does nothing otherwise. By analyzing the statistical properties of the best-effort Poisson sensing policy, we develop an optimum estimator of the underlying random event. The asymptotic mean-squared error (MSE) of the estimation are expressed as an explicit closed-form expression of several key system parameters, such as the ratio between the average energy harvesting rate and consumption rate, the time correlation of the random event of interests, and the energy allocation between sensing and transmission. %The analytical results are used to identify optimum system operation parameters. Numerical results show that the proposed best-effort Poisson sensing policy outperforms existing uniform sensing policies.

Level set estimation (LSE) is the process of classifying the region(s) that the values of an unknown function exceed a certain threshold. It has a wide range of applications such as spectrum sensing or environment monitoring. In this paper, we study the the optimal LSE of a linear random field that changes with respect to time. A linear sensor network is used to take discrete samples of the spatially-temporally correlated random field in both the space and time domain, and the sensors operate under a total power constraint. The samples are congregated at a fusion center (FC), which performs LSE of the random field by using the noisy observation of the samples. Under the Gaussian process (GP) framework, we first develop an optimal LSE algorithm that can minimize the LSE error probability. The results are then used to derive the exact LSE error probability with the assistance of frequency domain analysis. The analytical LSE error probability is expressed as an explicit function of a number of system parameters, such as the distance between two adjacent nodes, the sampling period in the time domain, the signal-to-noise ratio (SNR), and the spatial-temporal correlation of the random field. With the analytical results, we can identify the optimum node distance and sampling period that can minimize the LSE error probability.

This paper investigates the information-theoretic limits of the additive white Gaussian noise (AWGN) energy-harvesting (EH) channel in the finite blocklength regime. The EH process is characterized by a sequence of i.i.d. random variables with finite variances. We use the save-and-transmit strategy proposed by Ozel and Ulukus (2012) together with Shannon's non-asymptotic achievability bound to obtain a lower bound on the achievable rate for the AWGN EH channel. The first-order term of the lower bound on the achievable rate is equal to C and the second-order (backoff from capacity) term is proportional to equation, where n denotes the blocklength and C denotes the capacity of the EH channel, which is the same as the capacity without the EH constraints. The constant of proportionality of the backoff term is found and qualitative interpretations are provided.

This paper takes the impact of charging and discharging operations on battery degradation into consideration, and studies the optimal energy management policy for an energy harvesting communication system under a battery usage constraint. Specifically, in each time slot, we assume the harvested energy can be used to power the transmitter immediately without entering into the battery, or stored into the battery for now and retrieved later for transmission. Whenever the battery is charged or discharged, a cost will be incurred to account for its impact on battery degradation. We impose an long-term average cost constraint on the battery, which is translated to the average number of charge/discharge operations per unit time. The objective is to develop an online policy to maximize the long-term average throughput of the transmitter under energy causality constraint and the battery usage constraint. We first relax the energy causality constraint on the system, and impose an energy flow conservation constraint instead. We show that the optimal energy management policy has a double-threshold structure: if the amount of energy arrives in each time slot lies in between the two thresholds, it will be used immediately without involving the battery; otherwise, the battery will be charged or discharged accordingly to maintain a constant transmit power. We then modify the double-threshold policy slightly to accommodate the energy causality constraint, and analyze its long-term performance. We show that the system achieves the same long-term average performance, thus it is optimal.

This paper investigates the information-theoretic limits of energy-harvesting (EH) channels in the finite blocklength regime. The EH process is characterized by a sequence of i.i.d. random variables with finite variances. We use the save-andtransmit strategy proposed by Ozel and Ulukus (2012) together with Shannon's non-asymptotic achievability bound to obtain lower bounds on the achievable rates for both additive white Gaussian noise channels and discrete memoryless channels under EH constraints. The first-order terms of the lower bounds of the achievable rates are equal to C and the second-order (backoff from capacity) terms are proportional to -√log n/n, where n denotes the blocklength and C denotes the capacity of the EH channel, which is the same as the capacity without the EH constraints. The constant of proportionality of the backoff term is found and qualitative interpretations are provided.

In this paper, we study the optimal sensing scheduling problem for an energy harvesting sensor. The objective is to strategically select the sensing time such that the long-term time-average sensing performance is optimized. In the sensing system, it is assumed that the sensing performance depends on the time durations between two consecutive sensing epochs. Example applications include reconstructing a wide-sense stationary random process by using discrete-time samples collected by a sensor. We consider both scenarios where the battery size is infinite and finite, assuming the energy harvesting process is a Poisson random process. We first study the infinite battery case and identify a performance limit on the long-term time average sensing performance of the system. Motivated by the structure of the performance limit, we propose a best-effort uniform sensing policy, and prove that it achieves the limit asymptotically, thus it is optimal. We then study the finite battery case, and propose an energy-aware adaptive sensing scheduling policy. The policy dynamically chooses the next sensing epoch based on the battery level at the current sensing epoch. We show that as the battery size increases, the sensing performance under the adaptive sensing policy asymptotically converges to the limit achievable by the system with infinite battery, thus it is asymptotically optimal. The convergence rate is also analytically characterized.

Level set estimation (LSE) is the process of using noisy observations of an unknown function to estimate the region(s) where the function values lie above a given threshold. It has a wide range of applications in many scientific and engineering areas, such as spectrum sensing or environment monitoring. In this paper, we study the optimum LSE of a time-varying random field under a total power constraint. A sensor performs uniform sampling of the random field and sends the samples to a fusion center, which estimates the level set by using distorted observations of the samples. Under a total power constraint, a higher sampling rate means less energy per sample, which may negatively impact the estimation performance, but also a stronger correlation between adjacent samples, which can improve the estimation accuracy. Thus it is critical to identify the optimum sampling rate that can minimize the LSE error provability. With the help of a Gaussian process (GP) prior model, we first develop an optimum LSE algorithm based on GP regression. The exact analytical LSE error probability of the LSE algorithm is then derived by considering a number of factors, such as the power consumptions of both sensing and transmission, the power constraint of the sensor, the sampling rate, and the probability distributions of the random field. To simplify analysis, we also obtain a closed-form upper bound of the LSE error probability. The optimum sampling rate is identified by using the analytical error probabilities.

In this paper, we study the optimal sensing scheduling policy for an energy harvesting sensing system equipped with a finite battery. The objective is to strategically select the sensing epochs such that the long-term average sensing performance is optimized. In the sensing system, it is assumed that the sensing performance depends on the time duration between two consecutive sensing epochs. Example applications include reconstructing a wide-sense stationary random process by using discrete-time samples collected by a sensor. The randomness of the energy harvesting process and the finite battery constraint at the sensor make the optimal sensing scheduling very challenging. Assuming the energy harvesting process is a Poisson random process, we first identify a performance limit on the long-term average sensing performance of the system without the finite battery constraint. We then propose an energy-aware adaptive sensing scheduling policy, which dynamically chooses the next sensing epoch based on the battery level at the current sensing epoch. We show that as the battery size increases, the sensing performance under the adaptive sensing policy asymptotically converges to the performance limit of the system with an infinite battery, thus it is asymptotically optimal. The convergence rate is also analytically characterized.

In this paper, we study the optimum estimation of a continuous-time random process by using discrete-time samples taken by a sensor powered by energy harvesting power sources. The system employs a best-effort sensing scheme to cope with the stochastic nature of the energy harvesting sources. The best-effort sensing scheme defines a set of equally-spaced candidate sensing instants, and the sensor performs sensing at a given candidate sensing instant if there is sufficient energy available, and remains silent otherwise. It is shown through asymptotic analysis that when the energy harvesting rate is strictly less than the energy consumption rate, there is a non-negligible percentage of silent symbols due to energy outage. For a given average energy harvesting rate, a larger sampling period means a smaller energy outage probability and/or more energy per sample, but a weaker temporal correlation between two adjacent samples. Such a tradeoff relationship is captured by developing a closed-form expression of the estimation MSE, which analytically identifies the interactions among the various system parameters, such as the ratio between the energy harvesting rate and energy consumption rate, the sampling period, and the energy allocation between sensing and transmission. It is shown through theoretical analysis that the optimum performance can be achieved by adjusting the sampling period and sampling energy such that the average energy harvesting rate is equal to the average consumption rate.

In this paper, we consider an energy harvesting multiple access channel (MAC) where the transmitters are powered by energy harvested from the ambient environment. We assume that the energy harvesting processes at the transmitters can be modeled as independent Bernoulli processes with parameters λis, and the channel states between the transmitters and the receiver are independent Bernoulli processes with parameter µis. An active transmitter always transmits with a fixed power and consumes one unit amount of energy in a time slot. Under the assumption that µi≥ λi, ∀i, our objective is to schedule the transmissions adaptively according to the instantaneous channel and battery states of transmitters, so that the long-term average sum-throughput of the MAC is maximized in expectation. We first show that for a general asymmetric scenario where λis and µis are not identical across the transmitters, the expected long-term average sum-throughput has an upper bound for any transmission scheduling policy satisfying the energy causality constraints. We then consider a special symmetric scenario where λis and µis are uniform among transmitters. We propose a randomized longest-connected-queue transmission scheduling policy and show that it achieves the upper bound almost surely as time T approaches infinity, thus it is optimal.

In this paper, we consider a collaborative sensing scenario where sensing nodes are powered by energy harvested from the ambient environment. In each time slot, an active sensor consumes one unit amount of energy to take an observation and transmit it back to a fusion center (FC). After receiving observations from all of the active sensors in a time slot, the FC aims to extract information from them. We assume that the sensing utility generated by the observations is a concave function of the number of the active sensing nodes in that slot. Our objective is to develop a sensing scheduling policy so that the time average utility generated by the sensors is maximized. We first consider an offline setting, where the energy harvesting profile over duration$[0,T-1]$for each sensor is known beforehand. Assuming infinite battery capacity at sensors, we show that the optimal scheduling structure has a “majorization” property, and propose a procedure to construct a collaborative sensing policy with the identified structure explicitly. We then consider an online setting, under which the energy harvesting profile is available causally. Assuming the energy harvesting processes at individual sensors are independent but not necessarily identical Bernoulli processes, we show that the expected long-term time average sensing utility has an upper bound under any feasible scheduling policy satisfying the energy causality constraints. We then propose a randomized myopic policy, which aims to select a number of sensors with the highest energy levels to perform the sensing task in each slot. We show that the time average utility generated under the proposed policy converges to the upper bound almost surely as time$T$approaches infinity, thus it is optimal. The corresponding convergence rate is also explicitly characterized.

In this paper, we seek answer to the question: can a wireless sensing system with energy harvesting power supplies perform as well as one with conventional power supplies? Due to the stochastic nature of the energy harvested from the ambient environment, uniform sampling employed by conventional sensing systems is usually infeasible for energy harvesting sensing systems. We propose a simple best-effort sensing scheme, which defines a set of equally-spaced candidate sensing instants. At a given candidate sensing instant, the sensor will perform sensing if there is sufficient energy available, and it will remain silent otherwise. It is analytically shown that the percentage of silent candidate sensing instants diminishes as time increases, if and only if the average energy harvesting rate is no less than the average energy consumption rate. The theoretical results are then used to guide the design of a practical sensing system that monitors a time-varying event. Both analysis and simulations show that the energy harvesting system with the best-effort sensing scheme can asymptotically achieve the same mean squared error (MSE) performance as one with uniform sensing and deterministic energy sources. Therefore, we provide a positive answer to the question from both theoretical and practical aspects.

In this paper, we study the optimal sampling policy for an energy harvesting sensing system, which is designed to estimate a wide-sense stationary random process by using discrete-time samples collected by a sensor. The energy in the sensor is consumed by taking observations and is replenished randomly with energy harvested from the ambient environment. Our goal is to identify the optimal sampling policy that minimizes the estimation mean squared error (MSE) under stochastic energy constraints. The problem can be formulated as a stochastic programming problem, which is generally difficult to solve. We identify an asymptotically optimal solution to the problem by exploiting the properties of random processes with power-law decaying covariance. Specifically, with the help of a newly derived inverse covariance matrix of the random process, it is discovered that the linear minimum MSE (MMSE) estimation of the random process demonstrates a Markovian property. That is, the optimal estimation of any point in a time segment bounded by two consecutive samples can be achieved by using the knowledge of only the two bounding samples while ignoring all other samples. Such a Markovian property enables us to identify a lower bound of the long term average MSE. Motivated by the structure of the MSE lower bound, we then propose a simple best-effort sampling scheme by considering the stochastic energy constraints. It is shown that the best-effort sampling scheme is asymptotically optimal in the sense that, for almost every energy harvesting sample path, it achieves the MSE lower bound as time becomes large.

In this paper, we consider a collaborative sensing scenario where sensing nodes are powered by energy harvested from the environment. In each time slot, an active sensor consumes one unit amount of energy to take an observation and transmit it back to a fusion center (FC). After receiving observations from all of the active sensors in a time slot, the FC aims to extract information from them. We assume that the utility generated by the observations is a function of the number of the active sensing nodes in that slot. Assuming the energy harvesting processes at individual sensors are independent Bernoulli processes, our objective is to develop a sensing scheduling policy so that the expected long-term average utility generated by the sensors is maximized. Under the concavity assumption of the utility function, we first show that the expected time average utility has an upper bound for any feasible scheduling policy satisfying the energy causality constraint. We then propose a myopic policy, which aims to select a fixed number of sensors with the highest energy levels to perform the sensing task in each slot. The myopic policy essentially balances the current energy queue lengths in every time slot. We show that the time average utility generated under the myopic policy converges to the upper bound almost surely as time T approaches infinity, thus the myopic policy is optimal. The corresponding convergence rate is also explicitly characterized.

This paper characterizes an achievable channel coding rate for a noiseless binary communication channel with an energy harvesting (EH) transmitter at a given blocklength n and error probability ε. As energy arrives randomly at the transmitter, codewords must obey the cumulative stochastic energy constraints. The coupling of the energy constraints on the symbols in a codeword makes the analysis fundamentally different from that of discrete memoryless channels. We first adopt a random coding scheme to construct the codebook with statistical information of the EH process. We then analyze the statistics of the corresponding output sequence. Specifically, we prove that the average number of mismatches between the input codeword and the output sequence scales as O(√n). Based on such characterization, we then propose a decoding scheme, and analyze the corresponding probability of decoding error. Finally, we explicitly characterize the maximum size of the length-n codebook generated by the random coding scheme in order to achieve the average probability of error ε. This leads to a lower bound on the maximum achievable channel coding rate for the EH communication channel. We show that the gap between the lower bound and the corresponding channel capacity under an equivalent average power constraint scales in O(llog n/√n), where l is a constant depending on the error probability ?, and the statistics of the energy harvesting process.

In this paper, we investigate a two-way communication channel where users can harvest energy from nature and energy can be transferred in one-way from one of the users to the other. Energy required for data transmission is randomly harvested by the users throughout the communication duration and users have unlimited batteries to store energy for future use. In addition, there is a separate wireless energy transfer unit that facilitates energy transfer only in one-way and with efficiency α. We study the energy cooperation made possible by wireless energy transfer in the two-way channel. Assuming that both users know the energy arrivals in advance, we find jointly optimal offline energy management policies that maximize the sum throughput of the users. We show that this problem is a convex optimization problem, and find the solution by a generalized two-dimensional directional water-filling algorithm which transfers energy from one user to another while maintaining that the energy is allocated in the time dimension optimally. Optimal solution equalizes the energy levels as much as possible both among users and among slots, permitted by causality constraints of the energy arrivals and one-way energy transfer.

In energy harvesting communications, users transmit messages using energy harvested from nature during the course of communication. With an optimum transmit policy, the performance of the system depends only on the energy arrival profiles. In this paper, we introduce the concept of energy cooperation, where a user wirelessly transmits a portion of its energy to another energy harvesting user. This enables shaping and optimization of the energy arrivals at the energy-receiving node, and improves the overall system performance, despite the loss incurred in energy transfer. We consider several basic multi-user network structures with energy harvesting and wireless energy transfer capabilities: relay channel, two-way channel and multiple access channel. We determine energy management policies that maximize the system throughput within a given duration using a Lagrangian formulation and the resulting KKT optimality conditions. We develop a two-dimensional directional water-filling algorithm which optimally controls the flow of harvested energy in two dimensions: in time (from past to future) and among users (from energy-transferring to energy-receiving) and show that a generalized version of this algorithm achieves the boundary of the capacity region of the two-way channel.

In this paper, we aim to develop scheduling policies to maximize the stability region of a wireless network under the assumption that mutual information accumulation is implemented at the physical layer. This enhanced physical layer capability enables the system to accumulate information even when the link between two nodes is not good and a packet cannot be decoded within a slot. The result is an expansion of the stability region of the system. The accumulation process does not satisfy the i.i.d assumption that underlies many previous analysis in this area. Therefore it also brings new challenges to the problem. We propose two dynamic scheduling algorithms to overcome this difficulty. One performs scheduling every T slot, which inevitably increases average delay in the system, but approaches the boundary of the stability region. The second constructs a virtual system with the same stability region. Through controlling the virtual queues in the constructed system, we avoid the non-i.i.d difficulty and attain the stability region. We derive performance bounds under both algorithms and compare them through simulation results.

We consider a simple multi-hop communication scenario composed of a source node, a relay node and a destination node where the source and the relay can harvest energy from the nature. Energy required for communication arrives (is harvested) at the transmitter and an unlimited battery stores it before being consumed for transmission. In addition, the source can assist the relay by transferring a portion of its energy to the relay through a separate energy transfer unit. We address this energy cooperation between the source and the relay in a deterministic setting. Assuming that the source and the relay nodes are informed of the energy arrivals in advance, we find jointly optimal offline energy management policies for the source and the relay that maximize the end-to-end throughput. We show that this problem is a convex problem. In order to gain insight about the structure of the solution, we consider specific scenarios. In particular, we show that if the relay energy profile is higher at the beginning and lower at the end with only one intersection, then matching the power sequences of the source and the relay slot-by-slot is optimal. We also consider the case when the energy of the source is available at the beginning and show that transferring energy in the first slot is optimal.

A key challenge in wireless networking is the management of interference between transmissions. Identifying which transmitters interfere with each other is crucial. Complicating this task is the fact that the topology of wireless networks can change from time to time, and so the identification process may need to be carried out on a regular basis. Injecting active probing traffic to assess interference can lead to unacceptable overhead, and so this paper focuses on interference estimation based on passive traffic monitoring in networks that use the CSMA/CA (Carrier Sense Multiple Access/Collision Avoidance) protocol. A graph is used to represent the interference in the network, where the nodes represent transmitters and edges represent interference between pairs of transmitters. We investigate the problem of learning the graph structure based on passive observations of network traffic transmission patterns and information about successes or failures in transmissions. Previous work has focused on algorithms and validations in small testbed networks. This paper focuses on the scaling behavior of such methods which is unaddressed in prior work. In particular we establish bounds on the minimum observation period required to identify the interference graph reliably. The main results are expressed in terms of the total number of nodes n and the maximum number of interfering transmitters per node (i.e., maximum node degree) d. The effects of hidden terminal interference (i.e., interference not detectable via carrier sensing) on the observation time requirement are also quantified. We show that it is necessary and sufficient that the observation period grows like d2log n, and we propose a practical algorithm that reliably identifies the graph from this length of observation. We conclude that the observation requirements scale quite mildly with network size, and that the networks with sparse interference patterns can be more rapidly identified than those with dense interference patterns.

We consider the optimal packet scheduling problem in a single-user energy harvesting wireless communication system. In this system, both the data packets and the harvested energy are modeled to arrive at the source node randomly. Our goal is to adaptively change the transmission rate according to the traffic load and available energy, such that the time by which all packets are delivered is minimized. Under a deterministic system setting, we assume that the energy harvesting times and harvested energy amounts are known before the transmission starts. For the data traffic arrivals, we consider two different scenarios. In the first scenario, we assume that all bits have arrived and are ready at the transmitter before the transmission starts. In the second scenario, we consider the case where packets arrive during the transmissions, with known arrival times and sizes. We develop optimal off-line scheduling policies which minimize the time by which all packets are delivered to the destination, under causality constraints on both data and energy arrivals.

We consider the minimization of the transmission completion time with a battery limited energy harvesting transmitter in an M-user AWGN broadcast channel where the transmitter is able to harvest energy from the nature, using a finite storage capacity rechargeable battery. The harvested energy is modeled to arrive (be harvested) at the transmitter during the course of transmissions at arbitrary time instants. The transmitter has fixed number of packets for each receiver. Due to the finite battery capacity, energy may overflow without being utilized for data transmission. We derive the optimal offline transmission policy that minimizes the time by which all of the data packets are delivered to their respective destinations. We analyze the structural properties of the optimal transmission policy using a dual problem. We find the optimal total transmit power sequence by a directional water-filling algorithm. We prove that there exist M-1 cut-off power levels such that user i is allocated the power between the i-1st and the ith cut-off power levels subject to the availability of the allocated total power level. Based on these properties, we propose an algorithm that gives the globally optimal offline policy. The proposed algorithm uses directional water-filling repetitively. Finally, we illustrate the optimal policy and compare its performance with several suboptimal policies under different settings.

In this paper, we investigate the transmission completion time minimization problem in an additive white Gaussian noise (AWGN) broadcast channel, where the transmitter is able to harvest energy from the nature, using a rechargeable battery. The harvested energy is modeled to arrive at the transmitter during the course of transmissions. The transmitter has a fixed number of packets to be delivered to each receiver. The objective is to minimize the time by which all of the packets are delivered to their respective destinations. To this end, we optimize the transmit powers and transmission rates in a deterministic setting. We first analyze the structural properties of the optimal transmission policy in a two-user broadcast channel via the dual problem of maximizing the departure region by a fixed time T. We prove that the optimal total transmit power sequence has the same structure as the optimal single-user transmit power sequence in . In addition, the total power is split optimally based on a cut-off power level; if the total transmit power is lower than this cut-off level, all transmit power is allocated to the stronger user; otherwise, all transmit power above this level is allocated to the weaker user. We then extend our analysis to an M-user broadcast channel. We show that the optimal total power sequence has the same structure as the two-user case and optimally splitting the total power among M users involves M-1 cut-off power levels. Using this structure, we propose an algorithm that finds the globally optimal policy. Our algorithm is based on reducing the broadcast channel problem to a single-user problem as much as possible. Finally, we illustrate the optimal policy and compare its performance with several suboptimal policies under different settings.

In this paper, we investigate the transmission completion time minimization problem in a two-user additive white Gaussian noise (AWGN) broadcast channel, where the transmitter is able to harvest energy from the nature. The harvested energy is modeled to arrive at the transmitters randomly. In this paper, under a deterministic system setting, we assume that the energy harvesting times and harvested energy amounts are known before the transmission starts. The transmitter has a fixed number of packets to be delivered to each receiver. Our goal is to minimize the time by which all of the packets for both users are delivered to their respective destinations. To this end, we optimize the transmit powers and transmission rates intended for both users. We first analyze the structural properties of the optimal transmission policy. We prove that the optimal total transmit power has the same structure as the optimal single-user transmit power. We also prove that there exists a cut-off power level for the stronger user. If the optimal total transmit power is lower than this level, all transmit power is allocated to the stronger user, and when the optimal total transmit power is larger than this level, all transmit power above this level is allocated to the weaker user. Based on these structural properties of the optimal policy, we propose an algorithm that yields the globally optimal off-line scheduling policy.

In this paper, we investigate the optimal packet scheduling problem in a two-user multiple access communication system, where the transmitters are able to harvest energy from the nature. Under a deterministic system setting, we assume that the energy harvesting times and harvested energy amounts are known before the transmission starts. For the packet arrivals, we assume that packets have already arrived and are ready to be transmitted at the transmitter before the transmission starts. Our goal is to minimize the time by which all packets from both users are delivered to the destination through controlling the transmission powers and transmission rates of both users. We first develop a generalized iterative backward waterfilling algorithm to characterize the maximum departure region of the transmitters for any given deadline $T$. Then, based on the departure region at energy arrival epochs, we decompose the transmission completion time minimization problem into a convex optimization problem and solve it efficiently.

Wireless systems comprised of rechargeable nodes have a significantly prolonged lifetime and are sustainable. A distinct characteristic of these systems is the fact that the nodes can harvest energy throughout the duration in which communication takes place. As such, transmission policies of the nodes need to adapt to these harvested energy arrivals. In this paper, we consider optimization of the transmission policy of an energy harvesting transmitter which has a limited battery capacity, communicating in a wireless fading channel. In particular, we identify the optimal offline transmission policies that maximize the number of bits delivered by a deadline, and minimize the transmission completion time of the communication session. We introduce a directional water-filling algorithm which provides a simple and concise interpretation of the necessary optimality conditions as well as energy storage capacity and causality. We solve the throughput maximization problem for the fading channel using the directional water-filling algorithm, which simultaneously adapts to the energy harvested as well as the channel variations in time. We then solve the transmission completion time minimization problem by utilizing its equivalence to its throughput maximization counterpart.

We consider the minimization of the transmission completion time with a battery limited energy harvesting transmitter in a two-user AWGN broadcast channel. The transmitter has fixed number of packets for each receiver and energy is modeled to arrive (be harvested) at the transmitter at random instants. The battery at the transmitter has a finite storage capacity, hence energy may overflow without being utilized for data transmission. We derive the optimal offline transmission policy that minimizes the time by which all of the data packets are delivered to their respective destinations. We analyze the structural properties of the optimal transmission policy using a dual problem. We find the optimal total transmit power sequence by a directional water-filling algorithm. We prove that there exists a cut-off power level such that if the allocated power is lower than this level, then only the stronger user is served in that epoch; otherwise, the power above this level is allocated to the weaker user. Based on these properties, we propose an algorithm that gives the globally optimal offline policy. The proposed algorithm uses directional water-filling repetitively.

Wireless systems comprised of rechargeable nodes have a significantly prolonged lifetime and are sustainable. A distinct characteristic of these systems is the fact that the nodes can harvest energy throughout the duration in which communication takes place. As such, transmission policies of the nodes need to adapt to these harvested energy arrivals. In this paper, we consider optimization of point-to-point data transmission with an energy harvesting transmitter which has a limited battery capacity, communicating in a wireless fading channel. We consider two objectives: maximizing the throughput by a deadline, and minimizing the transmission completion time of the communication session. We optimize these objectives by controlling the time sequence of transmit powers subject to energy storage capacity and causality constraints. We, first, study optimal offline policies. We introduce a directional water-filling algorithm which provides a simple and concise interpretation of the necessary optimality conditions. We show the optimality of an adaptive directional water-filling algorithm for the throughput maximization problem. We solve the transmission completion time minimization problem by utilizing its equivalence to its throughput maximization counterpart. Next, we consider online policies. We use stochastic dynamic programming to solve for the optimal online policy that maximizes the average number of bits delivered by a deadline under stochastic fading and energy arrival processes with causal channel state feedback. We also propose near-optimal policies with reduced complexity, and numerically study their performances along with the performances of the offline and online optimal policies under various different configurations.

We consider a communication channel with two transmitters and one receiver, with an underlying rate region which is approximated as a general pentagon. Different from the Gaussian multiple access channel (MAC) capacity region, the sum-rate on the dominant face of this pentagon is not a constant. We allocate rates from this rate region to users according to their current queue lengths in order to minimize the average delay in the system. We formulate the problem as a Markov decision problem (MDP), and derive the structural properties of the corresponding discounted-cost MDP. We show that the delay-optimal policy has a switch curve structure. For the discounted-cost problem, we prove that the switch curve has a limit along one of the dimensions. The delay-optimal policy divides the entire queue state space into two via a switch curve. If the queue state is on one side of the switch curve, the system operates at one of the corner points of the rate pentagon which favors maximum sum-rate. When the queue state switches to the other side of the switch curve, the system operates at the other corner point of the rate pentagon which favors balancing the queue lengths. As a result, the system does not always operate at the sum-rate maximizing rate pair, but trades rate for balanced queue lengths for the goal of minimizing the overall delay. The existence of a limit in the switch curve along one of dimensions implies that, once the queue state is beyond the limit, the system always operates at one of the corner points, implying that the queues can be operated partially distributedly.

We consider a communication channel with two transmitters and one receiver, with an underlying rate region which is approximated as a general pentagon. Different from the Gaussian multiple access channel (MAC) capacity region, the sum-rate on the dominant face of this pentagon is not a constant. We allocate rates from this rate region to users according to their current queue lengths in order to minimize the average delay in the system. We formulate the problem as a Markov decision problem (MDP), and derive the structural properties of the corresponding discounted-cost MDP. We show that the delay-optimal policy has a switch curve structure. For the discounted-cost problem, we prove that the switch curve has a limit along one of the dimensions.

We investigate the problem of minimizing the overall transmission delay of packets in a multi-access wireless communication system, where the transmitters have average power constraints. We use a multi-dimensional Markov chain to model the medium access control layer behavior. The state of the Markov chain represents current queue lengths. Our goal is to minimize the average packet delay through controlling the probability of departure at each state, while satisfying the average power constraint for each queue. We consider a general asymmetric system, where the arrival rates to the queues, channel gains and average power constraints of the two users are arbitrary. We formulate the problem as a constrained optimization problem, and then transform it to a linear programming problem. We analyze the linear programming problem, and develop a procedure by which we determine the optimal solution analytically. We show that the optimal policy has a threshold structure: when the sum of the queue lengths is larger than a threshold, both users should transmit a packet during the current slot; when the sum of the queue lengths is smaller than a threshold, only one of the users, the one with the longer queue, should transmit a packet during the current slot. We provide numerical examples for both symmetric and asymmetric settings.

We investigate a delay minimization problem in a multiple access wireless communication system. We consider a discrete-time non-fading additive white Gaussian noise (AWGN) multiple access channel. In each slot, bits arrive at the transmitters randomly according to some distribution, which is i.i.d. from user to user and from slot to slot. Each transmitter has an average power constraint of P. Our goal is to allocate rates to users, from the multiple access capacity region, based on their current queue lengths, in order to minimize the average delay of the system. We formulate the problem as a Markov decision problem (MDP) with an average cost criterion. We first show that the value function is increasing, symmetric and convex in the queue length vector. Taking advantage of these properties, we show that the optimal rate allocation policy is one which tries to equalize the queue lengths as much as possible in each slot, while working on the dominant face of the capacity region.

We investigate the problem of minimizing the overall transmission delay of data packets in a single-user wireless communication system, where the transmitter has a fixed amount of energy to transmit all of the data packets. We consider two different scenarios. In the first scenario, we assume that packets arrive randomly at the transmitter. We propose two different approaches to solve this problem. First, we develop an iterative algorithm that allocates the total energy of the transmitter to its individual packets, in a way to minimize the total delay. As a second approach, we develop a dynamic programming formulation for the problem. In the second scenario, we assume that all of the packets have already arrived before the transmission starts. In this situation, the cost function has a fixed form, and is convex and differentiable. In this scenario, the iterative algorithm we develop is guaranteed to converge to the unique global optimal solution.