We theoretically justify the recent empirical finding of Teh et al. (2025) that a transformer pretrained on synthetically generated data achieves strong performance on empirical Bayes (EB) problems. We take an indirect approach to this question: rather than analyzing the model architecture or training dynamics, we ask why a pretrained Bayes estimator, trained under a prespecified training distribution, can adapt to arbitrary test distributions. Focusing on Poisson EB problems, we identify the existence of universal priors such that training under these priors yields a near-optimal regret bound of $\widetilde{O}(\frac{1}{n})$ uniformly over all test distributions. Our analysis leverages the classical phenomenon of posterior contraction in Bayesian statistics, showing that the pretrained Bayes estimator adapts to unknown test distributions precisely through posterior contraction. This perspective also explains the phenomenon of length generalization, in which the test sequence length exceeds the training length, as the model performs Bayesian inference using a fractional posterior.

Fast computation of a matrix product $W^\top X$ is a workhorse of modern LLMs. To make their deployment more efficient, a popular approach is that of using a low-precision approximation $\widehat W$ in place of true $W$ (“weight-only quantization”). Information theory demonstrates that an optimal algorithm for reducing precision of $W$ depends on the (second order) statistics of $X$ and requires a careful alignment of vector quantization codebook with PCA directions of $X$ (a process known as “waterfilling allocation”). Dependence of the codebook on statistics of $X$, however, is highly impractical. This paper proves that there exist a universal codebook that is simultaneously near-optimal for all possible statistics of $X$, in the sense of being at least as good as an $X$-adapted waterfilling codebook with rate reduced by 0.11 bit per dimension in the case when $W$ is Gaussian. Such universal codebook would be an ideal candidate for the low-precision storage format, a topic of active modern research, but alas the existence proof is non-constructive. Equivalently, our result shows existence of a net in $\mathbb{R}^n$ that is a nearly-optimal covering of a sphere simultaneously with respect to all Hilbert norms.

Recent work in machine learning community proposed multiple methods for performing lossy compression (quantization) of large matrices. This quantization is important for accelerating matrix multiplication (main component of large language models), which is often bottlenecked by the speed of loading these matrices from memory. Unlike classical vector quantization and rate-distortion theory, the goal of these new compression algorithms is to be able to approximate not the matrices themselves, but their matrix product. Specifically, given a pair of real matricesA,Ban encoder (compressor) is applied to each of them independently producing descriptions withRbits per entry. These representations subsequently are used by the decoder to estimate matrix productA⊤B. In this work, we provide a non-asymptotic lower bound on the mean squared error of this approximation (as a function of rateR) for the case of matricesA,Bwith iid Gaussian entries. Algorithmically, we construct a universal quantizer based on nested lattices with an explicit guarantee of approximation error for any (non-random) pair of matricesA,Bin terms of only Frobenius norms ∥Ā∥F, ∥B∥Fand ∥Ā⊤B∥F, where Ā,Bare versions ofA,Bwith zero-centered columns, respectively. For iid Gaussian matrices our quantizer achieves the lower bound and is, thus, asymptotically optimal. A practical low-complexity version of our quantizer achieves performance quite close to optimal. In addition, we derive rate-distortion function for matrix multiplication of iid Gaussian matrices, which exhibits an interesting phase-transition atR≈ 0.906 bit/entry, showing necessity of Johnson-Lindestrauss dimensionality reduction (sketching) in the low-rate regime.

Consider a $d$-uniform random hypergraph on $n$ vertices in which hyperedges are included iid so that the average degree is $n^\delta$. The projection of a hypergraph is a graph on the same $n$ vertices where an edge connects two vertices if and only if they belong to some hyperedge. The goal is to reconstruct the hypergraph given its projection. An earlier work of Bresler, Guo, and Polyanskiy (COLT 2024) showed that exact recovery for $d=3$ is possible if and only if $\delta < 2/5$. This work completely resolves the question for all values of $d$ for both exact and partial recovery and for both cases of whether multiplicity information about each edge is available or not. In addition, we show that the reconstruction fidelity undergoes an all-or-nothing transition at a threshold. In particular, this resolves all conjectures from Bresler, Guo, and Polyanskiy (COLT 2024).

We study the problem of sequential probability assignment under logarithmic loss, both with and without side information. Our objective is to analyze the \emph{minimax regret}—a notion extensively studied in the literature—in terms of geometric quantities, such as covering numbers and scale-sensitive dimensions. We show that the minimax regret for the case of no side information (equivalently, the Shtarkov sum) can be upper bounded in terms of \emph{sequential square-root entropy}, a notion closely related to Hellinger distance. For the problem of sequential probability assignment with side information, we develop both upper and lower bounds based on the aforementioned entropy. The lower bound matches the upper bound, up to log factors, for classes in the Donsker regime (according to our definition of entropy).

Post-training quantization (PTQ) has emerged as a critical technique for efficient deployment of large language models (LLMs). This work proposes NestQuant, a novel PTQ scheme for weights and activations that is based on self-similar nested lattices. Recent works have mathematically shown such quantizers to be information-theoretically optimal for low-precision matrix multiplication. We implement a practical low-complexity version of NestQuant based on Gosset lattice, making it a drop-in quantizer for any matrix multiplication step (e.g., in self-attention, MLP etc). For example, NestQuant quantizes weights, KV-cache, and activations of Llama-3-8B to 4 bits, achieving perplexity of 6.6 on wikitext2. This represents more than 55% reduction in perplexity gap with respect to unquantized model (perplexity of 6.14) compared to state-of-the-art Meta’s SpinQuant (perplexity 7.3), OstQuant (7.3) and QuaRot (8.2). Comparisons on bigger models (up to 70B) and on various LLM evaluation benchmarks confirm uniform superiority of NestQuant.

Recent work in machine learning community proposed multiple methods for performing lossy compression (quantization) of large matrices. This quantization is important for accelerating matrix multiplication (main component of large language models), which is often bottlenecked by the speed of loading these matrices from memory. Unlike classical vector quantization and rate-distortion theory, the goal of these new compression algorithms is to be able to approximate not the matrices themselves, but their matrix product. Specifically, given a pair of real matrices$A, B$an encoder (compressor) is applied to each of them independently producing descriptions with$R$bits per entry. These representations subsequently are used by the decoder to estimate matrix product$A^{\top} B$. In this work, we provide a non-asymptotic lower bound on the mean squared error of this approximation (as a function of rate$R$) for the case of matrices$A, B$with iid Gaussian entries. Algorithmically, we construct a universal quantizer based on nested lattices with an explicit guarantee of approximation error for any (non-random) pair of matrices$A$,$B$in terms of only Frobenius norms$\vert\bar{A}\vert_{F},\vert\bar{B}\vert_{F}$and$\left\vert\bar{A}^{\top} \bar{B}\right\vert_{F}$, where$\bar{A}, \bar{B}$are versions of$A, B$with zerocentered columns, respectively. For iid Gaussian matrices our quantizer achieves the lower bound and is, thus, asymptotically optimal. In particular, we derive the rate-distortion function for matrix multiplication of iid Gaussian matrices, which exhibits an interesting phase-transition at$R \approx 0.906$bit/entry. An extended version of this paper is available in [1].

The meta-task of obtaining and aligning representations through contrastive pretraining is steadily gaining importance since its introduction in CLIP and ALIGN. In this paper we theoretically explain the advantages of synchronizing with trainable inverse temperature and bias under the sigmoid loss, as implemented in the recent SigLIP and SigLIP2 models of Google DeepMind. Temperature and bias can drive the loss function to zero for a rich class of configurations that we call $(\mathsf{m}, \mathsf{br})$ -Constellations. $(\mathsf{m}, \mathsf{br})$ -Constellations are a novel combinatorial object related to spherical codes and are parametrized by a margin $\mathsf{m}$ and relative bias $\mathsf{br}$. We use our characterization of constellations to theoretically justify the success of SigLIP on retrieval, to explain the modality gap present in SigLIP, and to identify the necessary dimension for producing high-quality representations. Finally, we propose a reparameterization of the sigmoid loss with explicit relative bias, which improves training dynamics in experiments with synthetic data.

We study the effect of normalization schemes on token representations in deep transformers. Modeling their evolution as interacting particles on the sphere, we show that normalization acts as a form of speed regulation. This perspective enables a unified analysis of several schemes---including **Post-LN**, **Pre-LN**, **Mix-LN**, **Peri-LN**, **nGPT**---revealing how they influence clustering dynamics and representation collapse. Our framework clarifies how different schemes shape token representations across layers and provides a principled basis for comparing them, identifying **Peri-LN** as a particularly effective choice.

We propose a new density estimation algorithm. Given $n$ i.i.d. observations from a distribution belonging to a class of densities on $\mathbb{R}^d$, our estimator outputs any density in the class whose “perceptron discrepancy” with the empirical distribution is at most $O(\sqrt{d/n})$. The perceptron discrepancy is defined as the largest difference in mass two distribution place on any halfspace. It is shown that this estimator achieves the expected total variation distance to the truth that is almost minimax optimal over the class of densities with bounded Sobolev norm and Gaussian mixtures. This suggests that the regularity of the prior distribution could be an explanation for the efficiency of the ubiquitous step in machine learning that replaces optimization over large function spaces with simpler parametric classes (such as discriminators of GANs). We also show that replacing the perceptron discrepancy with the generalized energy distance of Székely and Rizzo (2013) further improves total variation loss. The generalized energy distance between empirical distributions is easily computable and differentiable, which makes it especially useful for fitting generative models. To the best of our knowledge, it is the first “simple” distance with such properties that yields minimax optimal statistical guarantees. In addition, we shed light on the ubiquitous method of representing discrete data in domain $[k]$ via embedding vectors on a unit ball in $\mathbb{R}^d$. We show that taking $d \asymp \log(k)$ allows one to use simple linear probing to evaluate and estimate total variation distance, as well as recovering minimax optimal sample complexity for the class of discrete distributions on $[k]$.

The graph projection of a hypergraph is a simple graph with the same vertex set and with an edge between each pair of vertices that appear in a hyperedge. We consider the problem of reconstructing a random $d$-uniform hypergraph from its projection. Feasibility of this task depends on $d$ and the density of hyperedges in the random hypergraph. For $d=3$ we precisely determine the threshold, while for $d\ge 4$ we give bounds. All of our feasibility results are obtained by exhibiting an efficient algorithm for reconstructing the original hypergraph, while infeasibility is information-theoretic. Our results also apply to mildly inhomogeneous random hypergrahps, including hypergraph stochastic block models (HSBM). A consequence of our results is an optimal HSBM recovery algorithm, improving on Gaudio and Joshi (2023a).

This work presents a modification of the self-attention dynamics proposed in Geshkovski et al to better reflect the practically relevant, causally masked attention used in transformer architectures for generative AI. This modification translates into an interacting particle system that cannot be interpreted as a mean-field gradient flow. Despite this loss of structure, we significantly strengthen the results of Geshkovski et al in this context: While previous rigorous results focused on cases where all three matrices (key, query, and value) were scaled identities, we prove asymptotic convergence to a single cluster for arbitrary key-query matrices and value matrix equal to the identity. Additionally, we establish a connection to the classical R\'enyi parking problem from combinatorial geometry to make initial theoretical steps towards demonstrating the existence of meta-stable states.

This article is a tutorial introduction to the field of unsourced multiple access (UMAC) protocols. We first provide a historical survey of the evolution of random access protocols, focusing specifically on the case in which uncoordinated users share a wireless broadcasting medium. Next, we highlight the change of perspective originated by the UMAC model, in which the physical and medium access layer’s protocols cooperate, thus reframing random access as a novel coding-theoretic problem. By now, a large variety of UMAC protocols (codes) emerged, necessitating a certain classification that we indeed propose here. Although some random access schemes require a radical change of the physical layer, others can be implemented with minimal changes to existing industry standards. As an example, we discuss a simple modification to the 5G New Radio (5GNR) Release 16 random access channel that builds on the UMAC theory and that dramatically improves energy efficiency for systems with even moderate number of simultaneous users (e.g., 5–10-dB gain for 10–50 users) and also enables handling of high number of users, something completely out of reach of the state of the art.

In the study of Ising models on large locally tree-like graphs, in both rigorous and non-rigorous methods one is often led to understanding the so-called belief propagation distributional recursions and its fixed points. We prove that there is at most one non-trivial fixed point for Ising models with zero or certain random external fields. Previously this was only known for sufficiently “low-temperature” models. Our main innovation is in applying information-theoretic ideas of channel comparison leading to a new metric (degradation index) between binary-input-symmetric (BMS) channels under which the Belief Propagation (BP) operator is a strict contraction (albeit non-multiplicative). A key ingredient of our proof is a strengthening of the classical stringy tree lemma of Evans-Kenyon-Peres-Schulman (2000). Our result simultaneously closes the following 6 conjectures in the literature: 1) independence of robust reconstruction accuracy to leaf noise in broadcasting on trees; 2) uselessness of global information for a labeled 2-community stochastic block model, or 2-SBM; 3) optimality of local algorithms for 2-SBM under noisy side information; 4) uniqueness of BP fixed point in broadcasting on trees in the Gaussian (large degree) limit; 5) boundary irrelevance in broadcasting on trees; 6) characterization of entropy (and mutual information) of community labels given the graph in 2-SBM.

Consider the problem of binary hypothesis testing. Given Z coming from either$\mathbb {P}^{\otimes m}$or$\mathbb {Q}^{\otimes m}$, to decide between the two with small probability of error it is sufficient, and in many cases necessary, to have$m\asymp 1/\varepsilon ^{2}$, where$\varepsilon $measures the separation between$\mathbb {P}$and$\mathbb {Q}$in total variation ($\textsf {TV}$). Achieving this, however, requires complete knowledge of the distributions and can be done, for example, using the Neyman-Pearson test. In this paper we consider a variation of the problem which we call likelihood-free hypothesis testing, where access to$\mathbb {P}$and$\mathbb {Q}$is given through n i.i.d. observations from each. In the case when$\mathbb {P}$and$\mathbb {Q}$are assumed to belong to a non-parametric family, we demonstrate the existence of a fundamental trade-off between n and m given by$nm\asymp n_{\textsf {GoF}}^{2}(\varepsilon)$, where$n_{\textsf {GoF}}(\varepsilon)$is the minimax sample complexity of testing between the hypotheses$H_{0}:\, \mathbb {P}=\mathbb {Q}$vs$H_{1}:\, \textsf {TV}(\mathbb {P},\mathbb {Q})\geq \varepsilon $. We show this for three families of distributions, in addition to the family of all discrete distributions for which we obtain a more complicated trade-off exhibiting an additional phase-transition. Our results demonstrate the possibility of testing without fully estimating$\mathbb {P}$and$\mathbb {Q}$, provided$m \gg 1/\varepsilon ^{2}$.

Suppose we are given access to $n$ independent samples from distribution $\mu$ and we wish to output one of them with the goal of making the outputdistributed as close as possible to a target distribution $\nu$. In this workwe show that the optimal total variation distance as a function of $n$ is givenby $\tilde\Theta(\frac{D}{f’(n)})$ over the class of all pairs $\nu,\mu$ with a bounded $f$-divergence $D_f(\nu\|\mu)\leq D$. Previously, this question was studied only for the case when the Radon-Nikodym derivative of $\nu$ with respect to $\mu$ is uniformly bounded. We then consider an application in theseemingly very different field of smoothed online learning, where we show that recent results on the minimax regret and the regret of oracle-efficient algorithmsstill hold even under relaxed constraints on the adversary (to have bounded $f$-divergence, as opposed to bounded Radon-Nikodym derivative). Finally, we also study efficacy of importance sampling for mean estimates uniformover a function class and compare importance sampling with rejectionsampling.

This paper considers an ML inspired approach to hypothesis testing known as classifier/classification-accuracy testing (CAT). In CAT, one first trains a classifier by feeding it labeled synthetic samples generated by the null and alternative distributions, which is then used to predict labels of the actual data samples. This method is widely used in practice when the null and alternative are only specified via simulators (as in many scientific experiments). We study goodness-of-fit, two-sample (TS) and likelihood-free hypothesis testing (LFHT), and show that CAT achieves (near-)minimax optimal sample complexity in both the dependence on the total-variation (TV) separation ε and the probability of error δ in a variety of non-parametric settings, including discrete distributions, d-dimensional distributions with a smooth density, and the Gaussian sequence model. In particular, we close the high probability sample complexity of LFHT for each class. As another highlight, we recover the minimax optimal complexity of TS over discrete distributions, which was recently established by Diakonikolas et al. (2021). The corresponding CAT simply compares empirical frequencies in the first half of the data, and rejects the null when the classification accuracy on the second half is better than random.

In the study of sparse stochastic block models (SBMs) one often needs to analyze a distributional recursion, known as the belief propagation (BP) recursion. Uniqueness of the fixed point of this recursion implies several results about the SBM, including optimal recovery algorithms for SBM (Mossel et al. (2016)) and SBM with side information (Mossel and Xu (2016)), and a formula for SBM mutual information (Abbe et al. (2021)). The 2-community case corresponds to an Ising model, for which Yu and Polyanskiy (2022) established uniqueness for all cases.In this paper we analyze the $q$-ary Potts model, i.e., broadcasting of $q$-ary spins on a Galton-Watson tree with expected offspring degree $d$ through Potts channels with second-largest eigenvalue $\lambda$. We allow the intermediate vertices to be observed through noisy channels (side information). We prove that BP uniqueness holds with and without side information when $d\lambda^2 \ge 1 + C \max\{\lambda, q^{-1}\}\log q$ for some absolute constant $C>0$ independent of $q,\lambda,d$. For large $q$ and $\lambda = o(1/\log q)$, this is asymptotically achieving the Kesten-Stigum threshold $d\lambda^2=1$. These results imply mutual information formulas and optimal recovery algorithms for the $q$-community SBM in the corresponding ranges.For $q\ge 4$, Sly (2011); Mossel et al. (2022) showed that there exist choices of $q,\lambda,d$ below Kesten-Stigum (i.e. $d\lambda^2 < 1$) but reconstruction is possible. Somewhat surprisingly, we show that in such regimes BP uniqueness does not hold at least in the presence of weak side information.Our technical tool is a theory of $q$-ary symmetric channels, that we initiate here, generalizing the classical and widely-utilized information-theoretic characterization of BMS (binary memoryless symmetric) channels.

We study the weak recovery problem on the $r$-uniform hypergraph stochastic block model ($r$-HSBM) with two balanced communities. In HSBM a random graph is constructed by placing hyperedges with higher density if all vertices of a hyperedge share the same binary label, and weak recovery asks to recover a non-trivial fraction of the labels. We introduce a multi-terminal version of strong data processing inequalities (SDPIs), which we call the multi-terminal SDPI, and use it to prove a variety of impossibility results for weak recovery. In particular, we prove that weak recovery is impossible below the Kesten-Stigum (KS) threshold if $r=3,4$, or a strength parameter $\lambda$ is at least $\frac 15$. Prior work Pal and Zhu (2021) established that weak recovery in HSBM is always possible above the KS threshold. Consequently, there is no information-computation gap for these cases, which (partially) resolves a conjecture of Angelini et al. (2015). To our knowledge this is the first impossibility result for HSBM weak recovery.As usual, we reduce the study of non-recovery of HSBM to the study of non-reconstruction in a related broadcasting on hypertrees (BOHT) model. While we show that BOHT’s reconstruction threshold coincides with KS for $r=3,4$, surprisingly, we demonstrate that for $r\ge 7$ reconstruction is possible also below KS. This shows an interesting phase transition in the parameter $r$, and suggests that for $r\ge 7$, there might be an information-computation gap for the HSBM. For $r=5,6$ and large degree we propose an approach for showing non-reconstruction below KS, suggesting that $r=7$ is the correct threshold for onset of the new phase.

We consider the problem of empirical Bayes estimation for (multivariate) Poisson means. Existing solutions that have been shown theoretically optimal for minimizing the regret (excess risk over the Bayesian oracle that knows the prior) have several shortcomings. For example, the classical Robbins estimator does not retain the monotonicity property of the Bayes estimator and performs poorly under moderate sample size. Estimators based on the minimum distance and non-parametric maximum likelihood (NPMLE) methods correct these issues, but are computationally expensive with complexity growing exponentially with dimension. Extending the approach of Barbehenn andZhao (2022), in this work we construct monotone estimators based on empirical risk minimization (ERM) that retain similar theoretical guarantees and can be computed much more efficiently. Adapting the idea of offset Rademacher complexity Liang et al. (2015) to the non-standard loss and function class in empirical Bayes, we show that the shape-constrained ERM estimator attains the minimax regret within constant factors in one dimension and within logarithmic factors in multiple dimensions.

We consider the question of estimating multi-dimensional Gaussian mixtures (GM) with com- pactly supported or subgaussian mixing distributions. Minimax estimation rate for this class (under Hellinger, TV and KL divergences) is a long-standing open question, even for dimension one. In this paper we characterize this rate (in all dimensions) in terms of the metric entropy of the class. Such characterizations originate from seminal works of Le Cam (1973); Birge ́ (1983); Haussler and Opper (1997); Yang and Barron (1999). However, for GMs a key ingredient missing from earlier work (and widely sought-after) is a comparison result showing that the KL and the squared Hellinger distance are within a constant multiple of each other uniformly over the class. Our main technical contribution is in showing this fact, from which we derive entropy characterization for estimation rate under Hellinger and KL. Interestingly, the sequential (online learning) estimation rate is characterized by the global entropy, while the single-step (batch) rate corresponds to local entropy, paralleling a similar recent discovery for the case of Gaussian sequence model in a pair of works Neykov (2022); Mourtada (2023). Additionally, since Hellinger is a proper metric, our comparison shows that GMs under KL satisfy a version of triangle inequality (with a multiplicative constant), implying that proper and improper estimation rates coincide.

We aim to understand the extent to which the noise distribution in a planted signal-plus-noise problem impacts its computational complexity. To that end, we consider the planted clique and planted dense subgraph problems, but in a different ambient graph. Instead of Erdős-Rényi $G(n, p)$, which has independent edges, we take the ambient graph to be the random graph with triangles (RGT) obtained by adding triangles to $G(n, p)$. We show that the RGT can be efficiently mapped to the corresponding $G(n, p)$, and moreover, that the planted clique (or dense subgraph) is approximately preserved under this mapping. This constitutes the first average-case reduction transforming dependent noise to independent noise. Together with the easier direction of mapping the ambient graph from Erdős-Rényi to RGT, our results yield a strong equivalence between models. In order to prove our results, we develop a new general framework for reasoning about the validity of average-case reductions based on low sensitivity to perturbations.

We study the single-channel source separation problem involving orthogonal frequency-division multiplexing (OFDM) signals, which are ubiquitous in many modern-day digital communication systems. Related efforts have been pursued in monaural source separation, where state-of-the-art neural architectures have been adopted to train an end-to-end separator for audio signals (as 1-dimensional time series). In this work, through a prototype problem based on the OFDM source model, we assess—and question—the efficacy of using audio-oriented neural architectures in separating signals based on features pertinent to communication waveforms. Perhaps surprisingly, we demonstrate that in some configurations, where perfect separation is theoretically attainable, these audio-oriented neural architectures perform poorly in separating co-channel OFDM waveforms. Yet, we propose critical domain-informed modifications to the network parameterization, based on insights from OFDM structures, that can confer about 30 dB improvement in performance.

In this work we demonstrate how a lack of synchronization can in fact be advantageous in the problem of random access. Specifically, we consider a multiple-access problem over a frame-asynchronous 2-user binary-input adder channel in the unsourced setup (2-UBAC). Previous work has shown that under perfect synchronization the per-user rates achievable with linear codes over the 2-UBAC are limited by 0.5 bit per channel use (compared to the capacity of 0.75). In this paper, we first demonstrate that arbitrary small (even single-bit) shift between the user’s frames enables (random) linear codes to attain full capacity of 0.75 bit/user. Furthermore, we derive density evolution equations for irregular LDPC codes, and prove (via concentration arguments) that they correctly track the asymptotic bit-error rate of a BP decoder. Optimizing the degree distributions we construct LDPC codes achieving per-user rates of 0.73 bit per channel use.

Consider a pair of input distributions which after passing through a Poisson channel become ϵ-close in total variation. We show that they must necessarily then be ϵ0.5+o(1)-close after passing through a Gaussian channel as well. In the opposite direction, we show that distributions inducing ϵ-close outputs over the Gaussian channel must induce ϵ1+o(1)-close outputs over the Poisson. This quantifies a well-known intuition that "smoothing" induced by Poissonization and Gaussian convolution are similar. As an application, we improve a recent upper bound of Han-Miao-Shen’2021 for estimating mixing distribution of a Poisson mixture in Gaussian optimal transport distance from n−0.1+o(1)to n−0.25+o(1).

In the study of Ising models on large locally tree-like graphs, in both rigorous and non-rigorous methods one is often led to understanding the so-called belief propagation (BP) distributional recursions and its fixed points. We prove that there is at most one non-trivial BP fixed point, which was only known previously for sufficiently "low-temperature" models. Our main innovation is in applying information-theoretic ideas of channel comparison leading to a new metric (degradation index) between binary-input-symmetric (BMS) channels under which the BP operator is a strict contraction (albeit non-multiplicative). A key ingredient of our proof is a strengthening of the classical stringy tree lemma of [1].

Given $n$ observations from two balanced classes, consider the task of labeling an additional $m$ inputs that are known to all belong to \emph{one} of the two classes. Special cases of this problem are well-known: with complete knowledge of class distributions ($n=\infty$) the problem is solved optimally by the likelihood-ratio test; when $m=1$ it corresponds to binary classification; and when $m\approx n$ it is equivalent to two-sample testing. The intermediate settings occur in the field of likelihood-free inference, where labeled samples are obtained by running forward simulations and the unlabeled sample is collected experimentally. In recent work it was discovered that there is a fundamental trade-off between $m$ and $n$: increasing the data sample $m$ reduces the amount $n$ of training/simulation data needed. In this work we (a) introduce a generalization where unlabeled samples come from a mixture of the two classes -- a case often encountered in practice; (b) study the minimax sample complexity for non-parametric classes of densities under \textit{maximum mean discrepancy} (MMD) separation; and (c) investigate the empirical performance of kernels parameterized by neural networks on two tasks: detection of the Higgs boson and detection of planted DDPM generated images amidst CIFAR-10 images. For both problems we confirm the existence of the theoretically predicted asymmetric $m$ vs $n$ trade-off.

Viewing Transformers as interacting particle systems, we describe the geometry of learned representations when the weights are not time-dependent. We show that particles, representing tokens, tend to cluster toward particular limiting objects as time tends to infinity. Using techniques from dynamical systems and partial differential equations, we show that type of limiting object that emerges depends on the spectrum of the value matrix. Additionally, in the one-dimensional case we prove that the self-attention matrix converges to a low-rank Boolean matrix. The combination of these results mathematically confirms the empirical observation made by Vaswani et al. [ VSP`17 ] that leaders appear in a sequence of tokens when processed by Transformers.

We propose a new method for separating superimposed sources using diffusion-based generative models. Our method relies only on separately trained statistical priors of independent sources to establish a new objective function guided by $\textit{maximum a posteriori}$ estimation with an $\textit{$\alpha$-posterior}$, across multiple levels of Gaussian smoothing. Motivated by applications in radio-frequency (RF) systems, we are interested in sources with underlying discrete nature and the recovery of encoded bits from a signal of interest, as measured by the bit error rate (BER). Experimental results with RF mixtures demonstrate that our method results in a BER reduction of 95\% over classical and existing learning-based methods. Our analysis demonstrates that our proposed method yields solutions that asymptotically approach the modes of an underlying discrete distribution. Furthermore, our method can be viewed as a multi-source extension to the recently proposed score distillation sampling scheme, shedding additional light on its use beyond conditional sampling. The project webpage is available at https://alpha-rgs.github.io.

We establish the capacity of a class of communication channels introduced by Makur. The$n$-letter input from a finite alphabet is passed through a discrete memoryless channel$P_{Z|X}$and then the output$n$-letter sequence is uniformly permuted. We show that the maximal communication rate (normalized by$\log n$) equals${\frac{1}{ 2}} ( \textsf {rank}(P_{Z|X})-1)$whenever$P_{Z|X}$is strictly positive. This is done by establishing a converse bound matching the achievability of Makur. The two main ingredients of our proof are: 1) a sharp bound on the Kullback-Leibler divergence of a uniformly sampled vector from a type class and observed through a DMC to an iid vector; and 2) the covering$\varepsilon $-net of a probability simplex with Kullback-Leibler divergence as a metric. In addition to strictly positive DMC we also find the noisy permutation capacity for$q$-ary erasure channels, the Z-channel and others.

We study the potential of data-driven deep learning methods for separation of two communication signals from an observation of their mixture. In particular, we assume knowledge on the generation process of one of the signals, dubbed signal of interest (SOI), and no knowledge on the generation process of the second signal, referred to as interference. This form of the single-channel source separation problem is also referred to as interference rejection. We show that capturing high-resolution temporal structures (nonstationarities), which enables accurate synchronization to both the SOI and the interference, leads to substantial performance gains. With this key insight, we propose a domain-informed neural network (NN) design that is able to improve upon both “off-the-shelf” NNs and classical detection and interference rejection methods, as demonstrated in our simulations. Our findings highlight the key role communication-specific domain knowledge plays in the development of data-driven approaches that hold the promise of unprecedented gains.

A number of engineering and scientific problems require representing and manipulating probability distributions over large alphabets, which we may think of as long vectors of reals summing to 1. In some cases it is required to represent such a vector with only b bits per entry. A natural choice is to partition the interval [0,1] into 2buniform bins and quantize entries to each bin independently. We show that a minor modification of this procedure – applying an entrywise non-linear function (compander) f(x) prior to quantization – yields an extremely effective quantization method. For example, for b = 8(16) and 105-sized alphabets, the quality of representation improves from a loss (under KL divergence) of 0.5(0.1) bits/entry to 10−4(10−9) bits/entry. Compared to floating point representations, our compander method improves the loss from 10−1(10−6) to 10−4(10−9) bits/entry. These numbers hold for both real-world data (word frequencies in books and DNA k-mer counts) and for synthetic randomly generated distributions. Theoretically, we set up a minimax optimality criterion and show that the compander $f(x) \propto \operatorname{ArcSinh} (\sqrt {(1/2)(K\log K)x} )$ achieves near-optimal performance, attaining a KL-quantization loss of ≍ 2−2blog2K for a K-letter alphabet and b →∞. Interestingly, a similar minimax criterion for the quadratic loss on the hypercube shows optimality of the standard uniform quantizer. This suggests that the ArcSinh quantizer is as fundamental for KL-distortion as the uniform quantizer for quadratic distortion.

We establish the capacity of a class of communication channels introduced in [2]. The n-letter input from a finite alphabet is passed through a discrete memoryless channel PZ|Xand then the output n-letter sequence is uniformly permuted. We show that the maximal communication rate (normalized by log n) equals $\frac{1}{2}\left( {\operatorname{rank} \left( {{P_{Z\mid X}}} \right) - 1} \right)$ whenever PZ|Xis strictly positive. This is done by establishing a converse bound matching the achievability of [2]. The two main ingredients of our proof are (1) a sharp bound on the entropy of a uniformly sampled vector from a type class and observed through a DMC; and (2) the covering ε-net of a probability simplex with Kullback-Leibler divergence as a metric. In addition to strictly positive DMC we also find the noisy permutation capacity for q-ary erasure channels, the Z-channel and others.

It has long been thought that high-dimensional data encountered in many practical machine learning tasks have low-dimensional structure, i.e., the manifold hypothesis holds. A natural question, thus, is to estimate the intrinsic dimension of a given population distribution from a finite sample. We introduce a new estimator of the intrinsic dimension and provide finite sample, non-asymptotic guarantees. We then apply our techniques to get new sample complexity bounds for Generative Adversarial Networks (GANs) depending only on the intrinsic dimension of the data.

We study an important specialization of the general problem of broadcasting on directed acyclic graphs, namely, that of broadcasting on two-dimensional (2D) regular grids. Consider an infinite directed acyclic graph with the form of a 2D regular grid, which has a single source vertex$X$at layer 0, and$k + 1$vertices at layer$k \geq 1$, which are at a distance of$k$from$X$. Every vertex of the 2D regular grid has outdegree 2, the vertices at the boundary have indegree 1, and all other non-source vertices have indegree 2. At time 0,$X$is given a uniform random bit. At time$k \geq 1$, each vertex in layer$k$receives transmitted bits from its parents in layer$k-1$, where the bits pass through independent binary symmetric channels with common crossover probability$\delta \in \left({0,\frac {1}{2}}\right)$during the process of transmission. Then, each vertex at layer$k$with indegree 2 combines its two input bits using a common deterministic Boolean processing function to produce a single output bit at the vertex. The objective is to recover$X$with probability of error better than$\frac {1}{2}$from all vertices at layer$k$as$k \rightarrow \infty $. Besides their natural interpretation in the context of communication networks, such broadcasting processes can be construed as one-dimensional (1D) probabilistic cellular automata, or discrete-time statistical mechanical spin-flip systems on 1D lattices, with boundary conditions that limit the number of sites at each time$k$to$k+1$. Inspired by the literature surrounding the “positive rates conjecture” for 1D probabilistic cellular automata, we conjecture that it is impossible to propagate information in a 2D regular grid regardless of the noise level$\delta $and the choice of common Boolean processing function. In this paper, we make considerable progress towards establishing this conjecture, and prove using ideas from percolation and coding theory that recovery of$X$is impossible for any$\delta \in \left({0,\frac {1}{2}}\right)$provided that all vertices with indegree 2 use either AND or XOR for their processing functions. Furthermore, we propose a detailed and general martingale-based approach that establishes the impossibility of recovering$X$for any$\delta \in \left({0,\frac {1}{2}}\right)$when all NAND processing functions are used if certain structured supermartingales can be rigorously constructed. We also provide strong numerical evidence for the existence of these supermartingales by computing several explicit examples for different values of$\delta $via linear programming.

For any channel$P_{Y|X}$the strong data processing constant is defined as the smallest number$\eta _{KL}\in [{0,1}]$such that$I(U;Y)\le \eta _{KL} I(U;X)$holds for any Markov chain$U-X-Y$. It is shown that the value of$\eta _{KL}$is given by that of the best binary-input subchannel of$P_{Y|X}$. The same result holds for any$f$-divergence, verifying a conjecture of Cohen, Kemperman and Zbaganu (1998).

The limit of the entropy in the stochastic block model (SBM) has been characterized in the sparse regime for the special case of disassortative communities [Coja-Oghlan et al. (2017)] and for the classical case of assortative communities but in the dense regime [Deshpande et al. (2016)]. The problem has not been closed in the classical sparse and assortative case. This paper establishes the result in this case for any SNR besides for the interval (1, 3.513). It further gives an approximation to the limit in this window. The result is obtained by expressing the global SBM entropy as an integral of local tree entropies in a broadcasting on tree model with erasure side-information. The main technical advancement then relies on showing the irrelevance of the boundary in such a model, also studied with variants in [Kanade et al. (2016)], [Mossel et al. (2016)] and [Mossel and Xu (2015)]. In particular, we establish the uniqueness of the BP fixed point in the survey model for any SNR above 3.513 or below 1. This only leaves a narrow region in the plane between SNR and survey strength where the uniqueness of BP conjectured in these papers remains unproved.

We consider the question of sequential prediction under the log-loss in terms of cumulative regret. Namely, given a hypothesis class of distributions, learner sequentially predicts the (distribution of the) next letter in sequence and its performance is compared to the baseline of the best constant predictor from the hypothesis class. The well-specified case corresponds to an additional assumption that the data-generating distribution belongs to the hypothesis class as well. Here we present results in the more general misspecified case. Due to special properties of the log-loss, the same problem arises in the context of competitive-optimality in density estimation, and model selection. For the $d$-dimensional Gaussian location hypothesis class, we show that cumulative regrets in the well-specified and misspecified cases asymptotically coincide. In other words, we provide an $o(1)$ characterization of the distribution-free (or PAC) regret in this case – the first such result as far as we know. We recall that the worst-case (or individual-sequence) regret in this case is larger by an additive constant ${d\over 2} + o(1)$. Surprisingly, neither the traditional Bayesian estimators, nor the Shtarkov’s normalized maximum likelihood achieve the PAC regret and our estimator requires special “robustification” against heavy-tailed data. In addition, we show two general results for misspecified regret: the existence and uniqueness of the optimal estimator, and the bound sandwiching the misspecified regret between well-specified regrets with (asymptotically) close hypotheses classes.

Consider the problem of representing a distribution$\pi$on a large alphabet of size$k$up to fidelity$\varepsilon$in Kullback-Leibler (KL) divergence. Heuristically, arguing as for quadratic loss in high dimension, one expects that about$(k/2)\log(1/\varepsilon)$bits would be required. We show this intuition is correct by proving explicit non-asymptotic bounds for the minimal average distortion when$\pi$is randomly sampled from a symmetric Dirichlet prior on the simplex. Our method is to reduce the single-sample problem to the traditional setting of iid samples, but for a non-standard rate distortion question with the novel distortion measure$d(x, y)= x\log(x/y)$, which we call divergence distortion. Practically, our results advocate using a$x\mapsto x^{2/3}$compander (for small$x$) followed by a uniform scalar quantizer for storing large-alphabet distributions.

We investigate the problem of broadcasting a bit on a 2D regular grid. Consider a directed acyclic graph with the structure of a 2D regular grid, which has a single source vertex$X$at layer 0, and$k+1$vertices at distance of$k\geq 1$from$X$at layer$k$. Every vertex has outdegree 2, the boundary vertices have indegree 1, and the interior vertices have indegree 2. At time 0,$X$is given a uniform random bit. At time$k\geq 1$, each vertex in layer$k$receives bits from its parents in layer$k-1$, where the bits pass through binary symmetric channels with crossover probability$\delta\in\left(0,\frac{1}{2}\right)$. Each vertex with indegree 2 then combines its input bits with a common Boolean processing function to produce its output bit. The goal is to reconstruct$X$with probability of error less than$\frac{1}{2}$from all vertices at layer$k$as$k\rightarrow\infty$. Besides their natural interpretation in communication networks, such stochastic processes can be construed as 1D probabilistic cellular automata (PCA) with boundary conditions on the number of sites per layer. Inspired by the “positive rates conjecture” for 1D PCA, we establish that reconstruction of$X$is impossible for any$\delta$provided that either AND or XOR gates are employed as the common processing function. Furthermore, we show that if certain structured supermartingales exist, reconstruction is impossible for any$\delta$when a common NAND processing function is used. We also provide numerical evidence for the existence of these supermartingales using linear programming.

Consider a setting where a single bit is broadcast down the d-ary tree, where each edge acts as a binary symmetric channel with a crossover probability δ. The goal is to reconstruct the root bit given the values of all bits at a large distance$h$from the root. It is known the reconstruction is impossible iff (1 - 2δ)2d ≤ 1. In this paper, we show that in the regime where the latter product converges to 1 from the above, the distribution of the log-likelihood ratio (LLR) of the root bit given the far-away boundary (normalized by the square root of deviation of δ from criticality) converges to an explicit Gaussian distribution. This strengthens a similar result of Jain-Koehler-Liu-Mossel (COLT'2019) and enables us to resolve conjectures stated in Gu-Roozbehani-Polyanskiy (ISIT'2020) for the scaling of the probability of error and mutual information near criticality. Our results also provide a rationale for the ubiquitous$N$(µ, 2µ) approximation of the LLR distribution in the EXIT-chart heuristics.

Let X and Y be dependent random variables. This paper considers the problem of designing a scalar quantizer for Y to maximize the mutual information between the quantizer's output and X, and develops fundamental properties and bounds for this form of quantization, which is connected to the log-loss distortion criterion. The main focus is the regime of low I(X;Y), where it is shown that, if X is binary, a constant fraction of the mutual information can always be preserved usingO(log(1/I(X;Y))) quantization levels, and there exist distributions for which this many quantization levels are necessary. Furthermore, for larger finite alphabets 2X|X| /I(X;Y)))η·(|X| - 1)quantization levels.

Most information storage devices write data by modifying the local state of matter, in the hope that sub-atomic local interactions stabilize the state for sufficiently long time, thereby allowing later recovery. Motivated to explore how temporal evolution of physical states in magnetic storage media affects their capacity, this work initiates the study of information retention in locally-interacting particle systems. The system dynamics follow the stochastic Ising model (SIM) over a 2-dimensional √(n) × √(n) grid. The initial spin configuration X0serves as the user-controlled input. The output configuration Xt is produced by running t steps of Glauber dynamics. Our main goal is to evaluate the information capacity In(t) := maxpx0 I(X0; Xt) when time t scales with the system's size n. While the positive (but low) temperature regime is our main interest, we start by exploring the simpler zero-temperature dynamics. We first show that at zero temperature, order of √(n) bits can be stored in the system indefinitely by coding over stable, striped configurations. While √(n) is order optimal for infinite time, backing off to tn(t) are achievable. First, via linear coding arguments imply we show that In(t) = Θ(n) for t = O(n). To go beyond the linear scale, we develop a droplet-based achievability scheme that reliably stores Ω (n/ log n) for t = O(n log n) time (log n can be replaced with any o(n) function). Moving to the positive but low temperature regime, two main results are provided. First, we show that an initial configuration drawn from the Gibbs measure cannot retain more than a single bit for t ≥ exp(Cβn1/4+c) time. On the other hand, when scaling time with the inverse temperature β, the stripe-based coding scheme (that stores for infinite time at zero temperature) is shown to retain its bits for ecβ.

Consider a (multiple-access) wireless communication system where users are connected to a unique base station over a shared-spectrum radio links. Each user has a fixed number k of bits to send to the base station, and his signal gets attenuated by a random channel gain (quasi-static fading). In this paper we consider the many-user asymptotics of Chen-Chen-Guo'2017, where the number of users grows linearly with the blocklength. Differently, though, we adopt a per-user probability of error (PUPE) criterion (as opposed to classical joint-error probability criterion). Under PUPE the finite energy-per-bit communication is possible, and we are able to derive bounds on the tradeoff between energy and spectral efficiencies. We reconfirm the curious behaviour (previously observed for non-fading MAC) of the possibility of almost perfect multi-user interference (MUI) cancellation for user densities below a critical threshold. Further, we demonstrate the suboptimality of standard solutions such as orthogonalization (i.e. TDMA/FDMA) and treating interference as noise (i.e. pseudo-random CDMA without multi-user detection). Notably, the problem treated here can be seen as a variant of support recovery in compressed sensing for the unusual definition of sparsity with one non-zero entry per each contiguous section of 2kcoordinates. This identifies our problem with that of the sparse regression codes (SPARCs) and hence our results can be equivalently understood in the context of SPARCs with sections of length 2100. Finally, we discuss the relation of the almost perfect MUI cancellation property and the replica-method predictions.

We study a prototypical problem in empirical Bayes. Namely, consider a population consisting of $k$ individuals each belonging to one of $k$ types (some types can be empty). Without any structural restrictions, it is impossible to learn the composition of the full population having observed only a small (random) subsample of size $m = o(k)$. Nevertheless, we show that in the sublinear regime of $m =\omega(k/\log k)$, it is possible to consistently estimate in total variation the \emph{profile} of the population, defined as the empirical distribution of the sizes of each type, which determines many symmetric properties of the population. We also prove that in the linear regime of $m=c k$ for any constant $c$ the optimal rate is $\Theta(1/\log k)$. Our estimator is based on Wolfowitz’s minimum distance method, which entails solving a linear program (LP) of size $k$. We show that there is a single infinite-dimensional LP whose value simultaneously characterizes the risk of the minimum distance estimator and certifies its minimax optimality. The sharp convergence rate is obtained by evaluating this LP using complex-analytic techniques.

We revisit the problem of broadcasting on d-ary trees: starting from a Bernoulli(1/2) random variable X0at a root vertex, each vertex forwards its value across binary symmetric channels BSCδto d descendants. The goal is to reconstruct X0given the vector XLhof values of all variables at depth h. It is well known that reconstruction (better than a random guess) is possible as h →∞ if and only if δc(d). In this paper, we study the behavior of the mutual information and the probability of error when δ is slightly subcritical. The innovation of our work is application of the recently introduced "less-noisy" channel comparison techniques. For example, we are able to derive the positive part of the phase transition (reconstructability when δc) using purely information-theoretic ideas. This is in contrast with previous derivations, which explicitly analyze distribution of the Hamming weight of XLh(a so-called Kesten-Stigum bound).

We study a problem of constructing codes that transform a channel with high bit error rate (BER) into one with low BER (at the expense of rate). Our focus is on obtaining codes with smooth (“graceful”) input-output BER curves (as opposed to threshold-like curves typical for long error-correcting codes). This paper restricts attention to binary erasure channels (BEC) and contains two contributions. First, we introduce the notion of Low Density Majority Codes (LDMCs). These codes are non-linear sparse-graph codes, which output majority function evaluated on randomly chosen small subsets of the data bits. This is similar to Low Density Generator Matrix codes (LDGMs), except that the XOR function is replaced with the majority. We show that even with a few iterations of belief propagation (BP) the attained input-output curves provably improve upon performance of any linear systematic code. The effect of nonlinearity bootstraping the initial iterations of BP, suggests that LDMCs should improve performance in various applications where LDGMs have been used traditionally. Second, we establish several two-point converse bounds that lower bound the BER achievable at one erasure probability as a function of BER achieved at another one. The novel nature of our bounds is that they are specific to subclasses of codes (linear systematic and non-linear systematic) and outperform similar bounds implied by the area theorem for the EXIT function.

We discuss the problem of designing channel access architectures for enabling fast, low-latency, grant-free, and uncoordinated uplink for densely packed wireless nodes. Specifically, we study random-access codes, previously introduced for the AWGN MAC, in the practically more relevant case of Rayleigh fading, when channel gains are unknown to the decoder. We propose a random coding achievability bound, which we analyze both non-asymptotically and asymptotically. As a candidate practical solution, we propose an explicit iterative coding scheme. The performance of such a solution is surprisingly close to the finite blocklength bounds. Our main findings are twofold. First, just like in the AWGN MAC, we see that jointly decoding a large number of users leads to a surprising phase transition effect, where, at spectral efficiencies below a critical threshold, a perfect multi-user interference cancellation is possible. Second, while the presence of Rayleigh fading significantly increases the minimal required energy-per-bit, the inherent randomization introduced by the channel makes it much easier to attain the optimal performance via iterative schemes. We hope that a principled definition of the random-access model, together with their information-theoretic analysis, will open the road towards unified benchmarking and performance comparison of various random-access solutions for the 5G/6G.

This paper studies convergence of empirical measures smoothed by a Gaussian kernel. Specifically, consider approximating P*Nσ, for Nσ=△N(0, σ2Id), by P̑n*Nσunder different statistical distances, where P̑nis the empirical measure. We examine the convergence in terms of the Wasserstein distance, total variation (TV), Kullback-Leibler (KL) divergence, and χ2-divergence. We show that the approximation error under the TV distance and 1-Wasserstein distance (W1) converges at the rate eO(d)n-1/2in remarkable contrast to a (typical) n-1/drate for unsmoothed W1(and d ≥ 3). Similarly, for the KL divergence, squared 2-Wasserstein distance (W22), and χ2-divergence, the convergence rate is eO(d)n-1, but only if P achieves finite input-output χ2mutual information across the additive white Gaussian noise (AWGN) channel. If the latter condition is not met, the rate changes to ω (n-1) for the KL divergence and W22, while the χ2-divergence becomes infinite - a curious dichotomy. As an application we consider estimating the differential entropy h(S + Z), where S ~ P and Z ~ Nσare independent d-dimensional random variables. The distribution P is unknown and belongs to some nonparametric class, but n independently and identically distributed (i.i.d) samples from it are available. Despite the regularizing effect of noise, we first show that any good estimator (within an additive gap) for this problem must have a sample complexity that is exponential in d. We then leverage the above empirical approximation results to show that the absolute-error risk of the plug-in estimator converges as eO(d)n-1/2, thus attaining the parametric rate in n. This establishes the plug-in estimator as minimax rate-optimal for the considered problem, with sharp dependence of the convergence rate both in n and d. We provide numerical results comparing the performance of the plug-in estimator to that of general-purpose (unstructured) differential entropy estimators (based on kernel density estimation (KDE) or k nearest neighbors (kNN) techniques) applied to samples of S + Z. These results reveal a significant empirical superiority of the plug-in to state-of-the-art KDE and kNN methods. As a motivating utilization of the plug-in approach, we estimate information flows in deep neural networks and discuss Tishby's Information Bottleneck and the compression conjecture, among others.

We study the following generalization of the wellknown model of broadcasting on trees. Consider an infinite directed acyclic graph (DAG) with a unique source vertex X. Let the collection of vertices at distance k from X be called the kth layer, and suppose every non-source vertex has indegree d ≥ 2. At layer 0, the source vertex is given a random bit. At layer k ≥ 1, each vertex receives d bits from its parents in the (k-1)th layer, which are transmitted along edges that are independent binary symmetric channels (BSCs) with crossover probability δ ∈ (0, 1/2). Each vertex combines its d noisy inputs using a 2 deterministic d-ary Boolean processing function that generates the value at the vertex. The goal is to be able to reconstruct the original bit X with probability of error bounded away from 1/2 using the values of all vertices at an arbitrarily deep layer k. This question is closely related to models of reliable computation and storage, and information flow in biological networks. In this paper, we treat the case of randomly constructed DAGs, for which we show that broadcasting is only possible if the BSC noise level δ is below a certain (degree and function dependent) critical threshold. For d ≥ 3, and random DAGs with layers of size Ω(log(k)) and majority processing functions, we identify the critical threshold. For d = 2, we establish a similar result for the NAND processing function. We also prove a partial converse result for odd d ≥ 3 illustrating that the identified thresholds are impossible to improve by selecting different processing functions if the decoder is restricted to using a single vertex's value. Finally, for any BSC noise level δ, we construct explicit DAGs (using regular bipartite lossless expander graphs) with bounded degree and layers of size Θ(log(k)) admitting reconstruction. In particular, we show that the first r layers of such DAGs can be generated in either deterministic quasipolynomial time or randomized polylogarithmic time in r. These results portray a doubly-exponential advantage for storing a bit in bounded degree DAGs compared to trees, where d = 1 but layer sizes need to grow exponentially with depth in order for broadcasting to be possible.

Consider two sequences of n independent and identically distributed fair coin tosses, X = (X1, . . . , Xn) and Y = (Y1, . . . , Yn), which are ρ-correlated for each j, i.e. P[Xj= Yj] = 1+ρ/2 .We study the question of how large (small) the probability P[X ∈ A, Y ∈ B] can be among all sets A, B ⊂ {0, 1}nof a given cardinality. For sets |A|, |B| = Θ(2n) it is well known that the largest (smallest) probability is approximately attained by concentric (anti-concentric) Hamming balls, and this can be proved via the hypercontractive inequality (reverse hypercontractivity). Here we consider the case of |A|, |B| = 2Θ(n). By applying a recent extension of the hypercontractive inequality of Polyanskiy-Samorodnitsky (J. Functional Analysis, 2019), we show that Hamming balls of the same size approximately maximize P[X ∈ A, Y ∈ B] in the regime of p → 1. We also prove a similar tight lower bound, i.e. show that for p → 0 the pair of opposite Hamming balls approximately minimizes the probability P[X ∈ A, Y ∈ B].

In this paper, we consider a problem of sampling a Wiener process, with samples forwarded to a remote estimator over a channel that is modeled as a queue. The estimator reconstructs an estimate of the real-time signal value from causally received samples. We study the optimal online sampling strategy that minimizes the mean square estimation error subject to a sampling rate constraint. We prove that the optimal sampling strategy is a threshold policy, and find the optimal threshold. This threshold is determined by how much the Wiener process varies during the random service time and the maximum allowed sampling rate. Further, if the sampling times are independent of the observed Wiener process, the above sampling problem for minimizing the estimation error is equivalent to a sampling problem for minimizing the age of information. This reveals an interesting connection between the age of information and remote estimation error. Our comparisons show that the estimation error achieved by the optimal sampling policy can be much smaller than those of age-optimal sampling, zero-wait sampling, and periodic sampling.

The deployment of massive MIMO systems has revived much of the interest in the study of the large-system performance of multiuser detection systems. In this paper, we prove a non-trivial upper bound on the bit-error rate (BER) of the MAP detector for BPSK signal transmission and equal-power condition. In particular, our bound is approximately tight at high-SNR. The proof is simple and relies on Gordon's comparison inequality. Interestingly, we show that under the assumption that Gordon's inequality is tight, the resulting BER prediction matches that of the replica method under the replica symmetry (RS) ansatz. Also, we prove that, when the ratio of receive to transmit antennas exceeds 0.9251, the replica prediction matches the matched filter lower bound (MFB) at high-SNR. We corroborate our results by numerical evidence.

We study the estimation of the mutual information I(X;T_$\ell$) between the input X to a deep neural network (DNN) and the output vector T_$\ell$ of its $\ell$-th hidden layer (an “internal representation”). Focusing on feedforward networks with fixed weights and noisy internal representations, we develop a rigorous framework for accurate estimation of I(X;T_$\ell$). By relating I(X;T_$\ell$) to information transmission over additive white Gaussian noise channels, we reveal that compression, i.e. reduction in I(X;T_$\ell$) over the course of training, is driven by progressive geometric clustering of the representations of samples from the same class. Experimental results verify this connection. Finally, we shift focus to purely deterministic DNNs, where I(X;T_$\ell$) is provably vacuous, and show that nevertheless, these models also cluster inputs belonging to the same class. The binning-based approximation of I(X;T_$\ell$) employed in past works to measure compression is identified as a measure of clustering, thus clarifying that these experiments were in fact tracking the same clustering phenomenon. Leveraging the clustering perspective, we provide new evidence that compression and generalization may not be causally related and discuss potential future research ideas.

Motivated by questions of data stabilization in emerging magnetic storage technologies, we study the retention of information in interacting particle systems. The interactions between particles adhere to the stochastic Ising model (SIM) on the two-dimensional (2D) √n × √n grid. The measure of interest is the information capacity In(t) =△ maxpX0I(X0; Xt), where the initial spin configuration X0is a user-controlled input and the output configuration Xtis produced by running t steps of Glauber dynamics. After the results on the zero-temperature regime reported last year, this work focuses on the positive but low temperature regime. We first show that storing more than a single bit for an exponential time is impossible when the initial configuration is drawn from the equilibrium distribution. Specifically, if X0is drawn according to the Gibbs measure, then I(X0; Xt) ≤ 1 + o(1) for t ≥ exp (cn1/4+ε). On the other hand, when scaling time with β, we propose a stripe-based coding scheme that stores order of √n bits for exp(β) time. Key to the analysis of the scheme is a new result on the survival time of a single plus-labeled stripe in a sea of minuses. Together, the 1-bit upper bound and the striped-based storage scheme constitute initial steps towards a general analysis of In(t) for β > 0.

This paper establishes the optimality of the plugin estimator for the problem of differential entropy estimation under Gaussian convolutions. Specifically, we consider the estimation of the differential entropy h(X + Z), where X and Z are independent d-dimensional random variables with Z ~ N(0, σ2Id). The distribution of X is unknown and belongs to some nonparametric class, but n independently and identically distributed samples from it are available. We first show that despite the regularizing effect of noise, any good estimator (within an additive gap) for this problem must have an exponential in d sample complexity. We then analyze the absolute-error risk of the plug-in estimator and show that it converges as cd√/n, thus attaining the parametric estimation rate. This implies the optimality of the plug-in estimator for the considered problem. We provide numerical results comparing the performance of the plug-in estimator to general-purpose (unstructured) differential entropy estimators (based on kernel density estimation (KDE) or k nearest neighbors (kNN) techniques) applied to samples of X + Z. These results reveal a significant empirical superiority of the plug-in to state-of-the-art KDEand kNN-based methods.

We study a distributed hypothesis testing problem where two parties observe i.i.d. samples from two ρ-correlated standard normal random variables X and Y. The party that observes the X-samples can communicate R bits per sample to the second party, that observes the Y-samples, in order to test between two correlation values. We investigate the best possible type-II error subject to a fixed type-I error, and derive an upper (impossibility) bound on the associated type-II error exponent. Our techniques include representing the conditional Y-samples as a trajectory of the Ornstein-Uhlenbeck process, and bounding the associated KL divergence using the subadditivity of the Wasserstein distance and the Gaussian Talagrand inequality.

We consider the problem of transmitting reliably one bit of information across a tandem of binary symmetric channels interconnected by a relay/processor station. In our setting, the relay is instantaneous in the sense that its outputs are allowed to causally depend on previous received noisy bits. For this model, we investigate the optimal exponential decay rate of the average probability of error, when relaying one bit of information using n synchronous channel uses, by devising good relaying schemes.

We consider the classic joint source-channel coding problem of transmitting a memoryless source over a memoryless channel. The focus of this work is on the rate of convergence of the smallest attainable expected distortion to its asymptotic value, as a function of blocklength n. Our main result is that in general the convergence rate is not faster than n-1/2. In particular, we show that for the problem of transmitting i.i.d uniform bits over a binary symmetric channels with Hamming distortion, the smallest attainable distortion (bit error rate) is at least Ω(n-1/2) above the asymptotic value, if the "bandwidth expansion ratio" is above 1.

We discuss the problem of designing channel access architectures for enabling fast, low-latency, grant-free and uncoordinated uplink for densely packed wireless nodes. Specifically, we extend the concept of random-access code introduced at ISIT'2017 by one of the authors to the practically more relevant case of the AWGN multiple-access channel (MAC) subject to Rayleigh fading, unknown to the decoder. We derive bounds on the fundamental limits of random-access coding and propose an alternating belief-propagation scheme as a candidate practical solution. The latter's performance was found to be surprisingly close to the information-theoretic bounds. It is curious, thus, that while fading significantly increases the minimal required energy-per-bit Eb/N0(from about 0-2 dB to about 8-11 dB), it appears that it is much easier to attain the optimal performance over the fading channel with a practical scheme by leveraging the inherent randomization introduced by the channel. Finally, we mention that while a number of candidate solutions (MUSA, SCMA, RSMA, etc.) are being discussed for the 5G, the information-theoretic analysis and benchmarking has not been attempted before (in part due to lack of common random-access model). Our work may be seen as a step towards unifying performance comparisons of these methods.

Consider a (multiple-access) wireless communication system where users are connected to a unique base station over a shared-spectrum radio links. Each user has a fixed number k of bits to send to the base station, and his signal gets attenuated by a random channel gain (quasi-static fading). In this paper we consider the many-user asymptotics of Chen-Chen-Guo'2017, where the number of users grows linearly with the blocklength. In addition, we adopt a per-user probability of error criterion of Polyanskiy'2017 (as opposed to classical joint-error probability criterion). Under these two settings we derive bounds on the optimal required energy-per-bit for reliable multi-access communication. We confirm the curious behaviour (previously observed for non-fading MAC) of the possibility of perfect multi-user interference cancellation for user densities below a critical threshold. Further we demonstrate the suboptimality of standard solutions such as orthogonalization (i.e., TDMA/FDMA) and treating interference as noise (i.e. pseudo-random CDMA without multi-user detection).

We study a generalization of the problem of broadcasting on trees to the setting of directed acyclic graphs (DAGs). At time 0, a source vertex X transmits a uniform bit along binary symmetric channels (BSCs) to a set of vertices called layer 1. Each vertex except X has indegree d. At time k ≥ 1, vertices at layer k apply d-input Boolean processing functions to their received bits and send out the results to vertices at layer k + 1. We say that broadcasting is possible if we can reconstruct X with probability of error bounded away from 1/2 using the values of all vertices at an arbitrarily deep layer k. This question is closely related to models of reliable computation and storage, probabilistic cellular automata, and information How in biological networks. In this work, we analyze randomly constructed DAGs and demonstrate that broadcasting is only possible if the BSC noise level is below a certain (degree and function dependent) critical threshold. Specifically, for every d ≥ 3, we identify the critical threshold for random DAGs with layers of size Ω(log(k)) and majority processing functions. For d = 2, we establish a similar result for the NAND processing function. Furthermore, for odd d ≥ 3, we prove that the identified thresholds cannot be improved by other processing functions if reconstruction is required from a single vertex. Finally, for any BSC noise level, in quasi-polynomial or randomized polylogarithmic time in the depth, we construct deterministic bounded degree DAGs with layers of size Θ(log(k)) that admit reconstruction using lossless expander graphs.

We consider the Gaussian multiple-access channel with two critical departures from the classical asymptotics: a) number of users proportional to block-length and b) each user sends a fixed number of data bits. We provide improved bounds on the tradeoff between the user density and the energy-per-bit. Interestingly, in this information-theoretic problem we rely on Gordon's lemma from Gaussian process theory. From the engineering standpoint, we discover a surprising new effect: good coded-access schemes can achieve perfect multi-user interference cancellation at low user density.In addition, by a similar method we analyze the limits of false-discovery in binary sparse regression problem in the asymptotic regime of number of measurements going to infinity at fixed ratios with problem dimension, sparsity and noise level. Our rigorous bound matches the formal replica-method prediction for some range of parameters with imperceptible numerical precision.

We characterize the communication complexity of the following distributed estimation problem. Alice and Bob observe infinitely many iid copies of ρ-correlated unit-variance (Gaussian or ±1 binary) random variables, with unknown ρ∈[−1,1]. By interactively exchanging k bits, Bob wants to produce an estimate ρ of ρ. We show that the best possible performance (optimized over interaction protocol Π and estimator ρ) satisfies infΠ ρsupρE [|ρ−ρ|2] = k−1 (1/2 ln2 + o(1)). Curiously, the number of samples in our achievability scheme is exponential in k; by contrast, a naive scheme exchanging k samples achieves the same Ω(1/k) rate but with a suboptimal prefactor. Our protocol achieving optimal performance is one-way (non-interactive). We also prove the Ω(1/k) bound even when ρ is restricted to any small open sub-interval of [−1,1] (i.e. a local minimax lower bound). Our proof techniques rely on symmetric strong data-processing inequalities and various tensorization techniques from information-theoretic interactive common-randomness extraction. Our results also imply an Ω(n) lower bound on the information complexity of the Gap-Hamming problem, for which we show a direct information-theoretic proof.

We investigate the problem of uncoordinated massive random access in the quasi-static asynchronous Rayleigh fading channel. In the previous work [1], the authors assumed a completely synchronous scenario which is impossible in any practical implementation. This paper extends the previous work to the asynchronous case. As energy efficiency is of critical importance for massive machine-type communication (mMTC), our main goal is to minimize the energy-per-bit required to achieve the target probability of error. Another issue required for mMTC is a transmitter simplicity. As in the synchronous case, we focus on grant-free transmission and do not use preambles and other synchronization sequences. We propose a practical implementation of a transmission scheme based on synchronization error estimation and cancellation in the frequency domain. The simulation shows that the proposed transmission scheme's performance is very close to the synchronous case. The only source of E_b/N_0 loss is the need for an additional cyclic prefix that helps to solve the synchronization error cancellation problem in the frequency domain.

Consider the linear space of functions on the binary hypercube and the linear operator $S_\delta$ acting by averaging a function over a Hamming sphere of radius $\delta n$ around every point. It is shown that this operator has a dimension-independent bound on the norm $L_p \to L_2$ with $p = 1+(1-2\delta)^2$. This result evidently parallels a classical estimate of Bonami and Gross for $L_p \to L_q$ norms for the operator of convolution with a Bernoulli noise. The estimate for $S_\delta$ is harder to obtain since the latter is neither a part of a semigroup nor a tensor power. The result is shown by a detailed study of the eigenvalues of $S_\delta$ and $L_p\to L_2$ norms of the Fourier multiplier operators $\Pi_a$ with symbol equal to a characteristic function of the Hamming sphere of radius $a$ (in the notation common in boolean analysis $\Pi_a f=f^{=a}$, where $f^{=a}$ is a degree-$a$ component of function $f$). A sample application of the result is given: Any set $A\subset \mathbb{F}_2^n$ with the property that $A+A$ contains a large portion of some Hamming sphere (counted with multiplicity) must have cardinality a constant multiple of $2^n$.

We consider list decoding in the zero-rate regime for two cases-the binary alphabet and the spherical codes in Euclidean space. Specifically, we study the maximal τ ∈ [0, 1] for which there exists an arrangement of M balls of relative Hamming radius τ in the binary hypercube (of arbitrary dimension) with the property that no point of the latter is covered by L or more of them. As M → ∞ the maximal τ decreases to a well-known critical value τL. In this paper, we prove several results on the rate of this convergence. For the binary case, we show that the rate is Θ(M-1) when L is even, thus extending the classical results of Plotkin and Levenshtein for L = 2. For L = 3, the rate is shown to be Θ(M-(2/3)). For the similar question about spherical codes, we prove the rate is Ω(M-1) and O(M-(2L/L(2)-L+2)).

In this paper, we consider a channel model that is often used to describe mobile wireless scenarios: multiple-antenna additive white Gaussian noise channels subject to random (fading) gains with full channel state information at the receiver. The dynamics of the fading process are approximated by a piecewise-constant process (frequency non-selective isotropic block fading). This paper addresses the finite block-length fundamental limits of this channel model. Specifically, we give a formula for the channel dispersion-a quantity governing the delay required to achieve capacity. The multiplicative nature of the fading disturbance leads to a number of interesting technical difficulties that required us to enhance traditional methods for finding the channel dispersion. Alas, one difficulty remains: the converse (impossibility) part of our result holds under an extra constraint on the growth of the peak-power with blocklength. Our results demonstrate, for example, that while the capacities of nt × nr and nr × nt antenna configurations coincide (under fixed received power), the coding delay can be sensitive to this switch. For example, at the received SNR of 20 dB, the 16 × 100 system achieves capacity with codes of length (delay) which is only 60% of the length required for the 100×16 system. Another interesting implication is that for the MISO channel, the dispersionoptimal coding schemes require employing orthogonal designs such as Alamouti's scheme-a surprising observation considering the fact that Alamouti's scheme was designed for reducing demodulation errors, not improving coding rate. Finding these dispersion-optimal coding schemes naturally gives a criteria for producing orthogonal design-like inputs in dimensions where orthogonal designs do not exist.

Most information systems store data by modifying the local state of the matter, in the hope that atomic (or subatomic) local interactions would stabilize the state for sufficiently long time, thereby allowing later recovery. In this work we initiate the study of information retention properties of locally-interacting systems. We model the time-dependent interactions between the different particles via the stochastic Ising model (SIM). The initial spin configuration X0serves as the user-controlled input. The output configuration Xtis produced by running t steps of the Glauber chain. Our main goal is to evaluate the information capacity In(t) =̂ maxpX0I(X0; Xt) when the time t scales with the size or the system n according to various rates. For the zero-temperature SIM on the two-dimensional √n×√n grid and free boundary condition, it is easy to show that In(t)=Θ(n) as long as t=O(n). In addition, we show that order of √n bits can be stored for infinite time (and even with zero error). The √n achievability is optimal when t→∞ and n is fixed. Our main result is in extending achievability to super-linear (in n) times via a coding scheme that reliably stores more than √n bits (in orders of magnitude). The analysis of the scheme decomposes the system into Ω(√n) independent Z-channels whose crossover probability is found via the (recently rigorously established) Lifshitz law of phase boundary movement. Finally, two order optimal characterizations of In(t), for all t, are given for the grid dynamics with an external magnetic field and for the dynamics on the Honeycomb lattice. It shown that In(t)=Θ(n) in both cases, suggesting their superiority over the grid without an external field for storage purposes.

Consider a binary linear code of length N, minimum distance dmin, transmission over the binary erasure channel with parameter 00 if the minimum distance is large. In particular the width of the transition is of order O(1/√dmin). We strengthen this result by showing that under suitable conditions on the weight distribution of the code, the transition width can be as small as O(1/N1/2-κ), for any κ > 0, even if the minimum distance of the code is not linear. This condition applies e.g., to Reed-Mueller codes. Since O(1/N1/2) is the smallest transition possible for any code, we speak of “almost” optimal scaling. We emphasize that the width of the transition says nothing about the location of the transition. Therefore this result has no bearing on whether a code is capacity-achieving or not. As a second contribution, we present a new estimate on the derivative of the EXIT function, the proof of which is based on the Blowing-Up Lemma.

Two problems, namely multiple-description source coding and joint source-channel broadcasting of a common source, are addressed. For the multiple-description problem, we revisit Ozarow's technique for establishing impossibility results, and extend it to general sources and distortion measures. For the problem of sending a source over a broadcast channel, we revisit the bounding technique of Reznik, Feder and Zamir, and extend it to general sources, distortion measures and broadcast channels. Although the obtained bounds do not improve over existing results in the literature, they are relatively easy to evaluate, and their derivation reveals the similarities between the two bounding techniques.

Let Znbe iid Bernoulli (δ) and Unbe uniform on the set of all binary vectors of weight δn (Hamming sphere). As is well known, the entropies of Znand Unare within O(logn). However, if Xnis another binary random variable independent of Znand Un, we show that H(Xn+Un) and H(Xn+Zn) are within O(√n) and this estimate is tight. The bound is shown via coupling method. Tightness follows from the observation that the channels xn→ xn+Unand xn→ xn+Znhave similar capacities, but the former has zero dispersion. Finally, we show that despite the √n slack in general, the Mrs. Gerber Lemma for H(Xn+Un) holds with only an O(logn) correction compared to its brethren for H(Xn+Zn).

Consider a linear code defined as a mapping between vector spaces of dimensions k and n. Let β* denote the minimal (relative) weight among all images of input vectors of full Hamming weight k. Operationally, β* characterizes the threshold for adversarial (erasure) noise beyond which decoder is guaranteed to produce estimate of k-input with 100% symbol error rate (SER). This paper studies the relation between β* and δ, the minimum distance of the code, which gives the threshold for 0 % SER. An optimal tradeoff between β* and δ is obtained (over large alphabets) and all linear codes achieving β*=1 are classified: they are repetition-like. More generally, a design criteria is proposed for codes with favorable graceful degradation properties. As an example, it is shown that in an overdetermined system of n homogeneous linear equations in k variables (over a field) it is always possible to satisfy some k-1 equations with non-zero assignments to every unknown, provided that any subset of k equations is linearly independent. This statement is true if and only if n ≥ 2k-1.

It is well known that the mutual information between two random variables can be expressed as the difference of two relative entropies that depend on an auxiliary distribution, a relation sometimes referred to as the golden formula. This paper is concerned with a finite-blocklength extension of this relation. This extension consists of two elements: 1) a finiteblocklength channel-coding converse bound by Polyanskiy and Verdú, which involves the ratio of two Neyman-Pearson β functions (beta-beta converse bound); and 2) a novel beta-beta channel-coding achievability bound, expressed again as the ratio of two Neyman-Pearson β functions. To demonstrate the usefulness of this finite-blocklength extension of the golden formula, the beta-beta achievability and converse bounds are used to obtain a finite-blocklength extension of Verdú's wideband-slope approximation. The proof parallels the derivation of the latter, with the beta-beta bounds used in place of the golden formula. The beta-beta (achievability) bound is also shown to be useful in cases where the capacity-achieving output distribution is not a product distribution due to, e.g., a cost constraint or structural constraints on the codebook, such as orthogonality or constant composition. As an example, the bound is used to characterize the channel dispersion of the additive exponential-noise channel and to obtain a finite-blocklength achievability bound (the tightest to date) for multiple-input multiple-output Rayleigh-fading channels with perfect channel state information at the receiver.

This paper quantifies the intuitive observation that adding noise reduces available information by means of nonlinear strong data processing inequalities. Consider the random variables W → X → Y forming a Markov chain, where Y = X+Z with X and Z real valued, independent and X bounded in Li-norm. It is shown that I(W; Y) ≤ FI(I(W; X)) with FI(t)0, if and only if Z has a density whose support is not disjoint from any translate of itself. A related question is to characterize for what couplings (W, X) the mutual information I(W; Y) is close to maximum possible. To that end we show that in order to saturate the channel, i.e., for I(W; Y) to approach capacity, it is mandatory that I(W; X) → ∞ (under suitable conditions on the channel). A key ingredient for this result is a deconvolution lemma which shows that postconvolution total variation distance bounds the preconvolution Kolmogorov- Smirnov distance. Explicit bounds are provided for the special case of the additive Gaussian noise channel with quadratic cost constraint. These bounds are shown to be order optimal. For this case, simplified proofs are provided leveraging Gaussianspecific tools such as the connection between information and estimation (I-MMSE) and Talagrand's information-transportation inequality.

New lower and upper bounds on the reliability function of typewriter channels are given. Our lower bounds improve upon the (multiletter) expurgated bound of Gallager, furnishing a new and simple counterexample to a conjecture made in 1967 by Shannon, Gallager and Berlekamp on its tightness. The only other known counter example is due to Katsman, Tsfasman and Vladut who used algebraic-geometric codes on a q-ary symmetric channels, q ≥ 49. Here we prove, by introducing dependence between codewords of a random ensemble, that the conjecture is false even for a typewriter channel with q = 4 inputs. In the process, we also demonstrate that Lovász's proof of the capacity of the pentagon was implicitly contained (but unnoticed!) in the works of Jelinek and Gallager on the expurgated bound done at least ten years before Lovász. In the opposite direction, new upper bounds on the reliability function are derived for channels with an odd number of inputs by using an adaptation of Delsarte's linear programming bound. First, we derive a bound based on the minimum distance, which combines Lovász's construction for bounding the graph capacity with the McEliece-Rodemich-Rumsey-Welch construction for bounding the minimum distance of codes in the Hamming space. Then, for the particular case of cross-over probability 1/2, we derive an improved bound by also using the method of Kalai and Linial to study the spectrum distribution of codes.

This paper studies the basic question of whether a given channel V can be dominated (in the precise sense of being more noisy) by a q-ary symmetric channel. The concept of less noisy relation between channels originated in network information theory (broadcast channels) and is defined in terms of mutual information or Kullback-Leibler divergence. We provide an equivalent characterization in terms of χ2-divergence. Furthermore, we develop a simple criterion for domination by a q-ary symmetric channel in terms of the minimum entry of the stochastic matrix defining the channel V. The criterion is strengthened for the special case of additive noise channels over finite Abelian groups. Finally, it is shown that domination by a symmetric channel implies (via comparison of Dirichlet forms) a logarithmic Sobolev inequality for the original channel.

This paper addresses the problem of adding redundancy to a collection of physical objects so that the overall system is more robust to failures. In contrast to its information counterpart, which can exploit parity to protect multiple information symbols from a single erasure, physical redundancy can only be realized through duplication and substitution of objects. We propose a bipartite graph model for designing defect-tolerant systems, in which the defective objects are replaced by the judiciously connected redundant objects. The fundamental limits of this model are characterized under various asymptotic settings and both asymptotic and finite-size systems that approach these limits are constructed. Among other results, we show that the simple modular redundancy is in general suboptimal. As we develop, this combinatorial problem of defect tolerant system design has a natural interpretation as one of graph coloring, and the analysis is significantly different from that traditionally used in information redundancy for error-control codes.

The problem of population recovery refers to estimating a distribution based on incomplete or corrupted samples. Consider a random poll of sample size $n$ conducted on a population of individuals, where each pollee is asked to answer $d$ binary questions. We consider one of the two polling impediments: \beginitemize \item in lossy population recovery, a pollee may skip each question with probability $ε$; \item in noisy population recovery, a pollee may lie on each question with probability $ε$. \enditemize Given $n$ lossy or noisy samples, the goal is to estimate the probabilities of all $2^d$ binary vectors simultaneously within accuracy $δ$ with high probability. This paper settles the sample complexity of population recovery. For lossy model, the optimal sample complexity is $\tildeΘ(δ^ -2\max{\fracε1-ε,1})$, improving the state of the art by Moitra and Saks in several ways: a lower bound is established, the upper bound is improved and the result is dimension-free. Surprisingly, the sample complexity undergoes a phase transition from parametric to nonparametric rate when $ε$ exceeds $1/2$. For noisy population recovery, the sharp sample complexity turns out to be dimension-dependent and scales as $\exp(Θ(d^1/3 \log^2/3(1/δ)))$ except for the trivial cases of $ε=0,1/2$ or $1$. For both models, our estimators simply compute the empirical mean of a certain function, which is found by pre-solving a linear program (LP). Curiously, the dual LP can be understood as Le Cam’s method for lower-bounding the minimax risk, thus establishing the statistical optimality of the proposed estimators. The value of the LP is determined by complex-analytic methods.

Consider the family of all q-ary symmetric channels (q-SCs) with capacities decreasing from log(q) to 0. This paper addresses the following question: what is the member of this family with the smallest capacity that dominates a given channel V in the “less noisy” preorder sense. When the q-SCs are replaced by q-ary erasure channels, this question is known as the “strong data processing inequality.” We provide several equivalent characterizations of the less noisy preorder in terms of x2-divergence, Lowner (PSD) partial order, and spectral radius. We then illustrate a simple criterion for domination by a q-SC based on degradation, and mention special improvements for the case where V is an additive noise channel over an Abelian group of order q. Finally, as an application, we discuss how logarithmic Sobolev inequalities for q-SCs, which are well-studied, can be transported to an arbitrary channel V.

Let X and Y be dependent random variables. We consider the problem of designing a scalar quantizer for Y to maximize the mutual information between its output and X, and study fundamental properties and bounds for this form of quantization. Our main focus is the regime of low I(X; Y), where we show that for a binary X, there always exists an M-level quantizer attaining mutual information of Ω(-M · I(X;Y)/log(I(X;Y)) and that there exist pairs of X, Y for which the mutual information attained by any M-level quantizer is O(-M · I (X;Y)/ log (I(X;Y))).

We consider an uncoordinated Gaussian multiple access channel with a relatively large number of active users within each block. A low complexity coding scheme is proposed, which is based on a combination of compute-and-forward and coding for a binary adder channel. For a wide regime of parameters of practical interest, the energy-per-bit required by each user in the proposed scheme is significantly smaller than that required by popular solutions such as slotted-ALOHA and treating interference as noise.

This paper discusses the contemporary problem of providing multiple-access (MAC) to a massive number of uncoordinated users. First, we define a random-access code for Ka-user Gaussian MAC to be a collection of norm-constrained vectors such that the noisy sum of any Kaof them can be decoded with a given (suitably defined) probability of error. An achievability bound for such codes is proposed and compared against popular practical solutions: ALOHA, coded slotted ALOHA, CDMA, and treating interference as noise. It is found out that as the number of users increases existing solutions become vastly energy-inefficient. Second, we discuss the asymptotic (in blocklength) problem of coding for a K-user Gaussian MAC when K is proportional to blocklength and each user's payload is fixed. It is discovered that the energy-per-bit vs. spectral efficiency exhibits a rather curious tradeoff in this case.

In this paper, we consider a problem of sampling a Wiener process, with samples forwarded to a remote estimator via a channel that consists of a queue with random delay. The estimator reconstructs a real-time estimate of the signal from causally received samples. Motivated by recent research on age-of-information, we study the optimal sampling strategy that minimizes the mean square estimation error subject to a sampling frequency constraint. We prove that the optimal sampling strategy is a threshold policy, and find the optimal threshold. This threshold is determined by the sampling frequency constraint and how much the Wiener process varies during the channel delay. An interesting consequence is that even in the absence of the sampling frequency constraint, the optimal strategy is not zero-wait sampling in which a new sample is taken once the previous sample is delivered; rather, it is optimal to wait for a non-zero amount of time after the previous sample is delivered, and then take the next sample. Further, if the sampling times are independent of the observed Wiener process, the optimal sampling problem reduces to an age-of-information optimization problem that has been recently solved. Our comparisons show that the estimation error of the optimal sampling policy is much smaller than those of age-optimal sampling, zero-wait sampling, and classic uniform sampling.

This paper quantifies the fundamental limits of variable-length transmission of a general (possibly analog) source over a memoryless channel with noiseless feedback, under a distortion constraint. We consider excess distortion, average distortion, and guaranteed distortion (d-semifaithful codes). In contrast to the asymptotic fundamental limit, a general conclusion is that allowing variable-length codes and feedback leads to a sizable improvement in the fundamental delaydistortion tradeoff. In addition, we investigate the minimum energy required to reproduce k source samples with a given fidelity after transmission over a memoryless Gaussian channel, and we show that the required minimum energy is reduced with feedback and an average (rather than maximal) power constraint.

In this paper we consider a channel model that is often used to describe the mobile wireless scenario: multiple-antenna additive white Gaussian noise channels subject to random (fading) gain with full channel state information at the receiver. Dynamics of the fading process are approximated by a piecewise-constant process (frequency non-selective isotropic block fading). This work addresses the finite blocklength fundamental limits of this channel model. Specifically, we give a formula for the channel dispersion - a quantity governing the delay required to achieve capacity - and present achievability and (partial) converse bounds. Multiplicative nature of the fading disturbance leads to a number of interesting technical difficulties that required us to enhance traditional methods for finding channel dispersion. Knowledge of channel dispersion opens the possibility for studying the impact of channel dynamics, antenna selection rules, etc on the communication rate.

The capacity of a graph is defined as the rate of exponential growth of independent sets in the strong powers of the graph. In the strong power an edge connects two sequences if at each position their letters are equal or adjacent. We consider a variation of the problem where edges in the power graphs are removed between sequences which differ in more than a fraction δ of coordinates. The proposed generalization can be interpreted as the problem of determining the highest rate of zero undetected-error communication over a link with adversarial noise, where only a fraction δ of symbols can be perturbed and only some substitutions are allowed. We derive lower bounds on achievable rates by combining graph homomorphisms with a graph-theoretic generalization of the Gilbert-Varshamov bound. We then give an upper bound, based on Delsarte's linear programming approach, which combines Lovász' theta function with the construction used by McEliece et al. for bounding the minimum distance of codes in Hamming spaces.

We give new bounds on the reliability function of a typewriter channel with 5 inputs and crossover probability 1/2. The lower bound is more of theoretical than practical importance; it improves very marginally the expurgated bound, providing a counterexample to a conjecture on its tightness by Shannon, Gallager and Berlekamp which does not need the construction of algebraic-geometric codes previously used by Katsman, Tsfasman and Vlăduţ. The upper bound is derived by using an adaptation of the linear programming bound and it is essentially useful as a low-rate anchor for the straight line bound.

In this paper we present several results regarding distance preserving maps between nonbinary Hamming spaces and combinatorial (adversarial) joint source-channel coding. In an (α, β)-map from one Hamming space to another, any two sequences that are at least α relative distance apart, are mapped to sequences that are relative distance at least β apart. The motivation to study such maps come from (D, δ)-joint source-channel coding (JSCC) schemes, where any encoded sequence must be recovered within a relative distortion D, even in the presence of δ proportion of adversarial errors. We provide bounds on the parameters of both (α, β)-maps and (D,δ)-JSCC for nonbinary alphabets. We also provide constructive schemes for both, that are optimal for many cases.

It is shown that under suitable regularity conditions, differential entropy is O(√n)-Lipschitz as a function of probability distributions on ℝnwith respect to the quadratic Wasserstein distance. Under similar conditions, (discrete) Shannon entropy is shown to be O(n)-Lipschitz in distributions over the product space with respect to Ornstein's d̅-distance (Wasserstein distance corresponding to the Hamming distance). These results together with Talagrand's and Marton's transportation-information inequalities allow one to replace the unknown multi-user interference with its i.i.d. approximations. As an application, a new outer bound for the two-user Gaussian interference channel is proved, which, in particular, settles the “missing corner point” problem of Costa (1985).

This paper addresses the question of how to add redundancy to a collection of physical objects so that the overall system is more robust to failures. Physical redundancy can (generally) only be achieved by employing copy/substitute procedures. This is fundamentally different from information redundancy, where a single parity check simultaneously protects a large number of data bits against a single erasure. We propose a bipartite graph model of designing defect-tolerant systems where defective objects are repaired by reconnecting them to strategically placed redundant objects. The fundamental limits of this model are characterized under various asymptotic settings and both asymptotic and finite-size optimal systems are constructed. Mathematically, we say that a k by m bipartite graph corrects t defects over alphabet of size q if for every q-coloring of k left vertices there exists a coloring of m right vertices such that every left vertex is connected to at least t same-colored right vertices. We study the tradeoff between redundancy m/k and the total number of edges in the graph divided by k. The question is trivial when q ≥ k: the optimal solution is a simple t-fold replication. However, when q <; k some non-trivial savings are possible by leveraging the inherent repetition of colors.

A channel coding achievability bound expressed in terms of the ratio between two Neyman-Pearson β functions is proposed. This bound is the dual of a converse bound established earlier by Polyanskiy and Verdú (2014). The new bound turns out to simplify considerably the analysis in situations where the channel output distribution is not a product distribution, for example due to a cost constraint or a structural constraint (such as orthogonality or constant composition) on the channel inputs. Connections to existing bounds in the literature are discussed. The bound is then used to derive 1) the channel dispersion of additive non-Gaussian noise channels with random Gaussian codebooks, 2) the channel dispersion of an exponential-noise channel, 3) a second-order expansion for the minimum energy per bit of an additive white Gaussian noise channel, and 4) a lower bound on the maximum coding rate of a multiple-input multiple-output Rayleigh-fading channel with perfect channel state information at the receiver, which is the tightest known achievability result.

Motivated by the current interest in ultra-reliable, low-latency, machine-type communication systems, we investigate the tradeoff between reliability, throughput, and latency in the transmission of information over multiple-antenna Rayleigh block-fading channels. Specifically, we obtain finite-blocklength, finite-SNR upper and lower bounds on the maximum coding rate achievable over such channels for a given constraint on the packet error probability. Numerical evidence suggests that our bounds delimit tightly the maximum coding rate already for short blocklengths (packets of about 100 symbols). Furthermore, our bounds reveal the existence of a tradeoff between the rate gain obtainable by spreading each codeword over all available time-frequency-spatial degrees of freedom, and the rate loss caused by the need of estimating the fading coefficients over these degrees of freedom. In particular, our bounds allow us to determine the optimal number of transmit antennas and the optimal number of time-frequency diversity branches that maximize the rate. Finally, we show that infinite-blocklength performance metrics such as the ergodic capacity and the outage capacity yield inaccurate throughput estimates.

One of the basic tenets in information theory, the data processing inequality states that the output divergence does not exceed the input divergence for any channel. For channels without input constraints, various estimates on the amount of such contraction are known, Dobrushin's coefficient for the total variation being perhaps the most well-known. This paper investigates channels with an average input cost constraint. It is found that, while the contraction coefficient typically equals one (no contraction), the information nevertheless dissipates. A certain nonlinear function, the Dobrushin curve of the channel, is proposed to quantify the amount of dissipation. Tools for evaluating the Dobrushin curve of additive-noise channels are developed based on coupling arguments. Some basic applications in stochastic control, uniqueness of Gibbs measures, and fundamental limits of noisy circuits are discussed. As an application, it is shown that, in the chain of n power-constrained relays and Gaussian channels, the end-to-end mutual information and maximal squared correlation decay as O(log log n/log n), which is in stark contrast with the exponential decay in chains of discrete channels. Similarly, the behavior of noisy circuits (composed of gates with bounded fan-in) and broadcasting of information on trees (of bounded degree) does not experience threshold behavior in the signal-to-noise ratio (SNR). Namely, unlike the case of discrete channels, the probability of bit error stays bounded away from 1/2 regardless of the SNR.

It is shown that under suitable regularity conditions, differential entropy is O(√n)-Lipschitz as a function of probability distributions on Ilin with respect to the quadratic Wasserstein distance. Under similar conditions, (discrete) Shannon entropy is shown to be O(n)-Lipschitz in distributions over the product space with respect to Ornstein's d̅-distance (Wasserstein distance corresponding to the Hamming distance). These results together with Talagrand's and Marton's transportation-information inequalities allow one to replace the unknown multi-user interference with its independent identically distributed approximations. As an application, a new outer bound for the two-user Gaussian interference channel is proved, which, in particular, settles the missing corner point problem of Costa (1985).

Consider the problem of packing Hamming balls of a given relative radius subject to the constraint that they cover any point of the ambient Hamming space with multiplicity at most L. For odd L ≥ 3, an asymptotic upper bound on the rate of any such packing is proved. The resulting bound improves the best known bound (due to Blinovsky'1986) for rates below a certain threshold. The method is a superposition of the linear-programming idea of Ashikhmin, Barg, and Litsyn (that was previously used to improve the estimates of Blinovsky for L = 2) and a Ramsey-theoretic technique of Blinovsky. As an application, it is shown that for all odd L, the slope of the rate radius tradeoff is zero at zero rate.

This paper investigates the minimum energy required to transmit k information bits with a given reliability over a multiple-antenna Rayleigh block-fading channel, with and without channel state information (CSI) at the receiver. No feedback is assumed. It is well known that the ratio between the minimum energy per bit and the noise level converges to -1.59 dB as k goes to infinity, regardless of whether CSI is available at the receiver or not. This paper shows that the lack of CSI at the receiver causes a slowdown in the speed of convergence to -1.59 dB as k → ∞ compared with the case of perfect receiver CSI. Specifically, we show that, in the no-CSI case, the gap to -1.59 dB is proportional to ((log k)/k)1/3, whereas when perfect CSI is available at the receiver, this gap is proportional V to 1/√k. In both cases, the gap to -1.59 dB is independent of the number of transmit antennas and of the channel's coherence time. Numerically, we observe that, when the receiver is equipped with a single antenna, to achieve an energy per bit of -1.5 dB in the no-CSI case, one needs to transmit at least 7 × 107information bits, whereas 6 × 104bits suffice for the case of perfect CSI at the receiver.

With the boom of big data, traditional source coding techniques face the common obstacle to decode only a small portion of information efficiently. In this paper, we aim to resolve this difficulty by introducing a specific type of source coding scheme called locally decodable source coding (LDSC). Rigorously, LDSC is capable of recovering an arbitrary bit of the unencoded message from its encoded version, by only feeding a small number of the encoded message to the decoder, and we call the decoder t-local if only t encoded symbols are required.We consider both almost lossless (block error) and lossy (bit error) cases for LDSC. First, we show that using linear encoder and a decoder with bounded locality, the reliable compress rate can not be less than one. More importantly, we show that even with a general encoder and 2-local decoders (t = 2), the rate of LDSC is still one. On the contrary, the achievability bounds for almost lossless and lossy compressions with excess distortion suggest that optimal compression rate is achievable when O(log n) encoded symbols is queried by the decoder with block-length n. We also show that, rate distortion is achievable when the number of queries is scaled over n with a bound on the rate in finite-length regime. Although the achievability bounds are simply based on the concatenation of code blocks, they outperform the existing bounds in succinct data structures literature.

This work presents strong data processing results for the power-constrained additive Gaussian channel. Explicit bounds on the amount of decrease of mutual information under convolution with Gaussian noise are shown. The analysis leverages the connection between information and estimation (I-MMSE) and the following estimation-theoretic result of independent interest. It is proved that any random variable for which there exists an almost optimal (in terms of the mean-squared error) linear estimator operating on the Gaussian-corrupted measurement must necessarily be almost Gaussian (in terms of the Kolmogorov-Smirnov distance).

This paper quantifies the fundamental limits of variable-length transmission of a general (possibly analog) source over a memoryless channel with noiseless feedback, under a distortion constraint. We consider excess distortion, average distortion and guaranteed distortion (d-semifaithful codes). In contrast to the asymptotic fundamental limit, a general conclusion is that allowing variable-length codes and feedback leads to a sizable improvement in the fundamental delay-distortion tradeoff.

Consider the problem of packing Hamming balls of a given relative radius subject to the constraint that they cover any point of the ambient Hamming space with multiplicity at most L. For odd L ≥ 3 an asymptotic upper bound on the rate of any such packing is proven. The resulting bound improves the best known bound (due to Blinovsky' 1986) for rates below a certain threshold. The method is a superposition of the linear- programming idea of Ashikhmin, Barg and Litsyn (that was used previously to improve the estimates of Blinovsky for L = 2) and a Ramsey-theoretic technique of Blinovsky. As an application it is shown that for all odd L the slope of the rate-radius tradeoff is zero at zero rate.

This paper investigates the minimum energy required to transmit, with a given reliability, k information bits over a stationary memoryless Rayleigh-fading channel, under the assumption that neither the transmitter nor the receiver have a priori channel state information (CSI). It is well known that the ratio between the minimum energy per bit and the noise level converges to -1.59 dB as k goes to infinity, regardless of whether CSI is available at the receiver or not. This paper shows that lack of CSI at the receiver causes a slowdown in the speed of convergence to -1.59 dB as k → ∞ compared to the case of perfect receiver CSI. Specifically, we show that in the noCSI case, the gap to -1.59 dB is proportional to ((log k)/k)1/3, whereas when perfect CSI is available at the receiver, this gap is '/ proportional to 1/√(k). Numerically, we observe that to achieve an energy per bit of -1.5 dB in the no-CSI case, one needs to transmit at least 7 × 107information bits, whereas 6 × 104bits suffice for the case of perfect CSI at the receiver (same number of bits as for nonfading AWGN channels). Interestingly, all results (asymptotic and numerical) are unchanged if multiple transmit antennas and/or block fading is assumed.

This article continues the recent investigation of combinatorial joint source-channel coding. For the special case of a binary source and channel subject to distortion measured by Hamming distance, the lower (converse) bounds on achievable source distortion are improved for all values of channel noise. Operational duality between coding with bandwidth expansion factors ρ and 1 over ρ is established. Although the exact value of the asymptotic noise-distortion tradeoff curve is unknown (except at ρ = 1), some initial results on inter-relations between these curves for different values of ρ are shown and lead to statements about monotonicity and continuity in ρ.

We investigate the minimum transmitted energy required to reproduce k source samples with a given fidelity after transmission over a memoryless Gaussian channel. In particular, we analyze the reduction in transmitted energy that accrues thanks to the availability of noiseless feedback. Allowing a nonvanishing excess distortion probability ∈ boosts the asymptotic fundamental limit by a factor of 1-∈, with or without feedback. If feedback is available, achieving guaranteed distortion with finite average energy is possible.

This paper studies the fundamental limits of the minimum average length of lossless and lossy variable-length compression, allowing a nonzero error probability ε, for lossless compression. We give nonasymptotic bounds on the minimum average length in terms of Erokhin's rate-distortion function and we use those bounds to obtain a Gaussian approximation on the speed of approach to the limit, which is quite accurate for all but small blocklengths: (1 - ε)kH(S) - ((kV(S)/2π))1/2exp[-((Q-1(ε))2/2)], where Q-1(·) is the functional inverse of the standard Gaussian complementary cumulative distribution function, and V(S) is the source dispersion. A nonzero error probability thus not only reduces the asymptotically achievable rate by a factor of 1 - ε, but this asymptotic limit is approached from below, i.e, larger source dispersions and shorter blocklengths are beneficial. Variable-length lossy compression under an excess distortion constraint is shown to exhibit similar properties.

This paper investigates the maximal channel coding rate achievable at a given blocklength n and error probability ϵ, when the codewords are subjected to a long-term (i.e., averaged-over-all-codeword) power constraint. The second-order term in the large-n expansion of the maximal channel coding rate is characterized both for additive white Gaussian noise (AWGN) channels and for quasi-static fading channels with perfect channel state information available at both the transmitter and the receiver. It is shown that in both the cases, the second-order term is proportional to (n-1ln n)1/2. For the quasi-static fading case, this second-order term is achieved by truncated channel inversion, namely, by concatenating a dispersion-optimal code for an AWGN channel subject to a short-term power constraint, with a power controller that inverts the channel whenever the fading gain is above a certain threshold. Easy-to-evaluate approximations of the maximal channel coding rate are developed for both the AWGN and the quasi-static fading case.

This work addresses the question of finite block-length fundamental limits of coherently demodulated multi-antenna channels, subject to frequency non-selective isotropic fading. Specifically we present achievability bound for the channel dispersion - a quantity known to determine the delay required to achieve capacity. It is shown that a commonly used isotropic Gaussian input, which is only one of many possible capacity achieving distributions, is suboptimal. Optimal inputs minimizing channel dispersion turn out to include a family of modulation techniques known as orthogonal designs (in particular, Alamouti's scheme). For 8 transmit antennas numerical evaluation shows that up to 40% of additional penalty in delay is incurred by using isotropic codewords (compared to dispersion-optimal architecture exploiting transmit diversity).

We consider an opportunistic communication system in which a transmitter selects one of multiple channels over which to schedule a transmission, based on partial knowledge of the network state. We characterize a fundamental limit on the rate that channel state information must be conveyed to the transmitter in order to meet a constraint on expected throughput. This problem is modeled as a causal rate distortion optimization of a Markov source. We introduce a novel distortion metric capturing the impact of imperfect channel state information on throughput. We compute a closed-form expression for the causal information rate distortion function for the case of two channels, as well as an algorithmic upper bound on the causal rate distortion function. Finally, we characterize the gap between the causal information rate distortion and the causal entropic rate-distortion functions.

This paper studies the fundamental limits of the minimum average length of variable-length compression when a nonzero error probability ε is tolerated. We give non-asymptotic bounds on the minimum average length in terms of Erokhin's rate-distortion function and we use those bounds to obtain a Gaussian approximation on the speed of approach to the limit which is quite accurate for all but small blocklengths: equation where Q-1(·) is the functional inverse of the Q-function and V (S) is the source dispersion. A nonzero error probability thus not only reduces the asymptotically achievable rate by a factor of 1-ε, but also this asymptotic limit is approached from below, i.e. a larger source dispersion and shorter blocklengths are beneficial. Further, we show that variable-length lossy compression under excess distortion constraint also exhibits similar properties.

In information theory, structural system constraints are frequently described in the form of a directed acyclic graphical model (DAG). This paper addresses the question of classifying DAGs up to an isomorphism. By considering Gaussian densities, the question reduces to verifying equality of certain algebraic varieties. A question of computing equations for these varieties has been previously raised in the literature. Here it is shown that the most natural method adds spurious components with singular principal minors, proving a conjecture of Sullivant. This characterization is used to establish an algebraic criterion for isomorphism, and to provide a randomized algorithm for checking that criterion. Results are applied to produce a list of the isomorphism classes of tree models on 4 and 5 nodes.

Motivated by recent circuit designs for Flash ADCs with imperfect comparators, we investigate the problem of scalar quantization with noisy partition points, where the partition point locations are perturbed from the designated values by noise during the placement process. For this problem setting, we derive a high resolution approximation for mean square error, and analyze the optimal partition point density accordingly. Our results indicate that it is necessary to take the effect of noise into account in the design process. In particular, we derive the optimal partition point density when the input distribution is Gaussian or uniform, and show when noise variance exceeds a certain threshold, a peculiar phase transition occurs and the optimal point density degenerates into a delta function at the origin. These theoretical results allow to optimize the design of flash ADCs and gain 1 bit in resolution over existing designs.

This paper investigates the maximal channel coding rate achievable at a given blocklength n and error probability ε, when the codewords are subject to a long-term (i.e., averaged-over-all-codeword) power constraint. The second-order term in the large-n expansion of the maximal channel coding rate is characterized both for AWGN channels and for quasi-static fading channels with perfect channel state information at the transmitter and the receiver. It is shown that in both cases the second-order term is proportional to √(log n)/n.

This paper analyzes the channel dispersion of quasi-static multiple-input multiple-output fading channels with no channel state information at the transmitter. We show that the channel dispersion is zero under mild conditions on the fading distribution. The proof of our result is based on Stokes' theorem, which deals with the integration of differential forms on manifolds with boundary.

Consider a problem of forward error-correction for the additive white Gaussian noise (AWGN) channel. For finite blocklength codes, the backoff from the channel capacity is inversely proportional to the square root of the blocklength. In this paper, it is shown that the codes achieving this tradeoff must necessarily have peak-to-average power ratio (PAPR) proportional to logarithm of the blocklength. This is extended to codes approaching capacity slower, and to PAPR measured at the output of an orthogonal frequency division multiplexing modulator. As a by-product, the convergence of (Smith's) amplitude-constrained AWGN capacity to Shannon's classical formula is characterized in the regime of large amplitudes. This converse-type result builds upon recent contributions in the study of empirical output distributions of good channel codes.

This paper studies several properties of channel codes that approach the fundamental limits of a given (discrete or Gaussian) memoryless channel with a nonvanishing probability of error. The output distribution induced by an ϵ-capacity-achieving code is shown to be close in a strong sense to the capacity achieving output distribution. Relying on the concentration of measure (isoperimetry) property enjoyed by the latter, it is shown that regular (Lipschitz) functions of channel outputs can be precisely estimated and turn out to be essentially nonrandom and independent of the actual code. It is also shown that the output distribution of a good code and the capacity achieving one cannot be distinguished with exponential reliability. The random process produced at the output of the channel is shown to satisfy the asymptotic equipartition property.

This paper investigates the maximal achievable rate for a given blocklength and error probability over quasi-static multiple-input multiple-output fading channels, with and without channel state information at the transmitter and/or the receiver. The principal finding is that outage capacity, despite being an asymptotic quantity, is a sharp proxy for the finite-blocklength fundamental limits of slow-fading channels. Specifically, the channel dispersion is shown to be zero regardless of whether the fading realizations are available at both transmitter and receiver, at only one of them, or at neither of them. These results follow from analytically tractable converse and achievability bounds. Numerical evaluation of these bounds verifies that zero dispersion may indeed imply fast convergence to the outage capacity as the blocklength increases. In the example of a particular 1 × 2 single-input multiple-output Rician fading channel, the blocklength required to achieve 90% of capacity is about an order of magnitude smaller compared with the blocklength required for an AWGN channel with the same capacity. For this specific scenario, the coding/decoding schemes adopted in the LTE-Advanced standard are benchmarked against the finite-blocklength achievability and converse bounds.

The Chebyshev radius of a set in a metric space is defined to be the radius of the smallest ball containing the set. This quantity is closely related to the covering radius of the set and, in particular for Hamming set, is extensively studied in computational biology. This paper investigates some basic properties of radii of sets in n-dimensional Hamming space, provides a linear programing relaxation and gives tight bounds on the integrality gap. This results in a simple polynomial-time approximation algorithm that attains the performance of the best known such algorithms with shorter running time.

We investigate the maximal achievable rate for a given blocklength and error probability over quasi-static single-input multiple-output (SIMO) fading channels. Under mild conditions on the channel gains, it is shown that the channel dispersion is zero regardless of whether the fading realizations are available at the transmitter and/or the receiver. The result follows from computationally and analytically tractable converse and achievability bounds. Through numerical evaluation, we verify that, in some scenarios, zero dispersion indeed entails fast convergence to outage capacity as the blocklength increases. In the example of a particular 1×2 SIMO Rician channel, the blocklength required to achieve 90% of capacity is about an order of magnitude smaller compared to the blocklength required for an AWGN channel with the same capacity.

Recently, Tchamkerten and coworkers proposed a novel variation of the problem of joint synchronization and error correction. This paper considers a strengthened formulation that requires the decoder to estimate both the message and the location of the codeword exactly. Such a scheme allows for transmitting data bits in the synchronization phase of the communication, thereby improving bandwidth and energy efficiencies. It is shown that the capacity region remains unchanged under the exact synchronization requirement. Furthermore, asynchronous capacity can be achieved by universal (channel independent) codes. Comparisons with earlier results on another (delay compensated) definition of rate are made. The finite blocklength regime is investigated and it is demonstrated that even for moderate blocklengths, it is possible to construct capacity-achieving codes that tolerate exponential level of asynchronism and experience only a rather small loss in rate compared to the perfectly synchronized setting; in particular, the channel dispersion does not suffer any degradation due to asynchronism. For the binary symmetric channel, a translation (coset) of a good linear code is shown to achieve the capacity-synchronization tradeoff.

A minimax metaconverse has recently been proposed as a simultaneous generalization of a number of classical results and a tool for the nonasymptotic analysis. In this paper, it is shown that the order of optimizing the input and output distributions can be interchanged without affecting the bound. In the course of the proof, a number of auxiliary results of separate interest are obtained. In particular, it is shown that the optimization problem is convex and can be solved in many cases by the symmetry considerations. As a consequence, it is demonstrated that in the latter cases, the (multiletter) input distribution in information-spectrum (Verdú–Han) converse bound can be taken to be a (memoryless) product of single-letter ones. A tight converse for the binary erasure channel is rederived by computing the optimal (nonproduct) output distribution. For discrete memoryless channels, a conjecture of Poor and Verdú regarding the tightness of the information spectrum bound on the error exponents is resolved in the negative. Concept of the channel symmetry group is established and relations with the definitions of symmetry by Gallager and Dobrushin are investigated.

This paper introduces the problem of joint source-channel coding in the setup where channel errors are adversarial and the distortion is worst case. Unlike the situation in the case of stochastic source-channel model, the separation principle does not hold in adversarial setup. This surprising observation demonstrates that designing good distortion-correcting codes cannot be done by serially concatenating good covering codes with good error-correcting codes. The problem of the joint code design is addressed and some initial results are offered.

This paper investigates the best achievable performance by a hypothesis test satisfying a structural constraint: two functions are computed at two different terminals and the detector consists of a simple comparator verifying whether the functions agree. Such tests arise as part of study of fundamental limits of channel coding, but are also useful in other contexts. A simple expression for the Stein exponent is found and applied to showing a strong converse in the problem of multi-terminal hypothesis testing with rate constraints. Connections to the Gács-Körner common information and to spectral properties of conditional expectation operator are identified. Further tightening of results hinges on finding λ-blocks of minimal weight. Application of Delsarte's linear programming method to this problem is described.

This paper continues the investigation of the combinatorial formulation of the joint source-channel coding problem. In particular, the connections are drawn to error-reducing codes, isometric embeddings and list-decodable codes. The optimal performance for the repetition construction is derived and is shown to be achievable by low complexity Markov decoders. The compound variation of the problem is proposed and some initial results are put forward.

We study the maximal achievable rate R*(n, ∈) for a given block-length n and block error probability o over Rayleigh block-fading channels in the noncoherent setting and in the finite block-length regime. Our results show that for a given block-length and error probability, R*(n, ∈) is not monotonic in the channel's coherence time, but there exists a rate maximizing coherence time that optimally trades between diversity and cost of estimating the channel.

The backoff from capacity due to finite blocklength can be assessed accurately from the channel dispersion. This paper analyzes the dispersion of a single-user, scalar, coherent fading channel with additive Gaussian noise. We obtain a convenient two-term expression for the channel dispersion which shows that, unlike the capacity, it depends crucially on the dynamics of the fading process.

Channel dispersion plays a fundamental role in assessing the backoff from capacity due to finite blocklength. This paper analyzes the channel dispersion for a simple channel with memory: the Gilbert-Elliott communication model in which the crossover probability of a binary symmetric channel evolves as a binary symmetric Markov chain, with and without side information at the receiver about the channel state. With side information, dispersion is equal to the average of the dispersions of the individual binary symmetric channels plus a term that depends on the Markov chain dynamics, which do not affect the channel capacity. Without side information, dispersion is equal to the spectral density at zero of a certain stationary process, whose mean is the capacity. In addition, the finite blocklength behavior is analyzed in the non-ergodic case, in which the chain remains in the initial state forever.

The minimum achievable energy per bit over memoryless Gaussian channels has been previously addressed in the limit when the number of information bits goes to infinity, in which case it is known that the availability of noiseless feedback does not lower the minimum energy per bit, which is -1.59 dB below the noise level. This paper analyzes the behavior of the minimum energy per bit for memoryless Gaussian channels as a function ofk, the number of information bits. It is demonstrated that in this nonasymptotic regime, noiseless feedback leads to significantly better energy efficiency. In particular, without feedback achieving energy per bit of -1.57 dB requires coding over at leastk=106information bits, while we construct a feedback scheme that transmits a single information bit with energy -1.59 dB and zero error. We also show that unlesskis very small, approaching the minimal energy per bit does not require using the feedback link except to signal that transmission should stop.

Without feedback, the backoff from capacity due to non-asymptotic blocklength can be quite substantial for blocklengths and error probabilities of interest in many practical applications. In this paper, novel achievability bounds are used to demonstrate that in the non-asymptotic regime, the maximal achievable rate improves dramatically thanks to variable-length coding and feedback. For example, for the binary symmetric channel with capacity 1/2 the blocklength required to achieve 90% of the capacity is smaller than 200, compared to at least 3100 for the best fixed-blocklength code (even with noiseless feedback). Virtually all the advantages of noiseless feedback are shown to be achievable, even if the feedback link is used only to send a single signal informing the encoder to terminate the transmission (stop-feedback). It is demonstrated that the non-asymptotic behavior of the fundamental limit depends crucially on the particular model chosen for the “end-of-packet” control signal. Fixed-blocklength codes and related questions concerning communicating with a guaranteed delay are discussed, in which situation feedback is demonstrated to be almost useless even non-asymptotically.

The question of minimum achievable energy per bit over memoryless channels has been previously addressed in the limit of number of information bits going to infinity, in which case it is known that availability of noiseless feedback does not lower the minimum energy per bit. This paper analyzes the behavior of the minimum energy per bit for memoryless Gaussian channels as a function of the number of information bits. It is demonstrated that in this non-asymptotic regime, noiseless feedback leads to significantly better energy efficiency. A feedback coding scheme with zero probability of block error and finite energy per bit is constructed. For both achievability and converse, the feedback coding problem is reduced to a sequential hypothesis testing problem for Brownian motion.

Without feedback, the backoff from capacity due to non-asymptotic block length can be quite substantial for block lengths and error probabilities of interest in many practical applications. In this paper, novel achievability bounds are used to demonstrate that in the non-asymptotic regime, the maximal achievable rate improves dramatically thanks to variable-length coding with feedback. For example, for the binary symmetric channel with capacity 1/2 the blocklength required to achieve 90% of the capacity is smaller than 200, compared to at least 3100 for the best fixed-blocklength, non-feedback code. Virtually all the advantages of noiseless feedback are shown to be achievable with decision-feedback only. It is demonstrated that the non-asymptotic behavior of the fundamental limit depends crucially on the particular model chosen for the “end-of-packet” control signal.

This paper investigates the maximal channel coding rate achievable at a given blocklength and error probability. For general classes of channels new achievability and converse bounds are given, which are tighter than existing bounds for wide ranges of parameters of interest, and lead to tight approximations of the maximal achievable rate for blocklengthsnas short as 100. It is also shown analytically that the maximal rate achievable with error probability¿isclosely approximated by C - ¿(V/n) Q-1(¿) where C is the capacity, V is a characteristic of the channel referred to as channel dispersion , and Q is the complementary Gaussian cumulative distribution function.

The minimum block-length required to achieve a given rate and error probability can be easily and tightly approximated from two key channel parameters: the capacity and the channel dispersion. The channel dispersion gauges the variability of the channel relative to a deterministic bit pipe with the same capacity. This paper finds the dispersion of the additive white Gaussian noise (AWGN) channel, the parallel AWGN channel, and the Gaussian channel with non-white noise and intersymbol interference.

Channel dispersion plays a fundamental role in assessing the backoff from capacity due to finite blocklength. This paper analyzes the channel dispersion for a simple channel with memory: the Gilbert-Elliott communication model in which the crossover probability of a binary symmetric channel evolves as a binary symmetric Markov chain, with and without side information at the receiver about the channel state. With side information, although capacity is invariant to the chain dynamics, dispersion is shown to be the sum of two terms: due to the Markov chain dynamics and due to the the randomness in the error generation, respectively.

Three essentially different approaches to the constructive part of the channel coding theorem have been proposed by Shannon, Feinstein and Gallager, respectively, leading to upper bounds on the minimal error probability achievable with a given rate and blocklength. Here, new upper bounds are given on both average and maximal error probability, which are tighter than existing bounds for many ranges of blocklength and channel parameters of interest. Along with converse bounds, the new achievability bounds allow to approximate tightly the maximum rate achievable for a given blocklength and error probability for blocklengths as short as n = 200 for both the BSC and the BEC.