We study the amplitude-constrained additive white Gaussian noise channel. It is well known that the capacity-achieving input distribution for this channel is discrete and supported on finitely many points. The best known bounds show that the support size of the capacity-achieving distribution is lower-bounded by a term of order $A$ and upper-bounded by a term of order $A^2$, where $A$ denotes the amplitude constraint. It was conjectured in [1] that the linear scaling is optimal. In this work, we establish a new lower bound of order $A\sqrt{\log A}$, improving the known bound and ruling out the conjectured linear scaling. To obtain this result, we quantify the fact that the capacity-achieving output distribution is close to the uniform distribution in the interior of the amplitude constraint. Next, we introduce a wrapping operation that maps the problem to a compact domain and develop a theory of best approximation of the uniform distribution by finite Gaussian mixtures. These approximation bounds are then combined with stability properties of capacity-achieving distributions to yield the final support-size lower bound.

We study the amplitude-constrained additive white Gaussian noise channel. It is well known that the capacity-achieving input distribution for this channel is discrete and supported on finitely many points. The best known bounds show that the support size of the capacity-achieving distribution is lower-bounded by a term of orderAand upper-bounded by a term of orderA2, whereAdenotes the amplitude constraint. It was conjectured in [2] that the linear scaling is optimal. In this work, we establish a new lower bound of orderA√ logA, improving the known bound and ruling out the conjectured linear scaling. To obtain this result, we quantify the fact that the capacity-achieving output distribution is close to the uniform distribution in relative entropy. Next, we introduce a wrapping operation that maps the problem to a compact domain and develop a theory of best approximation of the uniform distribution by finite Gaussian mixtures. These approximation bounds are then combined with stability properties of capacity-achieving distributions to yield the final support-size lower bound.

This work establishes regularity conditions for consistency and asymptotic normality of the multiple parameter maximum likelihood estimator (MLE) from censored data, where the censoring mechanism is in the form of 1-bit measurements. The underlying distribution of the uncensored data is assumed to belong to the exponential family, with natural parameters expressed as a linear combination of the predictors, known as generalized linear model (GLM). As part of the analysis, the Fisher information matrix is also derived for both censored and uncensored data, which helps to quantify the impact of censoring and assess the performance of the MLE. The choice of a GLM allows one to consider a variety of practical examples where 1-bit estimation is of interest. In particular, it is shown how the derived results can be used to analyze two practically relevant scenarios: the Gaussian model with both unknown mean and variance, and the Poisson model with an unknown mean.

We study prior distributions for Poisson parameter estimation under$L^{1}$loss. Specifically, we construct a new family of prior distributions whose optimal Bayesian estimators (the conditional medians) can be any prescribed increasing function that satisfies certain regularity conditions. In the case of affine estimators, this family is distinct from the usual conjugate priors, which are gamma distributions. Our prior distributions are constructed through a limiting process that matches certain moment conditions. These results provide the first explicit description of a family of distributions, beyond the conjugate priors, that satisfy the affine conditional median property; and more broadly for the Poisson noise model they can give any arbitrarily prescribed conditional median.

In this paper, we examine the distribution and convergence properties of the estimation error $W = X - \hat X(Y)$, where $\hat X(Y)$ is the Bayesian estimator of a random variable X from a noisy observation Y = X + σZ where σ is the parameter indicating the strength of noise Z. Using the conditional expectation framework (that is, $\hat X(Y)$ is the conditional mean), we define the normalized error ${{\mathcal{E}}_\sigma } = \frac{W}{\sigma }$ and explore its properties.Specifically, in the first part of the paper, we characterize the probability density function of W and ${{\mathcal{E}}_\sigma }$. Along the way, we also find conditions for the existence of the inverse functions for the conditional expectations. In the second part, we study pointwise (i.e., almost sure) convergence of ${{\mathcal{E}}_\sigma }$ as σ → 0 under various assumptions about the noise and the underlying distributions. Our results extend some of the previous limits of ${{\mathcal{E}}_\sigma }$ as σ → 0 studied under the L2convergence, known as the MMSE dimension, to the pointwise case.

This paper explores Bayesian lower bounds on the minimum mean squared error (MMSE) that belong to the well-known Ziv-Zakai family. The Ziv-Zakai technique relies on connecting the bound to an$\mathsf M$-ary hypothesis testing problem. There are three versions of the Ziv-Zakai bound (ZZB): the first version relies on the so-calledvalley-filling function, the second one is a relaxation of the first bound which omits the valley-filling function, and the third one, namely the single-point ZZB (SZZB), replaces the integration present in the first two bounds with a single point maximization. The first part of this paper focuses on providing the most general version of the bounds. It is shown that these bounds hold without any assumption on the distribution of the estimand. This makes the bounds applicable to discrete and mixed distributions. Then, the SZZB is extended to an$\mathsf M$-ary setting and a version of it that holds for the multivariate setting is provided. In the second part, general properties of these bounds are provided. First, unlike the BayesianCramér-Rao bound, it is shown that all the versions of the ZZBtensorize. Second, a characterization of thehigh-noiseasymptotic is provided, which is used to argue about the tightness of the bounds. Third, a completelow-noiseasymptotic is provided under the assumptions of mixed-input distributions and Gaussian additive noise channels. In the low-noise, it is shown that the ZZB is generally tight, but there are examples for which the SZZB is not tight. In the third part, the tightness of the bounds is evaluated. First, it is shown that in the low-noise regime the ZZB without the valley-filling function, and, therefore, also the ZZB with the valley-filling function, are tight for mixed-input distributions and Gaussian additive noise channels. Second, for discrete inputs it is shown that the ZZB with the valley-filling function is always sub-optimal, and equal to zero without the valley-filling function. Third, unlike for the ZZB, an example is shown for which the SZZB is tight to the MMSE for discrete inputs. Fourth, sufficient and necessary conditions for the tightness of the bounds are provided. Finally, some examples are provided in which the bounds in the Ziv-Zakai family outperform other well-known Bayesian lower bounds, namely the Cramér-Rao bound and the maximum entropy bound.

The capacity-achieving input distribution of non-coherent Rayleigh fading channels with average- and peak-power constraints is known to be discrete with a finite number of points. We sharpen this result by deriving upper and lower bounds on the number of amplitude levels. The upper bounds are based on two techniques from complex analysis: counting the number of maxima of a function that characterizes the Karush-Kuhn-Tucker conditions and an oscillation theorem. The latter provides a stronger bound but applies only if the average power constraint is inactive.

This work considers a discrete-time Poisson noise channel with an input amplitude constraint A and a dark current parameter A. It is known that the capacity-achieving distribution for this channel is discrete with finitely many points. Recently, for$A$= 0, a lower bound of order ✓A and an upper bound of order A log2 (A) have been demonstrated on the cardinality of the support of the optimal input distribution. In this work, we improve these results in several ways. First, we provide upper and lower bounds that hold for non-zero dark current. Second, we produce a sharper upper bound with a far simpler technique. In particular, for$A$= 0, we sharpen the upper bound from the order of A log2 (A) to the order of A. Finally, some other additional information about the location of the support is provided.

This work considers a binomial noise channel. The paper can be roughly divided into two parts. The first part is concerned with the properties of the capacity-achieving distribution. In particular, for the binomial channel, it is not known if the capacity-achieving distribution is unique since the output space is finite (i.e., supported on integers 0, …, n) and the input space is infinite (i.e., supported on the interval [0, 1 D, and there are multiple distributions that induce the same output distribution. This paper shows that the capacity-achieving input distribution is unique by appealing to the total positivity property of the binomial kernel. In addition, we provide upper and lower bounds on the cardinality of the support of the capacity-achieving distribution. Specifically, an upper bound of order n/2 is shown, which improves on the previous upper bound of order$n$due to Witsenhausen. Moreover, a lower bound of order yin is shown. Finally, additional results about the locations and probability values of the support points are established.

Consider the task of estimating a random vector$X$from noisy observations$Y=X+Z$, where$Z$is a standard normal vector, under the$L^{p}$fidelity criterion. This work establishes that, for$1\leq p\leq 2$, the optimal Bayesian estimator is linear and positive definite if and only if the prior distribution on$X$is a (non-degenerate) multivariate Gaussian. Furthermore, for$p > 2$, it is demonstrated that there are infinitely many priors that can induce such an estimator.

A multi-input multi-output (MIMO) Gaussian chan-nel with two transmit antennas and two receive antennas is studied that is subject to an input peak-power constraint. The ca-pacity and the capacity-achieving input distribution are unknown in general. The problem is shown to be equivalent to a channel with an identity matrix but where the input lies inside and on an ellipse with principal axis length$r_{p}$and minor axis length$r_{m}$. If$r_{p}\leq\sqrt{2}$, then the capacity-achieving input has support on the ellipse. A sufficient condition is derived under which a two-point distribution is optimal. Finally, if$r_{m} < r_{p}\leq\sqrt{2}$, then the capacity-achieving distribution is discrete.

This paper investigates the rate-distortion function, under a squared error distortion$D$, for an n-dimensional random vector uniformly distributed on an$(n-1)$-sphere of radius$R$. First, an expression for the rate-distortion function is derived for any values of$n, D$, and$R$. Second, two types of asymptotics with respect to the rate-distortion function of a Gaussian source are characterized. More specifically, these asymptotics concern the low-distortion regime (that is,$D\rightarrow 0$) and the high-dimensional regime (that is,$n\rightarrow\infty$).

Classification is a fundamental task in many applications on which data-driven methods have shown outstanding performances. However, it is challenging to determine whether such methods have achieved the optimal performance. This is mainly because the best achievable performance is typically unknown and hence, effectively estimating it is of prime importance. In this paper, we consider binary classification problems and we propose an estimator for the false positive rate (FPR) of the Bayes classifier, that is, the optimal classifier with respect to accuracy, from a given dataset. Our method utilizes soft labels, or real-valued labels, which are gaining significant traction thanks to their properties. We thoroughly examine various theoretical properties of our estimator, including its consistency, unbiasedness, rate of convergence, and variance. To enhance the versatility of our estimator beyond soft labels, we also consider noisy labels, which encompass binary labels. For noisy labels, we develop effective FPR estimators by leveraging a denoising technique and the Nadaraya-Watson estimator. Due to the symmetry of the problem, our results can be readily applied to estimate the false negative rate of the Bayes classifier.

This work studies non-coherent Rayleigh fading channels subject to average- and peak-power constraints. Several properties of the optimal input distribution are derived based on the Karush-Kuhn- Tucker conditions. In particular, the capacity-achieving distribution is characterized in the small peak and average power regimes, upper and lower bounds on the optimal input probabilities are presented, insights about the locations of the support points are provided, and bounds on the channel capacity are established.

Consider the problem of estimating a random variable X from noisy observations$Y = X+ Z$, where Z is standard normal, under the$L^{1}$fidelity criterion. It is well known that the optimal Bayesian estimator in this setting is the conditional median. This work shows that the only prior distribution on X that induces linearity in the conditional median is Gaussian. Along the way, several other results are presented. In particular, it is demonstrated that if the conditional distribution$P_{X|Y=y}$is symmetric for all y, then X must follow a Gaussian distribution. Additionally, we consider other$L^{p}$losses and observe the following phenomenon: for$p \in [{1,2}]$, Gaussian is the only prior distribution that induces a linear optimal Bayesian estimator, and for$p \in (2,\infty)$, infinitely many prior distributions on X can induce linearity. Finally, extensions are provided to encompass noise models leading to conditional distributions from certain exponential families.

This paper studies the problem of data permutation recovery, where the goal is to estimate the ordering of an$n$-dimensional data vector given a noisy observation of it. The focus is on scenarios where the noise is additive Gaussian with an arbitrary known covariance matrix. The goal is to characterize the probability of error and its behavior when a linear decoder (that is, a linear estimator followed by a sorting operation) is employed. First, a general expression is derived for the probability of error when a linear decoder is used. The derived expression holds for any continuous distribution of the input data vector, and when the noise has memory. Then, the rates of convergence of the probability of error in the low-noise and high-noise regimes are investigated when a simple linear decoder is used. It is shown that in the low-noise regime, the probability of error can quadratically increase with$n$, and in the high-noise regime it behaves as$1-1/n!$for several distributions of interest. Finally, upper and lower bounds on the probability of correctness with respect to$n$are derived for the case of an i.i.d. data distribution and show that the rate of convergence is at least exponential in$n$. The results showcase that the permutation recovery problem is noise dominated, which motivates the study of more relaxed versions of the permutation recovery problem that are also discussed.

In this paper, we investigate the capacity of the$b- \mathbf{modulated}$nonlinear frequency division multiplexing (NFDM) systems. Recently, a tractable channel model was proposed for such optical fiber communication systems, describing the received$b-\mathbf{Modulated}$signal by an input-dependent complex Gaussian distributed noise. Considering this channel model, we prove that the capacity achieving input distribution has discrete amplitude, uniform independent phase (DAUIP). Noting that this distribution is supported on a finite number of concentric shells, we find a lower bound for the capacity by assuming a single shell support. The corresponding output distribution and mutual information expressions are derived in closed-form.

This work considers a co-located MIMO radar with MTtransmitting and MRreceiving antennas in a so-called massive MIMO regime, that is, where the number of virtual spatial antennas N = MTMRis large. Recently, it has been demonstrated that as N grows to infinity, one can fully characterize the false alarm and detection probabilities with very minimal assumptions on the disturbance vector. In this work, these results are partially refined and a lower bound on the probability of detection is provided for any fixed, finite N under certain randomness models for the noise. This result can serve as a rule of thumb for the design of massive MIMO radar systems by indicating the number of antennas required to attain a desired target detection probability.

Consider the problem of estimating a random variable X in Gaussian noise under L1fidelity criteria. It is well-known that in the L1setting, the optimal Bayesian estimator is given by the conditional median. The goal of this work is to characterize the set of prior distributions on X for which the conditional median corresponds to a linear estimator. This work shows that neither discrete nor compactly supported distributions can induce a linear conditional median. Moreover, under certain non-trivial restrictions on the set of allowed probability distributions, the Gaussian is shown to be the only solution that induces a linear conditional median.

This paper explores the Ziv-Zakai bound (ZZB), which is a well-known Bayesian lower bound on the Minimum Mean Squared Error (MMSE). First, it is shown that the ZZB holds without any assumption on the distribution of the estimand, that is, the estimand does not necessarily need to have a probability density function. The ZZB is then further analyzed in the high-noise and low-noise regimes and shown to always tensorize. Finally, the tightness of the ZZB is investigated under several aspects, such as the number of hypotheses and the usefulness of the valley-filling function. In particular, a sufficient and necessary condition for the tightness of the bound with continuous inputs is provided, and it is shown that the bound is never tight for discrete input distributions with a support set that does not have an accumulation point at zero.

Classification is a fundamental task in science and engineering on which machine learning methods have shown outstanding performances. However, it is challenging to determine whether such methods have achieved the Bayes error rate, that is, the lowest error rate attained by any classifier. This is mainly due to the fact that the Bayes error rate is not known in general and hence, effectively estimating it is paramount. Inspired by the work by Ishida et al. (2023), we propose an estimator for the Bayes error rate of supervised multi-class classification problems. We analyze several theoretical aspects of such estimator, including its consistency, unbiasedness, convergence rate, variance, and robustness. We also propose a denoising method that reduces the noise that potentially corrupts the data labels, and we improve the robustness of the proposed estimator to outliers by incorporating the median-of-means estimator. Our analysis demonstrates the consistency, asymptotic unbiasedness, convergence rate, and robustness of the proposed estimators. Finally, we validate the effectiveness of our theoretical results via experiments both on synthetic data under various noise settings and on real data.

It is well known that central order statistics exhibit a central limit behavior and converge to a Gaussian distribution as the sample size grows. This paper strengthens this known result by establishing an entropic version of the central limit theorem that ensures a stronger mode of convergence using the relative entropy. This upgrade in convergence is shown at the expense of extra regularity conditions, which can be considered as mild. To prove this result, ancillary results on order statistics are derived, which might be of independent interest. For instance, a rather general bound on the moments of order statistics, and an upper bound on the mean squared error of estimating the$p \in (0,1)$-th quantile of an unknown cumulative distribution function, are derived. Finally, a discussion on the necessity of the derived conditions for convergence and on the rate of convergence and monotonicity of the relative entropy is provided.

Consider a pair of random vectors$( {\mathbf{X}}, {\mathbf{Y}}) $and the conditional expectation operator$ \mathbb {E}[ {\mathbf{X}}| {\mathbf{Y}}={\mathbf{y}}]$. This work studies analytical properties of the conditional expectation by characterizing various derivative identities. The paper consists of two parts. In the first part of the paper, a general derivative identity for the conditional expectation is derived. Specifically, for the Markov chain$ {\mathbf{U}}\leftrightarrow {\mathbf{X}}\leftrightarrow {\mathbf{Y}}$, a compact expression for the Jacobian matrix of$ \mathbb {E}[ \psi ( {\mathbf{Y}}, {\mathbf{U}})| {\mathbf{Y}}= {\mathbf{y}}]$for a smooth function$\psi $is derived. In the second part of the paper, the main identity is specialized to the exponential family and two main applications are shown. First, it is demonstrated that, via various choices of the random vector$ {\mathbf{U}}$and function$\psi $, one can recover and generalize several known identities (e.g., Tweedie’s formula) and derive some new ones. For example, a new relationship between conditional expectations and conditional cumulants is established. Second, it is demonstrated how the derivative identities can be used to establish new lower bounds on the estimation error. More specifically, using one of the derivative identities in conjunction with a Poincaré inequality, a new lower bound on the minimum mean squared error, which holds for all prior distributions on the input signal, is derived. The new lower bound is shown to be tight in the high-noise regime for the additive Gaussian noise setting.

Consider a channel$\mathbf {Y}= \mathbf {X}+ \mathbf {N}$where$\mathbf {X}$is an$n$-dimensional random vector, and$\mathbf {N}$is a multivariate Gaussian vector with a full-rank covariance matrix$\boldsymbol {\mathsf {K}}_{ \mathbf {N}}$. The object under consideration in this paper is the conditional mean of$\mathbf {X}$given$\mathbf {Y}={\mathbf{y}}$, that is${\mathbf{y}} \mapsto \mathbb {E} [\mathbf {X}| \mathbf {Y}={\mathbf{y}}]$. Several identities in the literature connect$\mathbb {E}[\mathbf {X}| \mathbf {Y}={\mathbf{y}}]$to other quantities such as the conditional variance, score functions, and higher-order conditional moments. The objective of this paper is to provide a unifying view of these identities. In the first part of the paper, a general derivative identity for the conditional mean estimator is derived. Specifically, for the Markov chain$\mathbf {U}\leftrightarrow \mathbf {X}\leftrightarrow \mathbf {Y}$, it is shown that the Jacobian matrix of$\mathbb {E}[\mathbf {U}| \mathbf {Y}={\mathbf{y}}]$is given by$\boldsymbol {\mathsf {K}}_{ \mathbf {N}}^{-1} \boldsymbol {\mathsf {Cov}} (\mathbf {X}, \mathbf {U}| \mathbf {Y}={\mathbf{y}})$where$\boldsymbol {\mathsf {Cov}} (\mathbf {X}, \mathbf {U}| \mathbf {Y}={\mathbf{y}})$is the conditional covariance. In the second part of the paper, via various choices of the random vector$\mathbf {U}$, the new identity is used to recover and generalize many of the known identities and derive some new identities. First, a simple proof of the Hatsel and Nolte identity for the conditional variance is shown. Second, a simple proof of the recursive identity due to Jaffer is provided. The Jaffer identity is then further explored, and several equivalent statements are derived, such as an identity for the higher-order conditional expectation (i.e.,$\mathbb {E}[\mathbf {X}^{k}| \mathbf {Y}]$) in terms of the derivatives of the conditional expectation. Third, a new fundamental connection between the conditional cumulants and the conditional expectation is demonstrated. In particular, in the univariate case, it is shown that the$k$-th derivative of the conditional expectation is proportional to the$(k+1)$-th conditional cumulant. A similar expression is derived in the multivariate case.

A common way of characterizing minimax estimators in point estimation is by moving the problem into the Bayesian estimation domain and finding a least favorable prior distribution. The Bayesian estimator induced by a least favorable prior, under mild conditions, is then known to be minimax. However, finding least favorable distributions can be challenging due to inherent optimization over the space of probability distributions, which is infinite-dimensional. This paper develops a dimensionality reduction method that allows us to move the optimization to a finite-dimensional setting with an explicit bound on the dimension. The benefit of this dimensionality reduction is that it permits the use of popular algorithms such as projected gradient ascent to find least favorable priors. Throughout the paper, in order to make progress on the problem, we restrict ourselves to Bayesian risks induced by a relatively large class of loss functions, namely Bregman divergences.

This paper considers a discrete time-Poisson noise channel which is used to model pulse-amplitude modulated optical communication with a direct-detection receiver. The goal of this paper is to obtain insights into the capacity and the structure of the capacity-achieving distribution for the channel under the amplitude constraint A and in the presence of dark current λ. Using recent theoretical progress on the structure of the capacity-achieving distribution, this paper develops a numerical algorithm, based on the gradient ascent and Blahut-Arimoto algorithms, for computing the capacity and the capacity-achieving distribution. The algorithm is used to perform extensive numerical simulations for various regimes of A and λ.

We consider information-theoretic bounds on expected generalization error for statistical learning problems in a networked setting. In this setting, there are K nodes, each with its own independent dataset, and the models from each node have to be aggregated into a final centralized model. We consider both simple averaging of the models as well as more complicated multi-round algorithms. We give upper bounds on the expected generalization error for a variety of problems, such as those with Bregman divergence or Lipschitz continuous losses, that demonstrate an improved dependence of 1/K on the number of nodes. These "per node" bounds are in terms of the mutual information between the training dataset and the trained weights at each node, and are therefore useful in describing the generalization properties inherent to having communication or privacy constraints at each node.

It is well known that central order statistics exhibit a central limit behavior and converge to a Gaussian distribution as the sample size n grows. This paper strengthens this known result by establishing an entropic version of the central limit theorem (CLT) that ensures a stronger mode of convergence using the relative entropy. In particular, an order $O(1/\sqrt n )$ rate of convergence is established under mild conditions on the parent distribution of the sample generating the order statistics. To prove this result, ancillary results on order statistics are derived, which might be of independent interest.

This paper studies secrecy-capacity of an n-dimensional Gaussian wiretap channel under the peak-power constraint. This work determines the largest peak-power constraint ${\overline {\text{R}} _n}$ such that an input distribution uniformly distributed on a single sphere is optimal; this regime is termed the low amplitude regime. The asymptotic of ${\overline {\text{R}} _n}$ as n goes to infinity is completely characterized as a function of noise variance at both receivers. Moreover, the secrecy-capacity is also characterized in a form amenable for computation. Furthermore, several numerical examples are provided, such as the example of the secrecy-capacity achieving distribution outside of the low amplitude regime.

This paper considers the problem of recovering the ranking of a data vector from noisy observations, up to a distortion. Specifically, the noisy observations consist of the original data vector corrupted by isotropic additive Gaussian noise, and the distortion is measured in terms of a distance function between the estimated ranking and the true ranking of the original data vector. First, it is shown that an optimal (in terms of error probability) decision rule for the estimation task simply outputs the ranking of the noisy observation. Then, the error probability incurred by such a decision rule is characterized in the low-noise regime, and shown to grow sublinearly with the noise standard deviation. This result highlights that the proposed approximate version of the ranking recovery problem is significantly less noise-dominated than the exact recovery considered in [Jeong, ISIT 2021].

This paper studies the minimum mean squared error (MMSE) of estimating X ∈ ℝdfrom the noisy observation Y ∈ ℝk, under the assumption that the noise (i.e., Y|X) is a member of the exponential family. The paper provides a new lower bound on the MMSE. Towards this end, an alternative representation of the MMSE is first presented, which is argued to be useful in deriving closed-form expressions for the MMSE. This new representation is then used together with the Poincaré inequality to provide a new lower bound on the MMSE. Unlike, for example, the Cramér-Rao bound, the new bound holds for all possible distributions on the input X. Moreover, the lower bound is shown to be tight in the high-noise regime for the Gaussian noise setting under the assumption that X is sub-Gaussian. Finally, several numerical examples are shown which demonstrate that the bound performs well in all noise regimes.

The Ziv-Zakai bound is a well-known lower bound on the minimum mean squared error. This article analyzes the performance of this bound in the practically relevant high-noise regime for a broad family of observation models. The goal is to understand whether this bound is tight, and in which scenarios it should be used. It is shown that, while the Ziv-Zakai bound is tight for a certain class of symmetric distributions, in general, it isnottight in the high-noise regime.

A general class of Bayesian lower bounds when the underlying loss function is a Bregman divergence is demonstrated. This class can be considered as an extension of the Weinstein–Weiss family of bounds for the mean squared error and relies on finding a variational characterization of Bayesian risk. This approach allows for the derivation of a version of the Cramér–Rao bound that is specific to a given Bregman divergence. This new generalization of the Cramér–Rao bound reduces to the classical one when the loss function is taken to be the Euclidean norm. In order to evaluate the effectiveness of the new lower bounds, the paper also develops upper bounds on Bayesian risk, which are based on optimal linear estimators. The effectiveness of the new bound is evaluated in the Poisson noise setting.

This paper considers the problem of recovering the permutation of an n-dimensional random vector X observed in Gaussian noise. First, a general expression for the probability of error is derived when a linear decoder (i.e., linear estimator followed by a sorting operation) is used. The derived expression holds with minimal assumptions on the distribution of X and when the noise has memory. Second, for the case of isotropic noise (i.e., noise with a diagonal scalar covariance matrix), the rates of convergence of the probability of error are characterized in the high and low noise regimes. In the low noise regime, for every dimension$n$, the probability of error is shown to behave proportionally to$\sigma$, where$\sigma$is the noise standard deviation. Moreover, the slope is computed exactly for several distributions and it is shown to behave quadratically in$n$. In the high noise regime, for every dimension$n$, the probability of correctness is shown to behave as$1/\sigma$, and the exact expression for the rate of convergence is also provided.

This work studies the secrecy-capacity of a scalar-Gaussian wiretap channel with an amplitude constraint on the input. It is known that for this channel, the secrecy-capacity-achieving distribution is discrete with finitely many points. This work improves such result by showing an upper bound of the order $\frac{A^{2}}{\sigma_{1}^{2}}$ where A is the amplitude constraint and $\sigma_{1}^{2}$ is the variance of the Gaussian noise over the legitimate channel.

Consider a pair of random vectors $(\mathrm{X}, \mathrm{Y})$ and the conditional expectation operator $\mathbb{E}[\mathrm{X} \mid \mathrm{Y}=\mathrm{y}]$. This work studies analytic properties of the conditional expectation by characterizing various derivative identities. The paper consists of two parts. In the first part of the paper, a general derivative identity for the conditional expectation is derived. Specifically, for the Markov chain $\mathrm{U} \leftrightarrow \mathrm{X} \leftrightarrow \mathrm{Y}$, a compact expression for the Jacobian matrix of $\mathbb{E}[\mathrm{U} \mid \mathrm{Y}=\mathrm{y}]$ is derived. In the second part of the paper, the main identity is specialized to the exponential family. Moreover, via various choices of the random vector U, the new identity is used to recover and generalize several known identities and derive some new ones. As a first example, a connection between the Jacobian of $\mathbb{E}[\mathrm{X} \mid \mathrm{Y}=\mathrm{y}]$ and the conditional variance is established. As a second example, a recursive expression between higher order conditional expectations is found, which is shown to lead to a generalization of the Tweedy’s identity. Finally, as a third example, it is shown that the k-th order derivative of the conditional expectation is proportional to the $(k+1)$-th order conditional cumulant.

This work considers a Poisson noise channel with an amplitude constraint. It is well-known that the capacity-achieving input distribution for this channel is discrete with finitely many points. We sharpen this result by introducing upper and lower bounds on the number of mass points. Concretely, an upper bound of order$\mathsf {A}\log ^{2}(\mathsf {A})$and a lower bound of order$\sqrt { \mathsf {A}}$are established where$\mathsf {A}$is the constraint on the input amplitude. In addition, along the way, we show several other properties of the capacity and capacity-achieving distribution. For example, it is shown that the capacity is equal to$- \log P_{Y^\star }(0)$where$P_{Y^\star }$is the optimal output distribution. Moreover, an upper bound on the values of the probability masses of the capacity-achieving distribution and a lower bound on the probability of the largest mass point are established. Furthermore, on the per-symbol basis, a nonvanishing lower bound on the probability of error for detecting the capacity-achieving distribution is established under the maximum a posteriori rule.

This work outlines a method for an application of empirical Bayes in the setting of semi-supervised learning. That is, we consider a scenario in which the training set is partially or entirely unlabeled. In addition to the missing labels, we also consider a scenario where the available training data might be shuffled (i.e., the features and labels are not matched).Specifically, we propose to train model-based empirical Bayes separately on the set of features and the set of labels and combine/mix the two models based on the proportion of unlabeled pairs. The method then can be used to recover the missing labels (i.e., create pseudo-labels) of the data set and, in addition, if the data is shuffled, recover the correct permutation of the data. The technique is evaluated for a multivariate Gaussian model and is shown to consistently outperform a maximum likelihood approach. Moreover, the procedure is shown to be a consistent estimator for a multivariate Gaussian model with an arbitrary (non-degenerate) covariance matrix.

This paper considers a problem of estimation of the Fisher information for location from a random sample of size n. First, an estimator proposed by Bhattacharya is revisited and improved convergence rates are derived. Second, a new estimator, termed clipped estimator, is proposed. The new estimator is shown to have superior rates of convergence as compared to the Bhattacharya estimator, albeit with different regularity conditions. Third, both of the estimators are evaluated for the practically relevant case of a random variable contaminated by Gaussian noise. Moreover, using Brown's identity, which relates the Fisher information to the minimum mean squared error (MMSE) in Gaussian noise, a consistent estimator for the MMSE is proposed.

This paper provides a general derivative identity for the conditional mean estimator of an arbitrary vector signal in Gaussian noise with an arbitrary covariance matrix. This new identity is used to recover and generalize many known identities in the literature and derive some new identities. For example, a new identity is discovered, which shows that an arbitrary higher-order conditional moment is completely determined by the first conditional moment.Several applications of the identities are shown. For instance, by using one of the identities, a simple proof of the uniqueness of the conditional mean estimator as a function of the distribution of the signal is shown. Moreover, one of the identities is used to extend the notion of empirical Bayes to higher-order conditional moments. Specifically, based on a random sample of noisy observations, a consistent estimator for a conditional expectation of any order is derived.

This paper considers a noisy data structure recovery problem. Specifically, the goal is to investigate the following question: Given a noisy observation of the data, according to which permutation was the original data sorted? The main focus is on scenarios where data is generated according to an isotropic Gaussian distribution, and the perturbation consists of adding Gaussian noise with diagonal scalar covariance matrix. This problem is posed within a hypothesis testing framework. First, the optimal decision criterion is characterized and shown to be identical to the hypothesis of the observation. Then, by leveraging the structure of the optimal decision criterion, the probability of error is characterized. Finally, the logarithmic behavior (i.e., the exponent) of the probability of error is derived in the regime where the dimension of the data goes to infinity.

This work considers a random sample X1,X2,…,Xndrawn independently and identically distributed from some known parent distribution PXwith X(1)≤ X(2)≤ … ≤ X(n)being the order statistics of the sample. Under the assumption of an invertible cumulative distribution function associated with the parent distribution PX, a distribution-free property is established showing that the f-divergence between the joint distribution of order statistics and the product distribution of order statistics does not depend on PX. Moreover, it is shown that the mutual information between two subsets of order statistics also satisfies a distribution-free property; that is, it does not depend on PX. Furthermore, the decoupling rates between X(r)and X(m)(i.e., rates at which the mutual information approaches zero) are characterized for various choices of (r,m). The work also considers discrete distributions, which do not satisfy the previously-stated invertibility assumption, and it is shown that no such distribution-free property holds: the mutual information between order statistics does depend on the parent distribution PX. Upper bounds on the decoupling rates in the discrete setting are also established.

Consider the conditional mean estimator of the random variable X from the noisy observation Y = X + N where N is zero mean Gaussian with variance σ2(i.e., E[X|Y]). This work characterizes the probability distribution of E[X|Y]. As part of the proof, several new identities and results are shown. For example, it is shown that the k-th derivative of the conditional expectation is proportional to the (k + 1)-th conditional cumulant. It is also shown that the compositional inverse of the conditional expectation is well-defined and is characterized in terms of a power series by using Lagrange inversion theorem.

This letter considers an M-ary hypothesis testing problem on an n-dimensional random vector perturbed by the addition of Gaussian noise. A novel expression for the gradient of the error probability, with respect to the covariance matrix of the noise, is derived and shown to be a function of the cross-covariance matrix between the noise matrix (i.e., the matrix obtained by multiplying the noise vector by its transpose) and Bernoulli random variables associated with the correctness event.

This paper considers estimation of a random variable in Poisson noise with signal scaling coefficient and dark current as explicit parameters of the noise model. Specifically, the paper focuses on properties of the conditional mean estimator as a function of the scaling coefficient, the dark current parameter, the distribution of the input random variable and channel realizations. With respect to the scaling coefficient and the dark current, several identities in terms of derivatives are established. For example, it is shown that the gradient of the conditional mean estimator with respect to the scaling coefficient and dark current parameter is proportional to the conditional variance. Moreover, a score function is proposed and a Tweedie-like formula for the conditional expectation is recovered. With respect to the distribution, several regularity conditions are shown. For instance, it is shown that the conditional mean estimator uniquely determines the input distribution. Moreover, it is shown that if the conditional expectation is close to a linear function in terms of mean squared error, then the input distribution is approximately gamma in the Lévy distance. Furthermore, sufficient and necessary conditions for linearity are found. Interestingly, it is shown that the conditional mean estimator cannot be linear when the dark current parameter of the Poisson noise is non-zero.

This paper studies an n -dimensional additive Gaussian noise channel with a peak-power-constrained input. It is well known that, in this case, when n=1 the capacity-achieving input distribution is discrete with finitely many mass points, and when n > 1 the capacity-achieving input distribution is supported on finitely many concentric shells. However, due to the previous proof technique, not even a bound on the exact number of mass points/shells was available. This paper provides an alternative proof of the finiteness of the number mass points/shells of the capacity-achieving input distribution while producing the first firm bounds on the number of mass points and shells, paving an alternative way for approaching many such problems. The first main result of this paper is an order tight implicit bound which shows that the number of mass points in the capacity-achieving input distribution is within a factor of two from the number of zeros of the downward shifted capacity-achieving output probability density function. Next, this implicit bound is utilized to provide a first firm upper on the support size of optimal input distribution, an O(A2) upper bound where A denotes the constraint on the input amplitude. The second main result of this paper generalizes the first one to the case when n > 1, showing that, for each and every dimension n ≥ 1, the number of shells that the optimal input distribution contains is O(A2). Finally, the third main result of this paper reconsiders the case n = 1 with an additional average power constraint, demonstrating a similar O(A2) bound.

In both wired and wireless communication networks, power allocation is an important technique to improve system performance. This paper investigates the power allocation problem for parallel Gaussian channels from an information-theoretic perspective with the aim of maximizing the sum of mutual informations (i.e., an achievable data rate). If all the inputs are Gaussian, it is well-known that the waterfilling policy provides an optimal solution. For arbitrary input distributions, a generalization of waterfilling, so-called mercury/waterfilling, provides an optimal solution in terms of the minimum mean square errors (MMSEs). However, the difficulty of obtaining closed-form analytical expressions of the MMSE often makes computing the mercury/waterfilling solution challenging. This paper proposes a robust waterfilling power allocation (RPA) policy for parallel Gaussian channels when the input distributions are close to Gaussian distributions in the Kullback-Leibler (KL) divergence (relative entropy). First, it is shown that the proposed policy results in water-levels that are close to the optimal ones in a well-defined sense. Second, tight bounds for the loss in mutual information (data rate) are given. This bounded loss property makes the proposed power allocation policy robust and approximately optimal, which is illustrated by means of various simulation setups. Moreover, the RPA policy provides a general framework for solving the power allocation problem for parallel channels, with the classical waterfilling being included as a special case. Finally, the RPA policy is argued to be scalable with the number of users since it inherently uses the classical low complexity waterfilling.

This paper investigates the power allocation problem for parallel Gaussian channels from an information-theoretic perspective with the aim of maximizing the sum of mutual informations (i.e., an achievable data rate). If all the inputs are Gaussian, it is well-known that the waterfilling policy provides an optimal solution. For arbitrary input distributions, a generalization of waterfilling, so-called mercury/waterfilling, provides an optimal power allocation in terms of the minimum mean square errors (MMSEs). However, the difficulty of obtaining closed-form analytical expression of the MMSE often makes the computation of mercury/waterfilling solution challenging. This paper proposes a robust waterfilling power allocation (RPA) policy for parallel Gaussian channels when the input distributions are close to Gaussian distributions in the Kullback-Leibler divergence (relative entropy). First, it is shown that the proposed policy results in water levels that are close to the optimum in a well-defined sense. Second, tight bounds for the loss in achievable rate are given. This bounded loss property makes the proposed power allocation policy robust and approximately optimal. Both aspects are illustrated by means of different simulation setups. Finally, the RPA is argued to be scalable with the number of users on the account of the fact that it inherently uses the classical low complexity waterfilling.

Providing massive connectivity is one of the key challenges for the next generation of wireless communication networks, and hence the capacity limits of massive connectivity need to be thoroughly studied. The uplink in the regime of massive connectivity is captured by the many-access channel (MnAC) model, assuming the number of users to be extremely large and comparable to the blocklength. This work investigates a generalized MnAC, in which the transmitters and/or the receiver can be equipped with multiple antennas, and the channel gain of each user is allowed to be different. This work characterizes the sum-message-length capacity of the multiple-input and multiple-output Gaussian MnAC in the regime where the number of users increases sub-linearly in the blocklength (i.e., Kn= o(n)).

This paper presents an estimation framework to assess the performance of the sorting function over data that is perturbed. In particular, the performance is measured in terms of the Minimum Mean Square Error (MMSE) between the values of the sorting function computed on the data without perturbation and the estimate that uses the sorting function applied to the perturbed data. It is first shown that, under certain conditions satisfied by the practically relevant Gaussian noise perturbation, the optimal estimator can be expressed as a linear combination of estimators on the unsorted data. Then, a suboptimal estimator is proposed, and its performance is evaluated and compared to the optimal estimator. Finally, a lower bound on the desired MMSE is derived when data is i.i.d. and has a Gaussian distribution. This is accomplished by solving a new problem that consists of estimating the norm of an unsorted vector from a noisy observation of it.

This paper studies an n-dimensional additive Gaussian noise channel with a peak-power-constrained input. It is well known that, in this case, the capacity-achieving input distribution is supported on finitely many concentric shells. However, due to the previous proof technique, neither the exact number of shells of the optimal input distribution nor a bound on it was available. This paper provides an alternative proof of the finiteness of the number shells of the capacity-achieving input distribution and produces the first firm upper bound on the number of shells, paving an alternative way for approaching many such problems. In particular, for every dimension n, it is shown that the number of shells is given by O(A2) where A is the constraint on the input amplitude. Moreover, this paper also provides bounds on the number of points for the case of n = 1 with an additional power constraint.

This paper considers estimation of a random variable in Poisson noise. Specifically, the paper focuses on properties of the conditional mean estimator as a function of the scaling coefficient, the dark current parameter, the distribution of the input random variable and channel realizations. With respect to the scaling coefficient and the dark current, several identities in terms of derivatives are established. For example, it is shown that the derivative of the conditional mean estimator with respect to the dark current parameter is proportional to the conditional variance. Moreover, a version of score function is proposed and a Tweedy-like formula for the conditional expectation is recovered. With respect to the distribution, several regularity conditions are shown. For instance, it is shown that the conditional mean estimator uniquely determines the input distribution. Moreover, it is shown that if the conditional expectation is close to a linear function in the mean squared error, then the input distribution is approximately gamma in the Lévy metric.

This paper studies the classical Bayesian normal mean estimation problem where the estimand is assumed to be contained in a bounded set. It is known that the least favorable distribution for this mean estimation problem is discrete with finitely many mass points. This work offers an alternative proof utilizing the variational diminishing property of Gaussian kernels.

This letter considers the task of estimating the norm of an n-dimensional Gaussian random vector given a noisy/perturbed observation of it. In particular, the focus is on the case of additive Gaussian noise perturbation, which is assumed to be independent of the original vector. First, an expression for the optimal estimator is derived, and then the corresponding minimum mean square error (MMSE) is computed. The regime of large vector size is also analyzed, and it is shown that the MMSE normalized by n equals zero when n → ∞.

Providing massive connectivity is one of the key challenges for the next generation of wireless communication networks, and hence the capacity limits of massive connectivity need to be thoroughly studied. The uplink in the regime of massive connectivity is captured by the many-access channel (MnAC) model, assuming the number of users to be extremely large and comparable to the blocklength. This work investigates a generalized MnAC, in which the transmitters and/or the receiver can be equipped with multiple antennas, and the channel gain of each user is allowed to be different. This model is referred to as the multiple-input and multiple-output (MIMO) MnAC model. In the MnAC paradigm, the message length (i.e., the number of bits communicated) is not necessarily linear in the blocklength. Therefore, instead of the conventional code rate, the message length is studied and defined as a function of the blocklength. This work characterizes the sum-message-length capacity (SMC) of the MIMO Gaussian MnAC in the regime where the number of users increases sub-linearly in the blocklength (i.e., Kn= o(n)). The SMC is numerically compared to lower bounds on achievable rate at finite blocklengths and is shown to be a good approximation for system performance. The impact of the number of antennas per user on SMC is also investigated. While in the single antenna MnAC model the conventional code rate is always zero, it is shown that in the MIMO MnAC it is possible to achieve positive rate by increasing the number of antennas per user. Furthermore, the antenna-user index is defined and the SMC is characterized for different antenna-user joint regimes. This provides useful insights for future MIMO MnAC system design.

This paper studies the capacity of an n-dimensional vector Gaussian noise channel subject to the constraint that an input must lie in the ball of radius R centered at the origin. It is known that in this setting, the optimizing input distribution is supported on a finite number of concentric spheres. However, the number, the positions, and the probabilities of the spheres are generally unknown. This paper characterizes necessary and sufficient conditions on the constraint R, such that the input distribution supported on a single sphere is optimal. The maximum R̅n, such that using only a single sphere is optimal, is shown to be a solution of an integral equation. Moreover, it is shown that ̅Rn scales as √n and the exact limit of R̅n/√n is found.

Differential privacy mechanism design has traditionally been tailored for a scalar-valued query function. Although many mechanisms such as the Laplace and Gaussian mechanisms can be extended to a matrix-valued query function by adding i.i.d. noise to each element of the matrix, this method is often suboptimal as it forfeits an opportunity to exploit the structural characteristics typically associated with matrix analysis. To address this challenge, we propose a novel differential privacy mechanism called the Matrix-Variate Gaussian (MVG) mechanism, which adds a matrix-valued noise drawn from a matrix-variate Gaussian distribution, and we rigorously prove that the MVG mechanism preserves (ε,δ)-differential privacy. Furthermore, we introduce the concept of directional noise made possible by the design of the MVG mechanism. Directional noise allows the impact of the noise on the utility of the matrix-valued query function to be moderated. Finally, we experimentally demonstrate the performance of our mechanism using three matrix-valued queries on three privacy-sensitive datasets. We find that the MVG mechanism can notably outperforms four previous state-of-the-art approaches, and provides comparable utility to the non-private baseline.

Massive machine-type communication (mMTC) has been identified as one of the three generic 5G services, with the aim of providing connectivity to a large number of devices. The concurrent massive access may lead to congestions due to limited access resources. Group paging (GP) has emerged as one of the promising solutions to alleviate network congestion by controlling access load. However, the performance of GP deteriorates drastically with the number of devices per paging group. This paper explores GP with pre-backoff strategy for a general mMTC scenario in which devices are allowed to have diverse access success probability (ASP) requirements, and proposes an ASP requirement guaranteed GP scheme with specific pre-backoff times (GPSP), with the aim of maximizing the total access rate. To fully adapt to mMTC applications, an efficient heuristic algorithm is designed. Numerical results demonstrate that the proposed GPSP scheme can effectively improve system performance in terms of average ASP, average access delay, and the average number of preamble transmissions.

This paper studies optimization of the minimum mean square error (MMSE) in order to characterize the structure of the least favorable prior distributions. In the first part, the paper characterizes the local behavior of the MMSE in terms of the input distribution and finds the directional derivative of the MMSE at the distribution$P_{\mathbf{X}}$in the direction of the distribution$Q_{\mathbf{X}}$. In the second part of the paper, the directional derivative together with the theory of convex optimization is used to characterize the structure of least favorable distributions. In particular, under mild regularity conditions, it is shown that the support of the least favorable distributions must necessarily be very small and is contained in a nowhere dense set of Lebesgue measure zero. The results of this paper produce both sufficient and necessary conditions for optimality, do not rely on Gaussian statistics assumptions, and are not sensitive to the dimensionality of random vectors. The results are evaluated for the univariate and multivariate random Gaussian cases, and the Poisson case. Finally, as one of the applications, it is shown how the results can be used to characterize the capacity of Gaussian MIMO channels with an amplitude constraint.

This paper focuses on the mutual information and minimum mean-squared error (MMSE) as a function a matrix-valued signal-to-noise ratio (SNR) for a linear Gaussian channel with arbitrary input distribution. As shown by Lamarca, the mutual-information is a concave function of a positive semidefinite matrix, which we call the matrix SNR. This implies that the mapping from the matrix SNR to the MMSE matrix is decreasing monotone. Building upon these functional properties, we start to construct a unifying framework that provides a bridge between classical information-theoretic inequalities, such as the entropy power inequality, and interpolation techniques used in statistical physics and random matrix theory. This framework provides new insight into the structure of phase transitions in coding theory and compressed sensing. In particular, it is shown that the parallel combination of linear channels with freely-independent matrices can be characterized succinctly via free convolution.

In this paper, an analysis of an input distribution that achieves the secrecy capacity of a general degraded additive noise wiretap channel is presented. In particular, using convex optimization methods, an input distribution that achieves the secrecy capacity is characterized by conditions expressed in terms of integral equations. The new conditions are used to study the structure of the optimal input distribution for three different additive noise cases: vector Gaussian; scalar Cauchy; and scalar exponential.

This paper studies the capacity of an n-dimensional vector Gaussian noise channel subject to the constraint that an input must lie in the ball of radius R centered at the origin. It is known that in this setting the optimizing input distribution is supported on a finite number of concentric spheres. However, the number, the positions and the probabilities of the spheres are generally unknown. This paper characterizes necessary and sufficient conditions on the constraint R such that the input distribution supported on a single sphere is optimal. The maximum $\overline{R}_{n}$, such that using only a single sphere is optimal, is shown to be a solution of an integral equation. Moreover, it is shown that $\overline{R}_{n}$ scales as $\sqrt{n}$ and the exact limit of $\overline{R}_{n}\overline{\sqrt{n}}$ is found.

Massive machine-type communication (mMTC) has been identified as one of the three generic 5G services, with the aim of providing connectivity to a large number of devices. The concurrent massive access in mMTC may lead to congestion due to limited access resources. Group paging (GP) is emerging as one of the promising solutions to alleviate network congestion by controlling access load. However, the performance of GP deteriorates drastically with the number of devices per paging group. This paper explores GP with a pre-backoff strategy for a general mMTC scenario in which devices are allowed to have diverse access success probability (ASP) requirements, and proposes a differentiated service-aware GP scheme with specific pre-backoff times (GPSP), with the aim of maximizing the total access rate while guaranteeing the ASP requirements for individual devices. An optimal solution to the GPSP problem is derived to provide a performance upper bound. As low-complexity algorithms are of key importance for mMTC applications, an efficient heuristic algorithm is further designed. Numerical results demonstrate that the proposed GPSP scheme can effectively improve the system performance in terms of average ASP, average access delay, and the average number of preamble transmissions.

This paper considers a Gaussian channel with one transmitter and two receivers. The goal is to maximize the communication rate at the intended/primary receiver subject to a disturbance constraint at the unintended/secondary receiver. The disturbance is measured in terms of the minimum mean square error (MMSE) of the interference that the transmission to the primary receiver inflicts on the secondary receiver. This paper presents a new upper bound for the problem of maximizing the mutual information subject to an MMSE constraint. The new bound holds for vector inputs of any length and recovers a previously known limiting (when the length of the vector input tends to infinity) expression from the work of Bustin et al. The key technical novelty is a new upper bound on the MMSE. This bound allows one to bound the MMSE for all signal-to-noise ratio (SNR) values below a certain SNR at which the MMSE is known (which corresponds to the disturbance constraint). The bound also complements the “single-crossing point property” of the MMSE that upper bounds the MMSE for all SNR values above a certain value at which the MMSE value is known. The MMSE upper bound provides a refined characterization of the phase-transition phenomenon, which manifests, in the limit as the length of the vector input goes to infinity, as a discontinuity of the MMSE for the problem at hand. For vector inputs of size n = 1, a matching lower bound, to within an additive gap of order O(log log(1/MMSE)) (where MMSE is the disturbance constraint), is shown by means of the mixed inputs technique recently introduced by Dytso et al.

In this work, novel upper and lower bounds on the capacity of channels with arbitrary constraints on the support of the channel input symbols are derived. As an immediate practical application, the case of multiple-input multiple- output channels with amplitude constraints is considered. The bounds are shown to be within a constant gap if the channel matrix is invertible and are tight in the high amplitude regime for arbitrary channel matrices. Moreover, in the high amplitude regime, it is shown that the capacity scales linearly with the minimum of the numbers of transmit and receive antennas, similarly to the case of average power-constrained inputs.

This paper considers a problem of communication over an additive noise channel where the noise is distributed according to a Generalized Gaussian (GG) distribution. In the first part of the paper, a number of properties of the family of GG distributions are derived which are of independent interest. For example, considerable attention is given to the properties of the characteristic function of the GG distribution. In the second part of the paper, the capacity of an additive noise channel with GG noise is considered under p-th absolute moment constraints. It is shown that, even though Shannon's upper bound is achievable in some instances, in general such achievability is not possible. Moreover, it is shown that discrete inputs can achieve capacity within a constant gap or full degree of freedom for any p-th absolute moment constraint. Following the seminal work of Smith, the paper also gives a condition under which discrete inputs are exactly optimal.

In this paper, a generalized Ozarow-Wyner capacity bound is presented that holds for arbitrary noise channels. The bound is then used to approximate the capacity of a large class of additive noise channels that are subject to a p-th moment input constraint, where p is some positive real number, as well as to the Cauchy noise channel with a logarithmic moment constraint. For both channel models the gap to the capacity is precisely specified.

The problem of estimating an arbitrary random vector from its observation corrupted by additive white Gaussian noise, where the cost function is taken to be the minimum mean pth error (MMPE), is considered. The classical minimum mean square error (MMSE) is a special case of the MMPE. Several bounds, properties, and applications of the MMPE are derived and discussed. The optimal MMPE estimator is found for Gaussian and binary input distributions. Properties of the MMPE as a function of the input distribution, signal-to-noiseratio (SNR) and order p are derived. The “single-crossing-point property” (SCPP) which provides an upper bound on the MMSE, and which together with the mutual information-MMSE relationship is a powerful tool in deriving converse proofs in multiuser information theory, is extended to the MMPE. Moreover, a complementary bound to the SCPP is derived. As a first application of the MMPE, a bound on the conditional differential entropy in terms of the MMPE is provided, which then yields a generalization of the Ozarow-Wyner lower bound on the mutual information achieved by a discrete input on a Gaussian noise channel. As a second application, the MMPE is shown to improve on previous characterizations of the phase transition phenomenon that manifests, in the limit as the length of the capacity achieving code goes to infinity, as a discontinuity of the MMSE as a function of SNR. As a final application, the MMPE is used to show new bounds on the second derivative of mutual information, or the first derivative of the MMSE.

This paper considers the minimum mean p-th error (MMPE) estimation problem: estimating a random vector in the presence of additive white Gaussian noise (AWGN) in order to minimize an Lpnorm of the estimation error. The MMPE generalizes the classical minimum mean square error (MMSE) estimation problem. This paper derives basic properties of the optimal MMPE estimator and MMPE functional. Optimal estimators are found for several inputs of interests, such as Gaussian and binary symbols. Under an appropriate p-th moment constraint, the Gaussian input is shown to be asymptotically the hardest to estimate for any p ≥ 1. By using a conditional version of the MMPE, the famous “MMSE single-crossing point” bound is shown to hold for the MMPE too for all p ≥ 1, up to a multiplicative constant. Finally, the paper develops connections between the conditional differential entropy and the MMPE, which leads to a tighter version of the Ozarow-Wyner lower bound on the rate achieved by discrete inputs on AWGN channels.

This paper shows that for the two-user Gaussian interference channel (G-IC) treating interference as noise without time sharing (TINnoTS) achieves the closure of the capacity region to within either a constant gap, or to within a gap of the order O(log(ln(min(S, I))/y)) up to a set of Lebesgue measure γ ∈ (0, 1], where S is the largest signal to noise ratio on the direct links and I is the largest interference to noise ratio on the cross links. As a consequence, TINnoTS is optimal from a generalized degrees of freedom (gDoF) perspective for all channel gains except for a subset of zero measure. TINnoTS with Gaussian inputs is known to be optimal within 1/2 bit for a subset of the weak interference regime. Rather surprisingly, this paper shows that TINnoTS is gDoF optimal in all parameter regimes, even in the strong and very strong interference regimes where joint decoding of Gaussian inputs is optimal. For approximate optimality of TINnoTS in all parameter regimes, it is critical to use non-Gaussian inputs. This paper thus proposes to use mixed inputs as channel inputs for the G-IC, where a mixed input is the sum of a discrete and a Gaussian random variable. Interestingly, with reference to the Han-Kobayashi achievable scheme, the discrete part of a mixed input is shown to effectively behave as a common message in the sense that, although treated as noise, its effect on the achievable rate region is as if it were jointly decoded together with the desired messages at a non-intended receiver. The practical implication is that a discrete interfering input is a friend, while an Gaussian interfering input is in general a foe. This paper also discusses other practical implications of the proposed TINnoTS scheme with mixed inputs. Since TINnoTS requires neither explicit joint decoding nor time sharing, the results of this paper are applicable to a variety of oblivious or asynchronous channels, such as the block asynchronous G-IC (which is not an information stable channel) and the G-IC with partial codebook knowledge at one or more receivers.

While a multi-letter limiting expression of the capacity region of the two-user Gaussian interference channel is known, capacity is generally considered to be open as this is not computable. Other computable capacity outer bounds are known to be achievable to within 1/2 bit using Gaussian inputs and joint decoding in the simplified Han and Kobayashi (single-letter) achievable rate region. This work shows that the simple scheme known as “treating interference as noise” without time-sharing attains the capacity region outer bound of the symmetric Gaussian interference channel to within either a constant gap, or a gap of order O(log log(SNR)), for all parameter regimes. The scheme is therefore optimal in the generalized Degrees of Freedom (gDoF) region sense almost surely. The achievability is obtained by using i.i.d. mixed inputs (i.e., a superposition of discrete and Gaussian random variables) in the multi-letter capacity expression, where the optimal number of points in the discrete part of the inputs, as well as the optimal power split among the discrete and continuous parts of the inputs, are characterized in closed form. An important practical implication of this result is that the discrete part of the inputs behaves as a “common message” whose contribution can be removed from the channel output, even though joint decoding is not employed. Moreover, time-sharing may be mimicked by varying the number of points in the discrete part of the inputs.

The study of the two-user Gaussian Interference Channel (IC) where one receiver lacks knowledge of the interfering codebook, also dubbed the IC with an oblivious receiver (IC-OR), is motivated by: (1) in heterogeneous, cognitive, distributed or dynamic networks, assuming that every node posses codebooks of every other node may not be practical, and (2) it is not clear whether and how much lack of codebook knowledge would affect the Han and Kobayashi (HK) achievable scheme, which involves joint decoding of intended and interfering messages and which appears not possible if nodes do not possess all codebooks. To address these issues, we evaluate a simplified HK (where the oblivious receiver treats interference as noise) with mixed inputs at the non-oblivious transmitter, i.e., a mixture of discrete and Gaussian random variables, where the power split between the two and the number of points of the discrete part are carefully chosen as a function of the channel parameters. The oblivious transmitter uses a purely Gaussian input. Surprisingly, for this choice of inputs, the capacity region of the symmetric Gaussian IC-OR is shown to be within 1 over 2 log (12πe) ≈ 3.34 bits of the best known outer bound for the classical Gaussian IC with full codebook knowledge at both receivers. Interestingly, this shows that a simplified HK where one receiver is restricted to treat interference as noise loses at most 1 over 2 log (12πe) ≈ 3.34 bits in performance. Moreover, the discrete part of the input behaves like a “common message” even though it is not jointly decoded (together with the intended messages) at the oblivious receiver.

In multiuser information theory, it is often assumed that every node in the network possesses all codebooks used in the network. This assumption may be impractical in distributed ad hoc, cognitive, or heterogeneous networks. This paper considers the two-user interference channel with one oblivious receiver (IC-OR), i.e., one receiver lacks knowledge of the interfering cookbook, whereas the other receiver knows both codebooks. This paper asks whether, and if so how much, the channel capacity of the IC-OR is reduced compared with that of the classical IC where both receivers know all codebooks. A novel outer bound is derived and shown to be achievable to within a gap for the class of injective semideterministic IC-ORs; the gap is shown to be zero for injective fully deterministic IC-ORs. An exact capacity result is shown for the general memoryless IC-OR when the nonoblivious receiver experiences very strong interference. For the linear deterministic IC-OR that models the Gaussian noise channel at high SNR, nonindependent identically distributed. Bernoulli(1/2) input bits are shown to achieve points not achievable by i.i.d. Bernoulli(1/2) input bits used in the same achievability scheme. For the real-valued Gaussian IC-OR, the gap is shown to be at most 1/2 bit per channel use, even though the set of optimal input distributions for the derived outer bound could not be determined. Toward understanding the Gaussian IC-OR, an achievability strategy is evaluated in which the input alphabets at the nonoblivious transmitter are a mixture of discrete and Gaussian random variables, where the cardinality of the discrete part is appropriately chosen as a function of the channel parameters. Surprisingly, as the oblivious receiver intuitively should not be able to jointly decode the intended and interfering messages (whose codebook is unavailable), it is shown that with this choice of input, the capacity region of the symmetric Gaussian IC-OR is to within 1/2 log (12πe)≈ 3.34 bits (per channel use per user) of an outer bound for the classical Gaussian IC with full codebook knowledge at both receivers.

This paper studies the sum-rate of a class of memoryless, real-valued additive white Gaussian noise interference channels (IC) achievable by treating interference as noise (TIN). We develop and analytically characterize the rates achievable by a new strategy that uses superpositions of Gaussian and discrete random variables as channel inputs. Surprisingly, we demonstrate that TIN is sum-generalized degrees of freedom optimal and can achieve to within an additive gap of O(1) or O(log log(SNR)) to the symmetric sum-capacity of the classical IC. We also demonstrate connections to other channels such as the IC with partial codebook knowledge and the block asynchronous IC.

This paper considers a variation of the classical two-user interference channel where the communication of two interfering source-destination pairs is aided by an additional node that has a priori knowledge of the messages to be transmitted, which is referred to as the cognitive relay. For this interference channel with a cognitive relay (ICCR), novel outer bounds and capacity region characterizations are derived. In particular, for the class of injective semi-deterministic ICCRs, a sum-rate upper bound is derived for the general memoryless ICCR and further tightened for the linear deterministic approximation (LDA) of the Gaussian noise channel at high SNR, which disregards the noise and focuses on the interaction among the users' signals. The capacity region of the symmetric LDA is completely characterized except for the regime of moderately weak interference and weak links from the CR to the destinations. The insights gained from the analysis of the LDA are then translated back to the symmetric Gaussian noise channel (GICCR). For the symmetric GICCR, an approximate characterization (to within a constant gap) of the capacity region is provided for a parameter regime where capacity was previously unknown. The approximately optimal scheme suggests that message cognition at a relay is beneficial for interference management as it enables simultaneous over the air neutralization of the interference at both destinations.

Shannon theoretic multi-user capacity problems are traditionally formulated under the assumption that all decoding nodes possess all codebooks. However, for certain networks such as cognitive ones, this may be an unrealistic assumption. We work towards understanding the impact of lack of codebook knowledge at some decoding nodes in the network. We do so by considering a two-user interference channel in which one of the receivers has no information about the codebook of the interfering transmitter, while the other receiver has both codebooks. We derive a novel outer bound for the special class of injective semi-deterministic interference channels which incorporates this codebook knowledge explicitly. For the linear deterministic channel, which models the Gaussian channel at high SNR, we demonstrate the surprising fact that non i.i.d. Bernoulli(1/2) points achieve points on the outer bound not achievable by Bernoulli(1/2) inputs. We then show that this is achievable to within a constant gap by a modified Han-Kobayashi scheme. We characterize the capacity region of the Gaussian noise channel to within 1/2 bit, even though we could not determine the set of optimal input distributions. Numerical evaluations suggest that if the non-oblivious transmitter uses a discrete input a larger sum-rate is achievable compared to the case where both users employ Gaussian codebooks or use time division in strong interference regime at high SNR.

The capacity of the Interference Channel with a Cognitive Relay, a channel model which generalizes the broadcast, interference and cognitive interference channels, is still an open question. Towards understanding this complex channel, we first consider the binary linear deterministic model that approximates the Gaussian channel at high SNR. We consider symmetric channel gains and show achievability of a tightened version of a previously known outer bound for almost all channel parameters. Of particular interest in this channel model is how the cognitive relay may be used to simultaneously relay as well as cancel/neutralize interference at the two receivers. The achievability schemes used to prove capacity use combinations of three main strategies at the cognitive relay that we term bit cancellation, bit sharing, and bit (self)cleaning. We highlight the capacity achieving schemes in the different regimes, pointing out some of the interesting new behaviors seen at the cognitive relay.