We investigate the capacity of noisy frequency-based channels, motivated by DNA data storage in the short-molecule regime, where information is encoded in the frequency of items types rather than their order. The channel output is a histogram formed by random sampling of items, followed by noisy item identification. While the capacity of the noiseless frequency-based channel has been previously addressed, the effect of identification noise has not been fully characterized. We present a converse bound on the channel capacity that follows from stochastic degradation and the data processing inequality. We then establish an achievable bound, which is based on a Poissonization of the multinomial sampling process, and an analysis of the resulting vector Poisson channel with inter-symbol interference. This analysis refines concentration inequalities for the information density used in Feinstein bound, and explicitly characterizes an additive loss in the mutual information due to identification noise. We apply our results to a DNA storage channel in the short-molecule regime, and quantify the resulting loss in the scaling of the total number of reliably stored bits.

In many sequential decision problems, an agent performs a repeated task. He then suffers regret and obtains information that he may use in the following rounds. However, sometimes the agent may also obtain information and avoid suffering regret by querying external sources. We study the trade-off between the information an agent accumulates and the regret it suffers. We invoke information-theoretic methods for obtaining regret lower bounds, that also allow us to easily re-derive several known lower bounds. We introduce the first Bayesian regret lower bounds that depend on the information an agent accumulates. We also prove regret upper bounds using the amount of information the agent accumulates. These bounds show that information measured in bits, can be traded off for regret, measured in reward. Finally, we demonstrate the utility of these bounds in improving the performance of a question-answering task with large language models, allowing us to obtain valuable insights.

While diffusion models generate high-quality images via probability flow, the theoretical understanding of this process remains incomplete. A key question is when probability flow converges to training samples or more general points on the data manifold. We analyze this by studying the probability flow of shallow ReLU neural network denoisers trained with minimal $\ell^2$ norm. For intuition, we introduce a simpler score flow and show that for orthogonal datasets, both flows follow similar trajectories, converging to a training point or a sum of training points. However, early stopping by the diffusion time scheduler allows probability flow to reach more general manifold points. This reflects the tendency of diffusion models to both memorize training samples and generate novel points that combine aspects of multiple samples, motivating our study of such behavior in simplified settings. We extend these results to obtuse simplex data and, through simulations in the orthogonal case, confirm that probability flow converges to a training point, a sum of training points, or a manifold point. Moreover, memorization decreases when the number of training samples grows, as fewer samples accumulate near training points.

We study the multi-terminal remote estimation problem under a rate constraint, in which the goal of the encoder is to help each decoder estimate a function over a certain distribution - while the distribution is known only to the encoder, the function to be estimated is known only to the decoders, and can also be different for each decoder. The decoders can observe correlated samples from prior distributions, instantiated through shared randomness with the encoder. To achieve this, we employ remote generation, where the encoder helps decoders generate samples from the underlying distribution by using the samples from the prior through importance sampling. While methods such as minimal random coding can be used to efficiently transmit samples to each decoder individually using their importance scores, it is unknown if the correlation among the samples from the priors can reduce the communication cost using the availability of a broadcast link. We propose a hierarchical importance sampling strategy that facilitates, in the case of nonzero Gács-Körner common information among the priors of the decoders, a common sampling step leveraging the availability of a broadcast channel. This is followed by a refinement step for the individual decoders. We present upper bounds on the bias and the estimation error for unicast transmission, which is of independent interest. We then introduce a method that splits into two phases, dedicated to broadcast and unicast transmission, respectively, and show the reduction in communication cost.

We explore the problem of mean estimation for a high-dimensional binary symmetric Gaussian mixture model, where the label (sign) follows a time-inhomogeneous Markov chain. We propose a spectral estimator based on a partition of a subset of the samples to blocks. We develop a computationally efficient algorithm to find the optimal blocks, and derive minimax lower bounds on the estimation loss of any estimator, which establish the effectiveness of our proposed estimator. The resulting minimax rate illuminates the interplay between the sample size, dimension, signal strength, and the memory on the loss.

We address the challenge of optimizing the capacity-achieving input distribution for a multinomial channel under the constraint of limited input support size, which is a crucial aspect in the design of DNA storage systems. We propose an algorithm that further elaborates the Multidimensional Dynamic Assignment Blahut-Arimoto (M-DAB) algorithm [1]. Our proposed algorithm integrates variational autoencoder for determining the optimal locations of input distribution, into the alternating optimization of the input distribution locations and weights.

We consider the problem of detecting the overlap between a pair of short fragments sampled in random locations from an exponentially longer sequence, via their possibly noisy reads. We consider a noiseless setting, in which the reads are noiseless, and the sequence is only assumed to be stationary and ergodic. Under mild conditions on the mixing property of the process generating the sequence, we characterize exactly the asymptotic error probability of the optimal Bayesian detector. Similarly, we consider a noisy setting, in which the reads are noisy versions of the sampled fragments obtained via a memoryless channel. We further assume that the sequence is stationary and memoryless, and similarly characterize exactly the asymptotic error probability of the optimal Bayesian detector for this case.

We consider a molecular channel, in which messages are encoded to the frequency of objects (or concentration of molecules) in a pool, and whose output during reading time is a noisy version of the input frequencies, as obtained by sampling with replacement from the pool. Motivated by recent DNA storage techniques, we focus on the regime in which the input resolution is unlimited. We derive an achievable bound on the capacity of such channel, and compare it to both the achievable bounds under limited input resolution, as well as to a converse bound.

We consider the problem of estimating the Shannon entropy of an unknown probability mass function (PMF)$P$from a countable alphabet with a finite unknown support. The estimator may query the exact value of$P$in any specific letter, but this operation is costly, and hence the amount of queries should be minimized. In parallel, the estimator can sample from an auxiliary PMF$Q$, which is assumed to be close to$P$in some sense. We propose and analyze two specific algorithms for this setting. The first is based on importance sampling and is shown to be asymptotically unbiased and consistent, but does not limit the number of queries. The second algorithm, called$Q$-sort, limits the number of queries. We prove that it has an exponentially decaying MSE. Finally, we also provide a simple characterization minimax lower bound on the error rate, which also tends to zero exponentially fast with the number of samples.

Universal compression can learn the source and adapt to it either in a batch mode (forward adaptation), or in a sequential mode (backward adaptation). We recast the sequential mode as a multi-armed bandit problem, a fundamental model in reinforcement-learning, and study the trade-off between exploration and exploitation in the lossy compression case. We show that a previously proposed “natural type selection” scheme can be cast as a reconstruction-directed MAB algorithm, for sequential lossy compression, and explain its limitations in terms of robustness and short-block performance. We then derive and analyze robust cost-directed MAB algorithms, which work at any block length.

In this paper, we consider a recent channel model of a nanopore sequencer proposed by McBain, Viterbo, and Saunderson (2024), termed the noisy nanopore channel (NNC). In essence, an NNC is a noisy duplication channel, whose input source has a specific Markov structure. We present computable lower and upper bounds on the channel capacity of selected NNCs, via simple information-theoretic inequalities. In particular, we provide a (tight) lower bound on the capacity of the noiseless NNC. We then consider the setting where the memory of the input process is large and the random noise introduces erasures. We demonstrate that for such an NNC, it is possible to achieve information rates close to the noise-free capacity, using simple encoding and decoding schemes.

Federated Learning (FL) incurs high communication costs in both uplink and downlink. The literature largely focuses on lossy compression of model updates in deterministic FL. In contrast, stochastic (Bayesian) FL considers distributions over parameters, enabling uncertainty quantification, better generalization, and, crucially, inherent communication-regularized training through a mirror-descent structure. In this paper, we consider both uplink and downlink communication in stochastic FL, and propose a communication framework based on remote source generation. Employing Minimal Random Coding (MRC) for remote generation, we allow the server and the clients to sample from local and global posteriors (sources), respectively, rather than transmitting locally sampled updates. The framework encompasses communication-regularized local optimization and principled compression of model updates, leveraging gradually updated prior distributions as side information. Through extensive simulations, we show that our method achieves $5-32\times$ reduction in total communication cost while preserving accuracy. We further analyze the communication cost, refining existing MRC bounds and enabling precise quantification of uplink and downlink trade-offs. We also extend our method to conventional FL via stochastic quantization and prove a contraction property for the biased MRC compressor to facilitate convergence analysis.

The problem of identifying the channel with the highest capacity among several discrete memoryless channels (DMCs) is considered. The problem is cast as a pure-exploration multi-armed bandit problem, which follows the practical use of training sequences to sense the communication channel statistics. A gap-elimination algorithm termedBestChanIDis proposed, which is oblivious to the capacity-achieving input distributions, and is guaranteed to output the DMC with the largest capacity, with a desired confidence. Furthermore, two additional algorithmsNaiveChanSelandMedianChanEl, which output with certain confidence a DMC with capacity close to the maximal, are also presented. Each of these algorithms is shown to be beneficial in a different regime and can be used as a subroutine ofBestChanID. To analyze the algorithms’ guarantees, a capacity estimator is proposed and tight confidence bounds on the estimator error are derived. Based on this estimator, the sample complexity of all the proposed algorithms is analyzed as a function of the desired confidence parameter, the number of channels, and the channels’ input and output alphabet sizes. The cost of best channel identification is shown to scale quadratically with the alphabet size, and a fundamental lower bound is derived on the number of channel senses required to identify the best channel with a certain confidence.

We consider a molecular channel, in which messages are encoded into the frequency of molecules in a pool, and whose output during reading time is a noisy version of the input frequencies, obtained by sampling with replacement from the pool. We tightly characterize the capacity of this channel using upper and lower bounds, when the number of molecules in the pool is constrained. We apply this result to the DNA storage channel in a accurately defined short-molecule regime, and show that even though the capacity of this channel is technically zero, it can still achieve very large information density.

We investigate the fundamental limits of reliable communication over multi-view channels, in which the channel output is comprised of a large number of independent noisy views of a transmitted symbol. We consider first the setting of multi-view discrete memoryless channels and then extend our results to general multi-view channels (using multi-letter formulas). We argue that the channel capacity and dispersion of such multi-view channels converge exponentially fast in the number of views to the entropy and varentropy of the input distribution, respectively. We identify the exact rate of convergence as the smallest Chernoff information between two conditional distributions of the output, conditioned on unequal inputs. For the special case of the deletion channel, we compute upper bounds on this Chernoff information. Finally, we present a new channel model we term the Poisson approximation channel — of possible independent interest — whose capacity closely approximates the capacity of the multi-view binary symmetric channel for any fixed number of views.

We consider a statistical version of curriculum learning (CL) in a parametric prediction setting. The learner is required to estimate a target parameter vector, and can adaptively collect samples from either the target model, or other source models that are similar to the target model, but less noisy. We consider three types of learners, depending on the level of side-information they receive. The first two, referred to as strong/weak-oracle learners, receive high/low degrees of information about the models, and use these to learn. The third, a fully adaptive learner, estimates the target parameter vector without any prior information. In the single source case, we propose an elimination learning method, whose risk matches that of a strong-oracle learner. In the multiple source case, we advocate that the risk of the weak-oracle learner is a realistic benchmark for the risk of adaptive learners. We develop an adaptive multiple elimination-rounds CL algorithm, and characterize instance-dependent conditions for its risk to match that of the weak-oracle learner. We consider instance-dependent minimax lower bounds, and discuss the challenges associated with defining the class of instances for the bound. We derive two minimax lower bounds, and determine the conditions under which the performance weak-oracle learner is minimax optimal.

In continual learning, catastrophic forgetting is affected by multiple aspects of the tasks. Previous works have analyzed separately how forgetting is affected by either task similarity or overparameterization. In contrast, our paper examines how task similarity and overparameterization jointly affect forgetting in an analyzable model. Specifically, we focus on two-task continual linear regression, where the second task is a random orthogonal transformation of an arbitrary first task (an abstraction of random permutation tasks). We derive an exact analytical expression for the expected forgetting — and uncover a nuanced pattern. In highly overparameterized models, intermediate task similarity causes the most forgetting. However, near the interpolation threshold, forgetting decreases monotonically with the expected task similarity. We validate our findings with linear regression on synthetic data, and with neural networks on established permutation task benchmarks.

We propose a game-based formulation for learning dimensionality-reducing representations of feature vectors, when only a prior knowledge on future prediction tasks is available. In this game, the first player chooses a representation, and then the second player adversarially chooses a prediction task from a given class, representing the prior knowledge. The first player aims to minimize, and the second player to maximize, the regret: The minimal prediction loss using the representation, compared to the same loss using the original features. We consider the canonical setting in which the representation, the response to predict and the predictors are all linear functions, and the loss function is the mean squared error. We derive the theoretically optimal representation in pure strategies, which shows the effectiveness of the prior knowledge, and the optimal regret in mixed strategies, which shows the usefulness of randomizing the representation. For general representation, prediction and loss functions, we propose an efficient algorithm to optimize a randomized representation. The algorithm only requires the gradients of the loss function, and is based on incrementally adding a representation rule to a mixture of such rules.

Motivated by communication systems with constrained complexity, we consider the problem of input symbol selection for discrete memoryless channels (DMCs). Given a DMC, the goal is to find a subset of its input alphabet, so that the optimal input distribution that is only supported on these symbols maximizes the capacity among all other subsets of the same size (or smaller). We observe that the resulting optimization problem is non-concave and non-submodular, and so generic methods for such cases do not have theoretical guarantees. We derive an analytical upper bound on the capacity loss when selecting a subset of input symbols based only on the properties of the transition matrix of the channel. We propose a selection algorithm that is based on input-symbols clustering, and an appropriate choice of representatives for each cluster, which uses the theoretical bound as a surrogate objective function. We provide numerical experiments to support the findings.

Whenever inspected by humans, reconstructed signals should not be distinguished from real ones. Typically, such a high perceptual quality comes at the price of high reconstruction error, and vice versa. We study this distortion-perception (DP) tradeoff over finite-alphabet channels, for the Wasserstein-l distance induced by a general metric as the perception index, and an arbitrary distortion matrix. Under this setting, we show that computing the DP function and the optimal reconstructions is equivalent to solving a set of linear programming problems. We provide a structural characterization of the DP tradeoff, where the DP function is piecewise linear in the perception index. We further derive a closed-form expression for the case of binary sources.

We consider a molecular channel, in which messages are encoded to the frequency of objects (or concentration of molecules) in a pool, and whose output during reading time is a noisy version of the input frequencies, as obtained by sampling with replacement from the pool. We tightly characterize the capacity of this channel using upper and lower bounds, when the number of objects in the pool of objects is constrained. We apply this result to the DNA storage channel in the short-molecule regime, and show that even though the capacity of this channel is technically zero, it can still achieve a large information density.

In this paper, we investigate the fundamental limits of reliable communication over a discrete memoryless channel (DMC) when there are a large number of noisy views of a transmitted symbol, i.e., when several copies of a single symbol are sent independently through the DMC. We argue that the channel capacity and dispersion of such a multi-view DMC converge exponentially quickly in the number of views to to the entropy and varentropy of the input distribution, respectively, and identify the exact rate of convergence. This rate equals the smallest Chernoff information between two conditional distributions of the output given unequal inputs. Our results hence help us characterize the largest finite-blocklength rates achievable for any fixed error probability. We also present a new channel model that we call the Poisson approximation channel-of possible independent interest-whose capacity closely approximates the capacity of the multi-view binary symmetric channel (BSC).

We consider the problem of reconstructing a sequence of independent and identically distributed symbols from a set of equal-size, consecutive, fragments, as well as a dependent reference sequence. First, in the regime in which the fragments are relatively long and typically no fragment appears more than once, we determine the scaling of the failure probability of the maximum-likelihood reconstruction algorithm for a perfect reconstruction, and bound it for a partial reconstruction. Second, we characterize the regime in which the fragments are relatively short and repeating fragments abound. We state a trade-off between the fraction of fragments that cannot be adequately reconstructed vs. the distortion level allowed for the reconstruction of each fragment, while still allowing vanishing failure probability.

We investigate robust linear regression where data may be contaminated by an oblivious adversary, i.e., an adversary that knows the data distribution but is otherwise oblivious to the realization of the data samples. This model has been previously analyzed under strong assumptions. Concretely, (i) all previous works assume that the covariance matrix of the features is positive definite; (ii) most of them assume that the features are centered. Additionally, all previous works make additional restrictive assumptions, e.g., assuming Gaussianity of the features or symmetric distribution of the corruptions. In this work, we investigate robust regression under a more general set of assumptions: (i) the covariance matrix may be either positive definite or positive semi definite, (ii) features may not be centered, (iii) no assumptions beyond boundedness (or sub-Gaussianity) of the features and the measurement noise. Under these assumptions we analyze a sequential algorithm, namely, a natural SGD variant for this problem, and show that it enjoys a fast convergence rate when the covariance matrix is positive definite. In the positive semi definite case we show that there are two regimes: if the features are centered, we can obtain a standard convergence rate; Otherwise, the adversary can cause any learner to fail arbitrarily.

We consider the problem of reliable communication over a discrete memoryless channel (DMC) with the help of a relay, termed the information bottleneck (IB) channel. There is no direct link between the source and the destination, and the information flows in two hops. The first hop is a noisy channel from the source to the relay. The second hop is a noiseless but limited-capacity backhaul link from the relay to the decoder. We further assume that the relay is oblivious to the transmission codebook. We examine two mismatch scenarios. In the first setting, we assume the decoder is restricted to use some fixed decoding rule, which is mismatched to the actual channel. In the second setting, we assume that the relay is restricted to use some fixed compression metric, which is again mismatched to the statistics of the relay input. We establish bounds on the random-coding capacity of both settings, some of which are shown to be ensemble tight.

We consider the problem of finding the channel with the highest capacity among several discrete memoryless channels (DMCs) with the same input-output alphabet sizes by means of exploration using multi-armed bandits. This setting is motivated by the problem of exploring channel statistics in communication systems by the invocation of training sequences. We particularly focus on the best arm identification problem and rank the candidate DMCs by their capacities. We propose a capacity estimator based on channel sensing and derive associated concentration results. Using this capacity estimator, we introduce BestChanID, a gap-elimination algorithm, oblivious to the capacity-achieving input distribution, which is guaranteed to output the best DMC, i.e., DMC with the largest capacity, with a desired confidence. We further introduce NaiveChanSel, an algorithm that outputs with certain confidence a DMC whose capacity is close to the largest capacity, and can be used as a subroutine in BestChanID. We analyze the sample complexity of both algorithms, i.e., the total number of channel senses, as a function of the desired confidence parameter, the number of available channels, and the input and output alphabet sizes of the channels. We show that the cost of best channel identification scales cubically with the alphabet size.

The problem of reconstructing a sequence of independent and identically distributed symbols from a set of equal size, consecutive, fragments, as well as a dependent reference sequence is considered. First, in the regime in which the fragments are relatively long, and typically no fragment appears more than once, the scaling of the failure probability of maximum likelihood reconstruction algorithm is exactly determined for perfect reconstruction and bounded for partial reconstruction. Second, the regime in which the fragments are relatively short and repeating fragments abound is characterized. A trade-off is stated between the fraction of fragments that fail to be adequately reconstructed vs. the distortion level allowed for the reconstruction of each fragment, while still allowing vanishing failure probability.

Neural network (NN) denoisers are an essential building block in many common tasks, ranging from image reconstruction to image generation. However, the success of these models is not well understood from a theoretical perspective. In this paper, we aim to characterize the functions realized by shallow ReLU NN denoisers --- in the common theoretical setting of interpolation (i.e., zero training loss) with a minimal representation cost (i.e., minimal $\ell^2$ norm weights). First, for univariate data, we derive a closed form for the NN denoiser function, find it is contractive toward the clean data points, and prove it generalizes better than the empirical MMSE estimator at a low noise level. Next, for multivariate data, we find the NN denoiser functions in a closed form under various geometric assumptions on the training data: data contained in a low-dimensional subspace, data contained in a union of one-sided rays, or several types of simplexes. These functions decompose into a sum of simple rank-one piecewise linear interpolations aligned with edges and/or faces connecting training samples. We empirically verify this alignment phenomenon on synthetic data and real images.

Acyclic schemes posses known benefits for database design, speeding up queries, and reducing space requirements. An acyclic join dependency (AJD) is lossless with respect to a universal relation if joining the projections associated with the schema results in the original universal relation. An intuitive and standard measure of loss entailed by an AJD is the number of redundant tuples generated by the acyclic join. Recent work has shown that the loss of an AJD can also be characterized by an information-theoretic measure. Motivated by the problem of automatically fitting an acyclic schema to a universal relation, we investigate the connection between these two characterizations of loss. We first show that the loss of an AJD is captured using the notion of KL-Divergence. We then show that the KL-divergence can be used to bound the number of redundant tuples. We prove a deterministic lower bound on the percentage of redundant tuples. For an upper bound, we propose a random database model, and establish a high probability bound on the percentage of redundant tuples, which coincides with the lower bound for large databases.

MOTIVATION: Optical genome mapping (OGM) is a technique that extracts partial genomic information from optically imaged and linearized DNA fragments containing fluorescently labeled short sequence patterns. This information can be used for various genomic analyses and applications, such as the detection of structural variations and copy-number variations, epigenomic profiling, and microbial species identification. Currently, the choice of labeled patterns is based on the available biochemical methods and is not necessarily optimized for the application. RESULTS: In this work, we develop a model of OGM based on information theory, which enables the design of optimal labeling patterns for specific applications and target organism genomes. We validated the model through experimental OGM on human DNA and simulations on bacterial DNA. Our model predicts up to 10-fold improved accuracy by optimal choice of labeling patterns, which may guide future development of OGM biochemical labeling methods and significantly improve its accuracy and yield for applications such as epigenomic profiling and cultivation-free pathogen identification in clinical samples. AVAILABILITY AND IMPLEMENTATION: https://github.com/yevgenin/PatternCode.

The problem of learning a channel decoder is considered for two channel models. The first model is an additive noise channel whose noise distribution is unknown and nonparametric. The learner is provided with a fixed codebook and a dataset comprised of independent samples of the noise, and is required to select a precision matrix for a nearest neighbor decoder in terms of the Mahalanobis distance. The second model is a non-linear channel with additive white Gaussian noise and unknown channel transformation. The learner is provided with a fixed codebook and a dataset comprised of independent input-output samples of the channel, and is required to select a matrix for a nearest neighbor decoder with a linear kernel. For both models, the objective of maximizing the margin of the decoder is addressed. Accordingly, for each channel model, a regularized loss minimization problem with a codebook-related regularization term and hinge-like loss function is developed, which is inspired by the support vector machine paradigm for classification problems. Expected generalization error bounds for the error probability loss function are provided for both models, under optimal choice of the regularization parameter. For the additive noise channel, a theoretical guidance for choosing the training signal-to-noise ratio is proposed based on this bound. In addition, for the non-linear channel, a high probability uniform generalization error bound is provided for the hypothesis class. For each channel, a stochastic sub-gradient descent algorithm for solving the regularized loss minimization problem is proposed, and an optimization error bound is stated. The performance of the proposed algorithms is demonstrated through several examples.

This paper introduces the informational multi-armed bandit (IMAB) model, in which at each round, a player chooses an arm, observes a symbol, and receives an unobserved reward in the form of the symbol’s self-information. Thus, the expected reward of an arm is the Shannon entropy of the probability mass function of the source that generates its symbols. The player aims to maximize the expected total reward associated with the entropy values of the arms played. Under the assumption that the alphabet size is known, two UCB-based algorithms are proposed for the IMAB model which consider the biases of the plug-in entropy estimator. The first algorithm optimistically corrects the bias term in the entropy estimation. The second algorithm relies on data-dependent confidence intervals that adapt to sources with small entropy values. Performance guarantees are provided by upper bounding the expected regret of each of the algorithms. Furthermore, in the Bernoulli case, the asymptotic behavior of these algorithms is compared to the Lai-Robbins lower bound for the pseudo regret. Additionally, under the assumption that the exact alphabet size is unknown, and instead the player only knows a loose upper bound on it, a UCB-based algorithm is proposed, in which the player aims to reduce the regret caused by the unknown alphabet size in a finite time regime. Numerical results illustrating the expected regret of the algorithms presented in the paper are provided.

Motivated by the emerging technology of oblivious processing in remote radio heads with universal decoders, we formulate and analyze in this paper a compound version of the information bottleneck problem. In this problem, a Markov chain X→Y→ Z is assumed, and the marginals PXand PYare set. The mutual information between X and Z is sought to be maximized over the choice of the conditional probability of Z given Y from a given class, under the worst choice of the joint probability of the pair (X,Y) from a different class. We provide values, bounds, and various characterizations for specific instances of this problem: the binary symmetric case, the scalar Gaussian case, the vector Gaussian case, the symmetric modulo-additive case, and the total variation constraints case. Finally, for the general case, we propose a Blahut-Arimoto type of alternating iterations algorithm to find a consistent solution to this problem.

The problem of learning a channel decoder for an unknown non-linear white Gaussian noise channel is considered. The learner is provided with a fixed codebook and a dataset comprised of n independent input-output samples of the channel, and is required to select a matrix for a nearest neighbor decoder with a linear kernel. The objective of maximizing the margin of the decoder is addressed. Accordingly, a regularized loss minimization problem with a codebook-related regularization term and a hinge-like loss function is developed, which is inspired by the support vector machine paradigm for classification problems. Expected generalization error bound for that hinge loss is provided for the solution of the regularized loss minimization, and shown to scale at a rate of O(1/(λn)), where λ is a regularization tradeoff parameter. In addition, a high probability uniform generalization error bound is provided for the hypothesis class, and shown to scale at a rate of $O(1/\sqrt n )$. A stochastic sub-gradient descent algorithm for solving the regularized loss minimization problem is proposed, and an optimization error bound is stated, which scales at a rate of $\tilde O(1/(\lambda T))$. The performance of the this algorithm is demonstrated by an example.

We consider the problem of learning a model from given data samples in which the predictor’s quality is measured by the log loss. We focus on the misspecified setting, in which the true model generating the data is chosen from a set different from the possible models that can be chosen by the learner. We establish minimax expected regret upper and lower bounds in terms of properly defined projected covering and packing entropies, and show their relation to M-projection geometric properties. We exemplify the bounds in a few settings.

We consider the DNA storage channel, in which M Deoxyribonucleic acid (DNA) molecules comprising each codeword, are stored without order, then sampled N times with replacement, and then sequenced over a discrete memoryless channel. For a constant coverage depth, M/N, and molecule length scaling Θ(log M), lower (achievability) and upper (converse) bounds on the capacity of the channel, as well as a lower (achievability) bound on the reliability function of the channel are provided. Both the lower and upper bounds on the capacity generalize a bound which was previously known to hold only for the binary symmetric sequencing channel, and only under certain restrictions on the molecule length scaling and the crossover probability parameters. When specified to binary symmetric sequencing channel, these restrictions are completely removed for the lower bound and are significantly relaxed for the upper bound. The lower bound on the reliability function is achieved under a universal decoder, and reveals that the dominant error event is that of outage – the event in which the capacity of the channel induced by the DNA molecule sampling operation does not support the target rate.

We introduce a multi-armed bandit problem with information-based rewards. At each round, a player chooses an arm, observes a symbol, and receives an unobserved reward in the form of the symbol’s self-information. The player aims to maximize the expected total reward associated with the entropy values of the arms played. We propose two algorithms based on upper confidence bounds (UCB) for this model. The first algorithm optimistically corrects the bias term in the entropy estimation. The second algorithm relies on data-dependent UCBs that adapt to sources with small entropy values. We provide performance guarantees by upper bounding the expected regret of each of the algorithms, and compare their asymptotic behavior to the Lai-Robbins lower bound. Finally, we provide numerical results illustrating the regret of the algorithms presented.

The information bottleneck (IB) problem of jointly stationary Gaussian sources is considered. A water-filling solution for the IB rate is given in terms of its SNR spectrum and whose rate is attained via frequency domain test-channel realization. A time-domain realization of the IB rate, based on linear prediction, is also proposed, which lends itself to an efficient implementation of the corresponding remote source-coding problem. A compound version of the problem is addressed, in which the joint distribution of the source is not precisely specified but rather in terms of a lower bound on the guaranteed mutual information. It is proved that a white SNR spectrum is optimal for this setting.

We consider a high-dimensional mean estimation problem over a binary hidden Markov model, which illuminates the interplay between memory in data, sample size, dimension, and signal strength in statistical inference. In this model, an estimator observes $n$ samples of a $d$-dimensional parameter vector $\theta_{*}\in\mathbb{R}^{d}$, multiplied by a random sign $ S_i $ ($1\le i\le n$), and corrupted by isotropic standard Gaussian noise. The sequence of signs $\{S_{i}\}_{i\in[n]}\in\{-1,1\}^{n}$ is drawn from a stationary homogeneous Markov chain with flip probability $\delta\in[0,1/2]$. As $\delta$ varies, this model smoothly interpolates two well-studied models: the Gaussian Location Model for which $\delta=0$ and the Gaussian Mixture Model for which $\delta=1/2$. Assuming that the estimator knows $\delta$, we establish a nearly minimax optimal (up to logarithmic factors) estimation error rate, as a function of $\|\theta_{*}\|,\delta,d,n$. We then provide an upper bound to the case of estimating $\delta$, assuming a (possibly inaccurate) knowledge of $\theta_{*}$. The bound is proved to be tight when $\theta_{*}$ is an accurately known constant. These results are then combined to an algorithm which estimates $\theta_{*}$ with $\delta$ unknown a priori, and theoretical guarantees on its error are stated.

This paper studies the problem of estimating the means $\pm\theta_{*}\in\mathbb{R}^{d}$ of a symmetric two-component Gaussian mixture $\delta_{*}\cdot N(\theta_{*},I)+(1-\delta_{*})\cdot N(-\theta_{*},I)$, where the weights $\delta_{*}$ and $1-\delta_{*}$ are unequal. Assuming that $\delta_{*}$ is known, we show that the population version of the EM algorithm globally converges if the initial estimate has non-negative inner product with the mean of the larger weight component. This can be achieved by the trivial initialization $\theta_{0}=0$. For the empirical iteration based on $n$ samples, we show that when initialized at $\theta_{0}=0$, the EM algorithm adaptively achieves the minimax error rate $\tilde{O}\Big(\min\Big\{\frac{1}{(1-2\delta_{*})}\sqrt{\frac{d}{n}},\frac{1}{\|\theta_{*}\|}\sqrt{\frac{d}{n}},\left(\frac{d}{n}\right)^{1/4}\Big\}\Big)$ in no more than $O\Big(\frac{1}{\|\theta_{*}\|(1-2\delta_{*})}\Big)$ iterations (with high probability). We also consider the EM iteration for estimating the weight $\delta_{*}$, assuming a fixed mean $\theta$ (which is possibly mismatched to $\theta_{*}$). For the empirical iteration of $n$ samples, we show that the minimax error rate $\tilde{O}\Big(\frac{1}{\|\theta_{*}\|}\sqrt{\frac{d}{n}}\Big)$ is achieved in no more than $O\Big(\frac{1}{\|\theta_{*}\|^{2}}\Big)$ iterations. These results robustify and complement recent results of Wu and Zhou (2019) obtained for the equal weights case $\delta_{*}=1/2$.

An additive noise channel is considered, in which the distribution of the noise is nonparametric and unknown. The problem of learning encoders and decoders based on noise samples is considered. For uncoded communication systems, the problem of choosing a codebook and possibly also a generalized minimal distance decoder (which is parameterized by a covariance matrix) is addressed. High probability generalization bounds for the error probability loss function, as well as for a hinge-type surrogate loss function are provided. A stochastic-gradient based alternating-minimization algorithm for the latter loss function is proposed. In addition, a Gibbs-based algorithm that gradually expurgates an initial codebook from codewords in order to obtain a smaller codebook with improved error probability is proposed, and bounds on its average empirical error and generalization error, as well as a high probability generalization bound, are stated. Various experiments demonstrate the performance of the proposed algorithms. For coded systems, the problem of maximizing the mutual information between the input and the output with respect to the input distribution is addressed, and uniform convergence bounds for two different classes of input distributions are obtained.

The DNA storage channel is considered, in which a codeword is comprised of$M$unordered DNA molecules. At reading time,$N$molecules are sampled with replacement, and then each molecule is sequenced. A coded-index concatenated-coding scheme is considered, in which the$m$th molecule of the codeword is restricted to a subset of all possible molecules (an inner code), which is unique for each$m$. The decoder has low-complexity, and is based on first decoding each molecule separately (the inner code), and then decoding the sequence of molecules (an outer code). Only mild assumptions are made on the sequencing channel, in the form of the existence of an inner code and decoder with vanishing error. The error probability of a random code as well as an expurgated code is analyzed and shown to decay exponentially with$N$. This establishes the importance of increasing the coverage depth$N/M$in order to obtain low error probability.

The DNA storage channel is considered, in which the$M$Deoxyribonucleic acid (DNA) molecules comprising each codeword are stored without order, sampled$N$times with replacement, and then sequenced over a discrete memoryless channel. For a constant coverage depth$M/N$and molecule length scaling$\Theta (\log M)$, lower (achievability) and upper (converse) bounds on the capacity of the channel, as well as a lower (achievability) bound on the reliability function of the channel are provided. Both the lower and upper bounds on the capacity generalize a bound which was previously known to hold only for the binary symmetric sequencing channel, and only under certain restrictions on the molecule length scaling and the crossover probability parameters. When specified to binary symmetric sequencing channel, these restrictions are completely removed for the lower bound and are significantly relaxed for the upper bound in the high-noise regime. The lower bound on the reliability function is achieved under a universal decoder, and reveals that the dominant error event is that ofoutage– the event in which the capacity of the channel induced by the DNA molecule sampling operation does not support the target rate.

An additive noise channel is considered, in which the noise distribution is unknown and does not known to belong to any parametric family. The problem of designing a codebook and a generalized minimal distance decoder (which is parameterized by a covariance matrix) based on samples of the noise is considered. High probability generalization bounds for the error probability loss function, as well as for a hinge-type surrogate loss function are provided. A stochastic-gradient based alternating-minimization algorithm for the latter loss function is presented. Bounds on the average empirical error and generalization error are provided for a Gibbs based algorithm that gradually expurgates codewords from a large initial codebook to obtain a smaller codebook with improved error probability.

This work studies the deviations of the error exponent of the constant composition code ensemble around its expectation, known as the error exponent of the typical random code (TRC). In particular, it is shown that the probability of randomly drawing a codebook whose error exponent is smaller than the TRC exponent is exponentially small; upper and lower bounds for this exponent are given, which coincide in some cases. In addition, the probability of randomly drawing a codebook whose error exponent is larger than the TRC exponent is shown to be double-exponentially small; upper and lower bounds to the double-exponential exponent are given. The results suggest that codebooks whose error exponent is larger than the error exponent of the TRC are extremely rare. The key ingredient in the proofs is a new large deviations result of type class enumerators with dependent variables.

The k-vectors algorithm for learning regression functions proposed here is akin to the well-known k-means algorithm. Both algorithms partition the feature space, but unlike the k-means algorithm, the k-vectors algorithm aims to reconstruct the response rather than the feature. The partitioning rule of the algorithm is based on maximizing the correlation (inner product) of the feature vector with a set of k vectors, and generates polyhedral cells, similar to the ones generated by the nearest-neighbor rule of the k-means algorithm. Similarly to k-means, the learning algorithm alternates between two types of steps. In the first type of steps, k labels are determined via a centroid-type rule (in the response space), which uses a surrogate hinge-type loss function to the mean squared error loss function. In the second type of steps, the k vectors which determine the partition are updated according to a multiclass classification rule, in the spirit of support vector machines. It is proved that both steps of the algorithm only require solving convex optimization problems, and that the algorithm is empirically consistent - as the length of the training sequence increases to infinity, fixedpoints of the empirical version of the algorithm tend to fixed points of the population version of the algorithm. Learnability of the predictor class posit by the algorithm is also established.

The distributed hypothesis testing (DHT) problem is considered, in which the joint distribution of a pair of sequences present at separated terminals, is governed by one of two possible hypotheses. The decision needs to be made by one of the terminals (the "decoder"). The other terminal (the "encoder") uses a noisy channel in order to help the decoder with the decision. This problem can be seen as a generalization of the side-information variant of the DHT problem, where the rate-limited link is replaced by a noisy channel. A recent work by Salehkalaibar and Wigger has derived an achievable Stein exponent for this problem, by employing concepts from the DHT scheme of Shimokawa et al., and from unequal error protection coding for a single special message. In this work we extend the view to a trade-off between the two error exponents, additionally building on multiple codebooks and two special messages with unequal error protection. As a by product, we also present an achievable exponent trade-off for a rate-limited link, which generalizes Shimokawa et al..

This paper contains two main contributions concerning the expurgation of hierarchical ensembles for the asymmetric broadcast channel. The first is an analysis of the optimal maximum likelihood (ML) decoders for the weak and strong user. Two different methods of code expurgation will be used, that will provide two competing error exponents. The second is the derivation of expurgated exponents under the generalized stochastic likelihood decoder (GLD). We prove that the expurgated exponents achieved for the hierarchical ensemble under GLD decoding are at least as good as the maximum between the random coding error exponents derived in an earlier work by Averbuch and Merhav (2018) and one of our ML-based expurgated exponents.

The distributed hypothesis testing problem with full side-information is studied. The trade-off (reliability function) between the two types of error exponents under limited rate is studied in the following way. First, the problem is reduced to the problem of determining the reliability function of channel codes designed for detection (in analogy to a similar result which connects the reliability function of distributed lossless compression and ordinary channel codes). Second, a single-letter random-coding bound based on a hierarchical ensemble, as well as a single-letter expurgated bound, are derived for the reliability of channel-detection codes. Both bounds are derived for a system which employs the optimal detection rule. We conjecture that the resulting random-coding bound is ensemble-tight, and consequently optimal within the class of quantization-and-binning schemes.

What is the value of one bit of side information to a guesser? We study this problem in a setup where Alice wishes to guess a uniform binary random vector, and can obtain a single bit of information from Bob, who observes this vector through a binary symmetric channel. Our goal is to charaterize the guessing efficiency, namely the maximal reduction factor in Alice's guessing-time moments obtainable by observing Bob's bit. We provide two lower bounds on the guessing efficiency by analyzing the performance of the Dictator and Majority functions, and two upper bounds via maximum entropy and Fourier-analytic/hypercontractivity arguments.

A Boolean function $g$ is said to be an optimal predictor for another Boolean function $f$ if it minimizes the probability that $f(X^n)=g(Y^n)$ among all functions, where $X^n$ is uniform over the Hamming cube and $Y^n$ is obtained from $X^n$ by independently flipping each coordinate with probability $\delta$. This paper is about self-predicting functions, which are those that coincide with their optimal predictor.

We investigate the maximal asymptotic rates of length-$n$ binary codes with VC-dimension at most $dn$ and minimum distance at least $\delta n$. Two upper bounds are obtained, one as a simple corollary of a result by Haussler and the other via a shortening approach combining the Sauer--Shelah lemma and the linear programming bound. Two lower bounds are given using Gilbert--Varshamov-type arguments over constant-weight and Markov-type sets.

We investigate the asymptotic rates of length-n binary codes with VC-dimension at most dn and minimum distance at least δn. Two upper bounds are obtained, one as a simple corollary of a result by Haussler and the other via a shortening approach combining Sauer-Shelah lemma and the linear programming bound. Two lower bounds are given using Gilbert-Varshamov type arguments over constant-weight and Markov-type sets.

The problem of modulating the value of a parameter onto a band-limited signal to be transmitted over a continuous-time, additive white Gaussian noise (AWGN) channel, and estimating this parameter at the receiver, is considered. The performance is measured by the mean power-α error (MPαE), which is defined as the worst-case αth order moment of the absolute estimation error. The optimal exponential decay rate of the MPαE as a function of the transmission time, is investigated. Two upper (converse) bounds on the MPαE exponent are derived, on the basis of known bounds for the AWGN channel of inputs with unlimited bandwidth. The bounds are computed for typical values of the error moment and the signal-to-noise ratio (SNR), and the SNR asymptotics of the different bounds are analyzed. The new bounds are compared to known converse and achievability bounds, which were derived from channel coding considerations.

A Shannon cipher system for memoryless sources in which distortion is allowed at the legitimate decoder is considered. The source is compressed using a secured rate distortion code, which satisfies a constraint on the compression rate, as well as a constraint on the exponential rate of the excess-distortion probability at the legitimate decoder. Secrecy is measured by the exponential rate of the exiguous-distortion probability at the eavesdropper, rather than by the traditional measure of equivocation. The perfect-secrecy exponent is defined as the maximal exiguous-distortion exponent achievable when the key rate is unlimited. The reproduction-based estimate exponent is defined as the maximal exiguous-distortion exponent achievable for a genie-aided eavesdropper, which knows the secret key. Under limited key rate, it is proved that the maximal achievable exiguous-distortion exponent is equal to the minimum between the key rate plus the reproduction-based estimate exponent, and the perfect-secrecy exponent. The result is generalized to a fairly general class of variable key-rate and coding-rate codes.

It was previously shown by Hashimoto that Forney's optimal erasure decoder can be significantly simplified, in the sense that a simplified decoder achieves the same random coding bounds, for the ensemble of independent and identically distributed codewords. In this paper, the analysis of simplified decoders is refined and generalized in several aspects. First, tighter random coding bounds for simplified decoders are derived, which equal the exact exponential behavior of the fixed composition ensemble average. Second, the exponential bounds are valid both in the erasure mode and in the list mode. Third, the analysis pertains to a rather general class of simplified decoders, including the case of mismatch in the threshold function of the decoder. Fourth, expurgated exponents, which are larger than the random coding exponents at low rates, are shown to be achievable using a significantly simpler decoder than Forney's optimal decoder. It is shown numerically that, from the aspect of exact random coding exponents, a decoder in the spirit of Hashimoto's is as good as Forney's in the erasure mode, as well as in the list mode.

The problem of block-coded communication where in each block the channel law belongs to one of two disjoint sets is considered. The decoder is aimed to decode only messages that have undergone a channel from one of the sets, and thus has to detect the set which contains the underlying channel. The simplified case where each of the sets is a singleton is studied first. The decoding error, false alarm, and misdetection probabilities of a given code are defined, and the optimum detection/decoding rule in a generalized Neyman-Pearson sense is derived. Sub-optimal detection/decoding rules are also introduced which are simpler to implement. Then, various achievable bounds on the error exponents are derived, including the exact single-letter characterization of the random coding exponents for the optimal detector/decoder. The random coding analysis is then extended to general sets of channels, and an asymptotically optimal detector/decoder under a worst case formulation of the error probabilities is derived, as well as its random coding exponents. The case of a pair of binary symmetric channels is discussed in detail.

Suppose Ynis obtained by observing a uniform Bernoulli random vector Xnthrough a binary symmetric channel. Courtade and Kumar asked how large the mutual information between Yn and a Boolean function b(Xn) could be, and conjectured that the maximum is attained by a dictator function. An equivalent formulation of this conjecture is that dictator minimizes the prediction cost in a sequential prediction of Ynunder logarithmic loss, given b(Xn). In this paper, we study the question of minimizing the sequential prediction cost under a different (proper) loss function-the quadratic loss. In the noiseless case, we show that majority asymptotically minimizes this prediction cost among all Boolean functions. We further show that for weak noise, majority is better than a dictator, and that for a strong noise dictator outperforms majority. We conjecture that for quadratic loss, there is no single sequence of Boolean functions that is simultaneously (asymptotically) optimal at all noise levels.

A Shannon cipher system for memoryless sources is considered, in which distortion is allowed at the legitimate decoder. The source is compressed using a rate distortion code secured by a shared key, which satisfies a constraint on the compression rate, as well as a constraint on the exponential rate of the excess-distortion probability at the legitimate decoder. Secrecy is measured by the exponential rate of the exiguous-distortion probability at the eavesdropper, rather than by the traditional measure of equivocation. The perfect secrecy exponent is defined as the maximal exiguous-distortion exponent achievable when the key rate is unlimited. Under limited key rate, it is proved that the maximal achievable exiguous-distortion exponent is equal to the minimum between the average key rate and the perfect secrecy exponent, for a fairly general class of variable key rate codes.

Suppose Ynis obtained by observing a uniform Bernoulli random vector Xnthrough a binary symmetric channel. Courtade and Kumar asked how large the mutual information between Y n and a Boolean function b(Xn) could be, and conjectured that the maximum is attained by the dictator function. An equivalent formulation of this conjecture is that dictator minimizes the prediction cost in sequentially predicting Ynunder logarithmic loss, given b(Xn). In this paper, we study the question of minimizing the sequential prediction cost under a different (proper) loss function - the quadratic loss. In the noiseless case, we show that majority asymptotically minimizes this prediction cost among all Boolean functions. We further show that for weak noise, majority is better than dictator, and that for strong noise dictator outperforms majority. We conjecture that for quadratic loss, there is no single Boolean function that is simultaneously optimal at all noise levels.

We study the problem of universal decoding for unknown discrete memoryless channels in the presence of erasure/list option at the decoder, in the random coding regime. In particular, we harness a universal version of Forney's classical erasure/list decoder developed in earlier studies, which is based on the competitive minimax methodology, and guarantees universal achievability of a certain fraction of the optimum random coding error exponents. In this paper, we derive an exact single-letter expression for the maximum achievable fraction. Examples are given in which the maximal achievable fraction is strictly less than unity, which imply that, in general, there is no universal erasure/list decoder, which achieves the same random coding error exponents as the optimal decoder for a known channel. This is in contrast to the situation in ordinary decoding (without the erasure/list option), where optimum exponents are universally achievable, as is well known. It is also demonstrated that previous lower bounds derived for the maximal achievable fraction are not tight in general. We then analyze a generalized random coding ensemble, which incorporate a training sequence, in conjunction with a suboptimal practical decoder (“plug-in” decoder), which first estimates the channel using the available training sequence, and then decodes the remaining symbols of the codeword using the estimated channel. One of the implications of our results is setting the stage for a reasonable criterion of optimal training. Finally, we compare the performance of the “plug-in” decoder and the universal decoder, in terms of the achievable error exponents, and show that the latter is noticeably better than the former.

We analyze the optimal trade-off between the error exponent and the excess-rate exponent for variable-rate Slepian-Wolf codes. We first derive upper (converse) bounds on the optimal error and excess-rate exponents, and then lower (achievable) bounds, via a simple class of variable-rate codes which assign the same rate to all source blocks of the same type class. The resulting Slepian-Wolf codes bridge between the two extremes of fixed-rate coding, which has minimal error exponent and maximal excess-rate exponent, and average-rate coding, which has maximal error exponent and minimal excess-rate exponent.

We consider the problem of erasure/list decoding using certain classes of simplified decoders. Specifically, we assume a class of erasure/list decoders, such that a codeword is in the list if its likelihood is larger than a threshold. This class of decoders both approximates the optimal decoder of Forney, and also includes the following simplified subclasses of decoding rules: The first is a function of the output vector only, but not the codebook (which is most suitable for high rates), and the second is a scaled version of the maximum likelihood decoder (which is most suitable for low rates). We provide singleletter expressions for the exact random coding exponents of any decoder in these classes, operating over a discrete memoryless channel. For each class of decoders, we find the optimal decoder within the class, in the sense that it maximizes the erasure/list exponent, under a given constraint on the error exponent. We establish the optimality of the simplified decoders of the first and second kind for low and high rates, respectively.

We study the problem of universal decoding for unknown discrete memoryless channels in the presence of erasure/list option at the decoder, in the random coding regime. Specifically, we harness a universal version of Forney's classical erasure/list decoder developed in earlier studies, which is based on the competitive minimax methodology, and guarantees universal achievability of a certain fraction of the optimum random coding error exponents. In this paper, we derive an exact single-letter expression for the maximum achievable fraction. Examples are given in which the maximal achievable fraction is strictly less than unity, which imply that, in general, there is no universal erasure/list decoder which achieves the same random coding error exponents as the optimal decoder for a known channel. This is in contrast to the situation in ordinary decoding (without the erasure/list option), where optimum exponents are universally achievable, as is well known. It is also demonstrated that previous lower bounds derived for the maximal achievable fraction are not tight in general.

We analyze the optimal tradeoff between the error exponent and the excess-rate exponent for variable-rate Slepian-Wolf codes. In particular, we first derive upper (converse) bounds on the optimal error and excess-rate exponents, and then lower (achievable) bounds, via a simple class of variable-rate codes which assign the same rate to all source blocks of the same type class. Then, using the exponent bounds, we derive bounds on the optimal rate functions, namely, the minimal rate assigned to each type class, needed in order to achieve a given target error exponent. The resulting excess-rate exponent is then evaluated. Iterative algorithms are provided for the computation of both bounds on the optimal rate functions and their excess-rate exponents. The resulting Slepian-Wolf codes bridge between the two extremes of fixed-rate coding, which has minimal error exponent and maximal excess-rate exponent, and average-rate coding, which has maximal error exponent and minimal excess-rate exponent.

We analyze the asymptotic performance of ensembles of random binning Slepian-Wolf codes, where each type class of the source might have a different coding rate. In particular, we first provide the exact encoder excess rate exponent as well as the decoder error exponent. Then, using the error exponent expression, we determine the optimal rate function, namely, the minimal rate for each type class needed to satisfy a given requirement on the decoder error exponent. The resulting excess rate exponent is then evaluated for the optimal rate function. Alternating minimization algorithms are provided for the calculation of both the optimal rate function and the excess rate exponent. It is thus exemplified that, compared to fixed-rate coding, larger error exponents may be achieved using variable-rate coding, at the price of a finite excess rate exponent.

We consider the problem of slotted asynchronous coded communication, where in each time frame (slot), the transmitter is either silent or transmits a codeword from a given (randomly selected) codebook. The task of the decoder is to decide whether transmission has taken place, and if so, to decode the message. We derive the optimum detection/decoding rule in the sense of the best trade-off among the probabilities of decoding error, false alarm, and misdetection. For this detection/decoding rule, we then derive single-letter characterizations of the exact exponential rates of these three probabilities for the average code in the ensemble. It is shown that previously suggested decoders care in general strictly sub-optimal.

We consider the problem of coded communication, where in each time frame, the transmitter is either silent or transmits a codeword from a given (randomly selected) codebook. The task of the decoder is to decide whether transmission has taken place, and if so, to decode the message. We derive the optimum detection/decoding rule in the sense of the best tradeoff among the probabilities of decoding error, false alarm, and misdetection. For this detection/decoding rule, we then derive single-letter characterizations of the exact exponential rates of these probabilities for the average code in the ensemble. It is shown that previously proposed decoders are in general strictly suboptimal.

In a previous work, a universal decoder in a competitive minimax sense was developed for unknown block fading linear white Gaussian channels. For a given codebook (with finite blocklength), a high SNR optimal metric for the decoder was found, whose direct calculation requires solving a non-convex optimization problem and may be formidable. In this paper, the metric calculation problem is facilitated by semidefinite programming, which leads to a low-complexity approximation for the metric. The competitive minimax performance of the optimal decoder (i.e., its worst case power loss compared to the maximum likelihood decoder, which has full knowledge of the channel) is evaluated, and upper lower bounds are derived for the performance evaluation of non-optimal decoders - the training sequence and the generalized likelihood test decoders.

We address the problem of communicating over an unknown linear fading channel with additive white Gaussian noise, in the high SNR regime. A block fading model is adopted where the channel fading vector is unknown, yet assumed constant during the block. For a given codebook, a competitive minimax criterion is used to find a decoder ignorant of the specific channel fading prevailing, yet its performance, relative to the Maximum Likelihood decoder, has the best worst case. For a codebook with two codewords, the decoder is found explicitly, and a numerical method is described to find its performance.