Collecting large quantities of high-quality data can be prohibitively expensive or impractical, and a bottleneck in machine learning. One may instead augment a small set of $n$ data points from the target distribution with data from more accessible sources, e.g. data collected under different circumstances or synthesized by generative models. We refer to such data as `surrogate data'. We study a weighted empirical risk minimization (ERM) approach for integrating surrogate data into training. We analyze mathematically this method under several classical statistical models, and validate our findings empirically on datasets from different domains. Our main findings are: $(i)$ Integrating surrogate data can significantly reduce the test error on the original distribution. Surprisingly, this can happen even when the surrogate data is unrelated to the original ones. We trace back this behavior to the classical Stein's paradox. $(ii)$ In order to reap the benefit of surrogate data, it is crucial to use optimally weighted ERM. $(iii)$ The test error of models trained on mixtures of real and surrogate data is approximately described by a scaling law. This scaling law can be used to predict the optimal weighting scheme, and to choose the amount of surrogate data to add.

A low-complexity coding scheme is developed to achieve the rate region of maximum likelihood decoding for interference channels. As in the classical rate-splitting multiple access scheme by Grant, Urbanke, and Whiting, the proposed coding scheme uses superposition of multiple codewords with successive cancellation decoding, which can be implemented using standard point-to-point encoders and decoders. Unlike rate-splitting multiple access, which is not rate-optimal for multiple receivers, the proposed coding scheme transmits codewords over multiple blocks in a staggered manner and recovers them successively over sliding decoding windows, achieving the single-stream optimal rate region as well as the more general Han–Kobayashi inner bound for the two-user interference channel. The feasibility of this scheme in practice is verified by implementing it using commercial channel codes over the two-user Gaussian interference channel.

We study polar coding for stochastic processes with memory. For example, a process may be defined by the joint distribution of the input and output of a channel. The memory may be present in the channel, the input, or both. We show that the ψ-mixing processes polarize under the standard Arıkan transform, under a mild condition. We further show that the rate of polarization of the low-entropy synthetic channels is roughly O(2-√N), where N is the blocklength. That is, essentially, the same rate as in the memoryless case.

We introduce a new approach to proving that a sequence of deterministic linear codes achieves capacity on an erasure channel under maximum a posteriori decoding. Rather than relying on the precise structure of the codes, our method exploits code symmetry. In particular, the technique applies to any sequence of linear codes where the blocklengths are strictly increasing, the code rates converge, and the permutation group of each code is doubly transitive. In other words, we show that symmetry alone implies near-optimal performance. An important consequence of this result is that a sequence of Reed-Muller codes with increasing block length and converging rate achieves capacity. This possibility has been suggested previously in the literature but it has only been proven for cases where the limiting code rate is 0 or 1. Moreover, these results extend naturally to all affine-invariant codes and, thus, to extended primitive narrow-sense BCH codes. This also resolves, in the affirmative, the existence question for capacity-achieving sequences of binary cyclic codes. The primary tools used in the proof are the sharp threshold property for symmetric monotone Boolean functions and the area theorem for extrinsic information transfer functions.

We study polar coding over channels and sources with memory. We show that ψ-mixing processes polarize under the standard transform, and that the rate of polarization to deterministic distributions is roughly O(2-√N) as in the memoryless case, where N is the blocklength. This implies that the error probability guarantees of polar channel and source codes extend to a large class of models with memory, including finite-order Markov sources and finite-state channels.

We introduce a new approach to proving that a sequence of deterministic linear codes achieves capacity on an erasure channel under maximum a posteriori decoding. Rather than relying on the precise structure of the codes, our method exploits code symmetry. In particular, the technique applies to any sequence of linear codes where the block lengths are strictly increasing, the code rates converge, and the permutation group of each code is doubly transitive. In a nutshell, we show that symmetry alone implies near-optimal performance.

A method to polarize channels universally is introduced. The method is based on combining channels of unequal capacities in each polarization step, as opposed to the standard method of combining identical channels. The locations of the good and bad channels that emerge upon polarization are only a function of the polar transform chosen, and are otherwise independent of the channel being polarized. This yields a simple method to design universal polar codes for discrete memoryless channels. It is also shown that the less noisy ordering of channels is preserved under polarization, and thus, a good polar code for a given channel will perform well over a less noisy one.

Humans have $23$ pairs of homologous chromosomes. The homologous pairs are almost identical pairs of chromosomes. For the most part, differences in homologous chromosome occur at certain documented positions called single nucleotide polymorphisms (SNPs).A haplotype of an individual is the pair of sequences of SNPs on the two homologous chromosomes. In this paper, we study the problem of inferring haplotypes of individuals from mate-pair reads of their genome. We give a simple formula for the coverage needed for haplotype assembly, under a generative model. The analysis here leverages connections of this problem with decoding convolutional codes.

We study the information theoretic limits of DNA assembly from paired reads. Each paired read consists of two subsequences of the DNA separated by a certain genomic distance. We show that this problem can be naturally cast as assembly of 2-D jigsaw puzzles. Using this representation, a necessary condition for assembly is derived and is shown to be nearly achieved on several probabilistic genome models.

A method to polarize channels universally is introduced. The method is based on combining two distinct channels in each polarization step as opposed to Arikan's original method of combining identical channels. This creates an equal number of only two types of channels, one of which becomes progressively better as the other becomes worse. The locations of the good channels are independent of the underlying channel, guaranteeing universality at rate 1/2. The method is generalized to construct codes of arbitrary rates.

A polar coding scheme for interference networks is introduced. The scheme builds on Arikan's monotone chain rules for multiple access channels and a method by Hassani and Urbanke to “align” two incompatible polarization processes. It achieves the Han-Kobayashi inner bound for two-user interference channels and generalizes to interference networks.

Superposition coding with successive cancellation decoding for interference channels is investigated as a low-complexity alternative to the rate-optimal simultaneous decoding. It is shown that regardless of the number of superposition layers and the code distribution of each layer, the standard rate-splitting scheme by Grant, Rimoldi, Urbanke, and Whiting for multiple access channels fails to achieve the simultaneous decoding inner bound on the capacity region for interference channels. A new coding scheme is proposed that uses coding over multiple blocks and sliding-window decoding. With at most two superposition layers, this scheme achieves the simultaneous decoding inner bound for any two-user-pair interference channels without using high-complexity simultaneous multiuser sequence detection. The proposed coding scheme can be also extended to achieve the performance of simultaneous decoding for general interference networks, including the Han-Kobayashi inner bound.

A new inner bound on the capacity region of the general index coding problem is established. Unlike most existing bounds that are based on graph theoretic or algebraic tools, the bound relies on a random coding scheme and optimal decoding, and has a simple polymatroidal single-letter expression. The utility of the inner bound is demonstrated by examples that include the capacity region for all index coding problems with up to five messages (there are 9846 nonisomorphic ones).

The problem of achieving the secrecy capacity of wiretap channels explicitly and with low complexity has been open since the work of Wyner in 1975. Recently, Mahdavifar and Vardy presented a solution to this problem, based on polar codes, for the class of symmetric and degraded wiretap channels. Their polar coding scheme achieves both security and reliability under the weak security criterion, but does not guarantee reliability under the strong security criterion. The main difficulty in providing both strong security and reliability using polar codes is the existence of a small number of bit-channels that are both unreliable and unsecure. In this paper, a multi-block polar coding scheme that resolves this difficulty is presented. It is shown that this coding scheme achieves the secrecy capacity of symmetric degraded wiretap channels while guaranteeing both reliability and strong security.

There are two variants of superposition coding schemes. Cover's original superposition coding scheme has code clouds of identical shape, while Bergmans's superposition coding scheme has code clouds of independently generated shapes. These two schemes yield identical achievable rate regions in several scenarios, such as the capacity region for degraded broadcast channels. This paper shows that under optimal decoding, these two superposition coding schemes can result in different rate regions. In particular, it is shown that for the two-receiver broadcast channel, Cover's scheme achieves a larger rate region than Bergmans's scheme in general.

Arikan's polar coding method is extended to two-user multiple-access channels. It is shown that if the two users of the channel use Arikan's construction, the resulting channels will polarize to one of five possible extremals, on each of which uncoded transmission is optimal. The sum rate achieved by this coding technique is the one that corresponds to uniform input distributions. The encoding and decoding complexities and the error performance of these codes are as in the single-user case: O(nlogn) for encoding and decoding, and o(2-n1/2-ε) for the block error probability, where n is the blocklength.

An open problem in polarization theory is whether all memoryless channels and sources with composite (that is, non-prime) alphabet sizes can be polarized with deterministic, Arıkan-like methods. This paper answers the question in the affirmative by giving a method to polarize all discrete memoryless channels and sources. The method yields codes that retain the low encoding and decoding complexity of binary polar codes.

It is shown that Arikan's recursive construction polarizes all q-ary, κ-th order ergodic Markov processes for all κ and prime q.

It is shown that given two copies of a q-ary input channel W, where q is prime, it is possible to create two channels W−and W+whose symmetric capacities satisfy I(W−) ≤ I(W) ≤ I(W+), where the inequalities are strict except in trivial cases. This leads to a simple proof of channel polarization in the q-ary case.

Polar codes were recently introduced by Arikan. They achieve the symmetric capacity of arbitrary binary-input discrete memoryless channels under a low complexity successive cancellation decoding scheme. The original polar code construction is closely related to the recursive construction of Reed-Muller codes and is based on the 2 × 2 matrix [1 0 : 1 1]. It was shown by Arikan Telatar that this construction achieves an error exponent of 1/2, i.e., that for sufficiently large blocklengths the error probability decays exponentially in the square root of the blocklength. It was already mentioned by Arikan that in principle larger matrices can be used to construct polar codes. In this paper, it is first shown that any ℓ × ℓ matrix none of whose column permutations is upper triangular polarizes binary-input memoryless channels. The exponent of a given square matrix is characterized, upper and lower bounds on achievable exponents are given. Using these bounds it is shown that there are no matrices of size smaller than 15×15 with exponents exceeding 1/2. Further, a general construction based on BCH codes which for large I achieves exponents arbitrarily close to 1 is given. At size 16 × 16, this construction yields an exponent greater than 1/2.

A generalization of Arıkan's polar code construction using transformations of the form G⊗nwhere G is an ℓ × ℓ matrix is considered. Necessary and sufficient conditions are given for these transformations to ensure channel polarization. It is shown that a large class of such transformations polarize binary-input memoryless channels.

Polar codes were recently introduced by Arıkan. They achieve the symmetric capacity of arbitrary binary-input discrete memoryless channels under a low complexity successive cancellation decoding strategy. The original polar code construction is closely related to the recursive construction of Reed-Muller codes and is based on the 2 × 2 matrix of the given equation. It was shown by Arıkan and Telatar that this construction achieves an error exponent of 1/2, i.e., that for sufficiently large blocklengths the error probability decays exponentially in the square root of the length. It was already mentioned by Arıkan that in principle larger matrices can be used to construct polar codes. A fundamental question then is to see whether there exist matrices with exponent exceeding 1/2. We characterize the exponent of a given square matrix and derive upper and lower bounds on achievable exponents. Using these bounds we show that there are no matrices of size less than 15 with exponents exceeding 1/2. Further, we give a general construction based on BCH codes which for large matrix sizes achieves exponents arbitrarily close to 1 and which exceeds 1/2 for size 16.

The Chong—Motani—Garg inner bound for the two-user interference channel is generalized to a special case of many-user interference channels. By taking a geometric viewpoint and using rate-splitting arguments, it is shown that all operating points in the Chong—Motani—Garg inner bound allow decoding by successive cancellation at all receivers.