It is shown that when Arıkan's n-level polarization transformation is applied to the binary erasure channel, each of the resulting individual 2nsubchannels has a sharp threshold, for sufficiently large n.

Sponsored search has been the major source of revenue for commercial web search engines. It is crucial for a sponsored search engine to retrieve ads that are relevant to user queries to attract clicks as advertisers only pay when their ads get clicked. Retrieving relevant ads for a query typically involves in first matching related ads to the query and then filtering out irrelevant ones. Both require understanding the semantic relationship between a query and an ad. In this work, we propose a novel embedding of queries and ads in sponsored search. The query embeddings are generated from constituent word n-gram embeddings that are trained to optimize an event level word2vec objective over a large volume of search data. We show through a query rewriting task that the proposed query n-gram embedding model outperforms the state-of-the-art word embedding models for capturing query semantics. This allows us to apply the proposed query n-gram embedding model to improve query-ad matching and relevance in sponsored search. First, we use the similarity between a query and an ad derived from the query n-gram embeddings as an additional feature in the query-ad relevance model used in Yahoo Search. We show through online A/B test that using the new relevance model to filter irrelevant ads offline leads to 0.47% CTR and 0.32% revenue increase. Second, we propose a novel online query to ads matching system, built on an open-source big-data serving engine [30], using the learned query n-gram embeddings. Online A/B test shows that the new matching technique increases the search revenue by 2.32% as it significantly increases the ad coverage for tail queries.

It is shown that when Ankan's n-level polarization transformation is applied to the binary erasure channel, each of the resulting individual 2nsubchannels has a sharp threshold, for sufficiently large n.

Word2vec is a popular family of algorithms for unsupervised training of dense vector representations of words on large text corpuses. The resulting vectors have been shown to capture semantic relationships among their corresponding words, and have shown promise in reducing a number of natural language processing (NLP) tasks to mathematical operations on these vectors. While heretofore applications of word2vec have centered around vocabularies with a few million words, wherein the vocabulary is the set of words for which vectors are simultaneously trained, novel applications are emerging in areas outside of NLP with vocabularies comprising several 100 million words. Existing word2vec training systems are impractical for training such large vocabularies as they either require that the vectors of all vocabulary words be stored in the memory of a single server or suffer unacceptable training latency due to massive network data transfer. In this paper, we present a novel distributed, parallel training system that enables unprecedented practical training of vectors for vocabularies with several 100 million words on a shared cluster of commodity servers, using far less network traffic than the existing solutions. We evaluate the proposed system on a benchmark data set, showing that the quality of vectors does not degrade relative to non-distributed training. Finally, for several quarters, the system has been deployed for the purpose of matching queries to ads in Gemini, the sponsored search advertising platform at Yahoo, resulting in significant improvement of business metrics.

Sponsored search represents a major source of revenue for web search engines. The advertising model brings a unique possibility for advertisers to target direct user intent communicated through a search query, usually done by displaying their ads alongside organic search results for queries deemed relevant to their products or services. However, due to a large number of unique queries, it is particularly challenging for advertisers to identify all relevant queries. For this reason search engines often provide a service of advanced matching, which automatically finds additional relevant queries for advertisers to bid on. We present a novel advance match approach based on the idea of semantic embeddings of queries and ads. The embeddings were learned using a large data set of user search sessions, consisting of search queries, clicked ads and search links, while utilizing contextual information such as dwell time and skipped ads. To address the large-scale nature of our problem, both in terms of data and vocabulary size, we propose a novel distributed algorithm for training of the embeddings. Finally, we present an approach for overcoming a cold-start problem associated with new ads and queries. We report results of editorial evaluation and online tests on actual search traffic. The results show that our approach significantly outperforms baselines in terms of relevance, coverage and incremental revenue. Lastly, as part of this study, we open sourced query embeddings that can be used to advance the field.

We initiate a study of constrained codes in which any codeword can be transformed into any other codeword by a sequence of single symbol changes such that the intermediate words (after each symbol change) all satisfy the underlying constraint. We shall refer to a set of constrained words with this property as being Hamming connected. Hamming connected constrained codes might be useful for encoding data in storage media when a constraint must be met upon writing data, as might be the case in some emerging storage technologies. The stated property would permit overwriting encoded data without violating the constraint during intermediate writes. We study the Hamming connectedness of (d, k)-run-length limited constraints and a few other special cases. We also consider the decidability of Hamming connectedness for finite memory constraints.

Given two arbitrary sequences of denoisers for block lengths tending to infinity we ask if it is possible to construct a third sequence of denoisers with an asymptotically vanishing (in block length) excess expected loss relative to the best expected loss of the two given denoisers for all clean channel input sequences. As in the setting of DUDE [1], which solves this problem when the given denoisers are sliding block denoisers, the construction is allowed to depend on the two given denoisers and the channel transition probabilities. We show that under certain restrictions on the two given denoisers the problem can be solved using a straightforward application of a known loss estimation paradigm. We then show by way of a counter-example that the loss estimation approach fails in the general case. Finally, we show that for the binary symmetric channel, combining the loss estimation with a randomization step leads to a solution to the stated problem under no restrictions on the given denoisers.

We present a framework based on covering codes over the set of binary n-words for coding data into n × n binary arrays with a prescribed upper bound on the Hamming weight (i.e., number of 1's) in each row and column. We obtain previously presented schemes for the case when the upper bound is n/2 as special cases of this framework and we study also another potentially practically relevant specialization when the underlying covering code is the first-order Reed-Muller code. Like the previous schemes for the n/2 case, the proposed framework and schemes may have applications in improving the performance of a next-generation memory based on programmable resistive devices arranged in a crossbar architecture.

We propose a sequence of universal denoisers motivated by the goal of extending the notion of twice-universality from universal data compression theory to the sliding window denoising setting. Given a sequence lengthnand a denoiser, thekth-order regret of the latter is the maximum excess expected denoising loss relative to sliding window denoisers with window length 2k+1, where, for a given clean sequence, the expectation is over all channel realizations and the maximum is over all clean sequences of lengthn. We define the twice-universality penalty of a denoiser as its excesskth-order regret when compared to a bound on thekth-order regret of the denoising algorithm DUDE with parameterk, and we are interested in denoisers with a negligible penalty for allksimultaneously. We consider a class of denoisers that apply one of a number of constituent denoisers based on minimizing an estimated denoising loss and establish a formal relationship between the error in the estimated denoising loss and the twice-universality penalty of the resulting denoiser. Given a sequence of window parameterskn, increasing innsufficiently fast, we use this approach to construct and analyze a specific sequence of denoisers that achieves a much smaller twice-universality penalty forkknthan the sequence of DUDE denoisers with parameterkn.

We define a g-ary antipodal matching to be a perfect matching in the bipartite graph with vertices corresponding to words of length ℓ over the integer alphabet Q = {0, 1, ..., q -1}, wherein the left and right vertices are those with respective component sums greater and smaller than ℓ(q -1)/2, and wherein two vertices are connected by an edge if one of the corresponding words dominates the other. We present two different constructions of efficiently computable g-ary antipodal matchings. We then show how such matchings can be used for encoding arbitrary data into n × n arrays over the alphabet Q all of whose row and column sums are at most n(q -1)/2. Such encoders might be useful for mitigating parasitic currents in a next generation memory technology based on crossbar arrays of resistive devices.

Minimax prediction of binary sequences is investigated for cases in which the predictor is forced to issue a piecewise constant prediction. The minimax strategy is characterized for Hamming loss whereas, for logarithmic loss, an asymptotically minimax strategy which achieves the leading term of the asymptotic minimax redundancy, is proposed. The average redundancy case is also analyzed for i.i.d. distributions. The piecewise constant prediction paradigm may be of relevance to resource constrained settings.

Let ${\mathcal{A}}_n$ be the set of all $n \times n$ binary matrices in which the number of $1$'s in each row and column is at most $n/2$. We show that $|{\mathcal{A}}_n| = 2^{n^2 - \rho n + \delta \sqrt{n}} \cdot n^{O(1)}$, for a constant $\rho \approx 1.42515$, and $\delta = \delta(n) \approx 1.46016$ for even $n$ and $0$ otherwise.

Two low complexity coding techniques are described for mapping arbitrary data to and from$ m\times n $binary arrays in which the Hamming weight of each row (respectively, column) is at most$ n/2 $(respectively,$ m/2 $). One technique is based on flipping rows and columns of an arbitrary binary array until the Hamming weight constraint is satisfied in all rows and columns, and the other is based on a certain explicitly constructed “antipodal” matching between layers of the Boolean lattice. Both codes have a redundancy of roughly$ m{+}n $and may have applications in next generation resistive memory technologies.

Consider the set Anof all n×n binary matrices in which the number of 1's in each row and column is at most n/2. We show that the redundancy, n2- log2|An|, of this set equals ρn + o(n), for a constant ρ ≈ 1.42515.

We describe two low complexity coding techniques for mapping arbitrary data to and from n × n binary arrays in which the Hamming weight of each row and column is at most n/2. One technique is based on flipping rows and columns of an arbitrary binary array until the Hamming weight constraint is satisfied in all rows and columns, and the other is based on a certain explicitly constructed “antipodal” matching between layers of the Boolean lattice. Both codes have a redundancy of roughly 2n and may have applications in next generation resistive memory technologies.

We pose the problem of energy-optimized lossless compression and analyze a simple compression framework in which energy consumption is given by a weighted sum of two components, respectively proportional to the compression rate and to the average number of bit flips that occur in a certain hardware register. The latter component, which we term variability, is meant to serve as a proxy for the energy consumption of the computations underlying the compression step. Our results include bounds on the rate-variability tradeoff for symbol-wise compression of discrete memoryless sources and a characterization of the asymptotically optimum tradeoff between rate and variability for block-wise compression.

We present an extension of the discrete universal denoiser DUDE, specialized for the denoising of grayscale images. The original DUDE is a low-complexity algorithm aimed at recovering discrete sequences corrupted by discrete memoryless noise of known statistical characteristics. It is universal, in the sense of asymptotically achieving, without access to any information on the statistics of the clean sequence, the same performance as the best denoiser that does have access to such information. The DUDE, however, is not effective on grayscale images of practical size. The difficulty lies in the fact that one of the DUDE's key components is the determination of conditional empirical probability distributions of image samples, given the sample values in their neighborhood. When the alphabet is relatively large (as is the case with grayscale images), even for a small-sized neighborhood, the required distributions would be estimated from a large collection of sparse statistics, resulting in poor estimates that would not enable effective denoising. The present work enhances the basic DUDE scheme by incorporating statistical modeling tools that have proven successful in addressing similar issues in lossless image compression. Instantiations of the enhanced framework, which is referred to as iDUDE, are described for examples of additive and nonadditive noise. The resulting denoisers significantly surpass the state of the art in the case of salt and pepper (S&P) and M -ary symmetric noise, and perform well for Gaussian noise.

We present a new outer bound for the general two-user discrete memoryless interference channel (IFC) and use it to establish the capacity region of the binary erasure IFC, whose determination was left open in . We also show that there are essentially two deterministic binary IFCs, in addition to the binary erasure IFC, whose capacity regions are not obvious from previous results. We determine the capacity region of one of these and apply the aforementioned general outer bound to obtain the best available bound on the maximum achievable sum-rate for the other. We also show that the new general outer bound is tight for one-sided deterministic IFCs that belong to the class studied by El Gamal and Costa.

A 2-D version of the classical maximum-likelihood sequence detection (MLSD) problem is considered for a binary antipodal signal that is corrupted by linear intersymbol interference (ISI) and then passed through a memoryless channel. For 1-D signals and fixed ISI, this detection problem is well-known to be solved using the Viterbi algorithm in time complexity that is linear in the sequence length. It is shown here that, in contrast, the 2-D MLSD problem is NP hard. Specifically, a decision formulation of the problem is shown to be NP complete for a particular 2-D ISI cascaded with either of two memoryless channels: one involving errors and erasures and the other corresponding to additive white Gaussian noise. The proof for the latter case is obtained through a reduction from a still NP complete restricted version of the former. These results are applied to proving the NP completeness of multi user detection under a Toeplitz constraint-a problem known to be equivalent to a variant of 1-D MLSD with growing ISI. This proves a conjecture posed by Verdú in 1989.

The classical framework of context-tree models used in sequential decision problems such as compression and prediction is generalized to a setting in which the observations are multi-tracked, multi-sided, or multi-directional, and for which it may be beneficial to consider contexts comprised of possibly differing numbers of symbols from each track or direction. Tree representations of context sets and pruning algorithms for those trees are extended from the uni-directional setting to two directions. We further show that such tree representations do not extend, in general, tomdirections,m>; 2, and that, as a result, determining the bestm-directional context set form>; 2 may be substantially more complex than in the case ofm≤ 2. An application of the proposed pruning algorithm to denoising, wherem=2 , is presented.

We propose a new sequence of universal denoisers motivated by the goal of extending the notion of twice-universality from universal data compression theory to the sliding window denoising setting. Given a sequence length n and a denoiser, we define the twice-universality penalty of the denoiser as the worst case excess expected denoising loss relative to sliding window denoisers with window length k above and beyond the worst case excess loss of DUDE with parameter k. Given a sequence of window parameters kn, increasing in n sufficiently fast, we use loss estimators to construct a sequence of denoisers that achieves a much smaller twice-universality penalty for knthan the sequence of DUDEs with parameter kn.

Exponential error bounds for the finite-alphabet interference channel (IFC) with two transmitter–receiver pairs, are investigated under the random coding regime. Our focus is on optimum decoding, as opposed to heuristic decoding rules that have been used in previous works, like joint typicality decoding, decoding based on interference cancellation, and decoding that considers the interference as additional noise. Indeed, the fact that the actual interfering signal is a codeword and not an independent and identically distributed (i.i.d.) noise process complicates the application of conventional techniques to the performance analysis of the optimum decoder. Using analytical tools rooted in statistical physics, we derive a single-letter expression for error exponents achievable under optimum decoding and demonstrate strict improvement over error exponents obtainable using suboptimal decoding rules, but which are amenable to more conventional analysis.

The degrees-of-freedom of a K-user Gaussian interference channel (GIFC) has been defined to be the multiple of (1/2) log2P at which the maximum sum of achievable rates grows with increasing P. In this paper, we establish that the degrees-of-freedom of three or more user, real, scalar GIFCs, viewed as a function of the channel coefficients, is discontinuous at points where all of the coefficients are non-zero rational numbers. More specifically, for all K > 2, we find a class of K-user GIFCs that is dense in the GIFC parameter space for which K/2 degrees-of-freedom are exactly achievable, and we show that the degrees-of-freedom for any GIFC with non-zero rational coefficients is strictly smaller than K/2. These results are proved using new connections with number theory and additive combinatorics.

We show how to improve on the technique of approximate enumerative coding for a family of two-dimensional constraints by encoding according to lower bounds based on the worst-case behavior of certain ratios of matrix products. For the case of the two-dimensional (d = 2;infin) run-length limited (RLL) constraint, the improved approach yields a lower bound of 0.4453 on the capacity of the constraint.

We study the concentration of denoiser loss estimators, with application to the selection of denoiser parameters for a given observed sequence (in particular, the window size k of the DUDE algorithm) via minimization of the estimated loss. We show that for a loss estimator proposed earlier, it is not possible to derive strong concentration results for certain pathological input sequences. By modifying the estimator slightly we obtain a loss estimator for which the DUDE's estimated loss strongly concentrates around the true loss provided kM2k= o(n), where M is the size of the alphabet and n the sequence length. We also show that for certain channels, it is possible to estimate the best k using a combination of the two loss estimators. Moreover, for non-pathological sequences and k = o(n¿), we derive concentration results for the original loss estimator and all channels.

We consider a set of S independent encoders that must transmit a set of correlated sources through a network of noisy, independent, broadcast channels to T receivers, with no interference at the receivers. For the general problem of sending correlated sources through broadcast networks, it is known that the source-channel separation theorem breaks down and the achievable rate region as well as the proper method of coding are unknown. For our scenario, however, we establish the optimal rate region using a form of joint source-channel coding. When the optimal channel input distribution from transmitter i to receiver j is independent of j, our result has a max-flow/min-cut interpretation. Specifically, in this case, our result implies that if it is possible to send the sources to each receiver separately while ignoring the others, then it is possible to send to all receivers simultaneously.

The degrees-of-freedom of aK-user Gaussian interference channel (GIC) has been defined to be the multiple of(1/2)log2Pat which the maximum sum of achievable rates grows with increasing powerP. In this paper, we establish that the degrees-of-freedom of three or more user, real, scalar GICs, viewed as a function of the channel coefficients, is discontinuous at points where all of the coefficients are nonzero rational numbers. More specifically, for allK> 2, we find a class ofK-user GICs that is dense in the GIC parameter space for whichK/2 degrees-of-freedom are exactly achievable, and we show that the degrees-of-freedom for any GIC with nonzero rational coefficients is strictly smaller thanK/2. These results are proved using new connections with number theory and additive combinatorics.

In the spirit of results on universal compression, we compare the performance of universal denoisers on discrete memoryless channels to that of the best performance obtained by an omniscient kth-order sliding-window denoiser, namely, one that is tuned to the transmitted noiseless sequence. We show that the additional loss incurred in the worst case by any universal denoiser on a length- n sequence grows at least like Omega(ck/radicn) , where c is a constant depending on the channel parameters and the loss function. This shows that for fixed k the additional loss incurred by the Discrete Universal Denoiser (DUDE) is no larger than a constant multiplicative factor of the best possible. Furthermore, we compare universal denoisers to denoisers that are aware of the distribution of the transmitted noiseless sequence. We show that, even for this weaker target loss, for any universal denoiser there exists some distribution for the noiseless sequence corresponding to a sequence of independent and identically distributed (i.i.d.) random variables whose optimum expected loss is lower than that incurred by the universal denoiser by Omega(1/radicn).

We consider a setting relevant to the design of storage systems in which a list of defective storage blocks, as determined at manufacture time, is stored in a high-speed system controller memory to enable the efficient bypassing of defective blocks in a transparent manner, external to the system. Conventionally, such lists have been compressed losslessly to save on the cost of the controller memory. Under the assumption of a total system cost that is a linear combination of the number of storage blocks and the controller memory size, we study the potential benefits of compressing the defect list using lossy algorithms. The only restriction is that the reconstructed defect list not label any defective storage blocks as being non-defective.

Exponential error bounds for the finite-alphabet interference channel (IPC) with two transmitter-receiver pairs, are investigated under the random coding regime. Our focus is on optimum decoding, as opposed to heuristic decoding rules that have been used in previous works, like joint typicality decoding, decoding based on interference cancellation, and decoding that considers the interference as additional noise. Indeed, the fact that the actual interfering signal is a codeword and not an i.i.d. noise process complicates the performance analysis of the optimum decoder. In addition to the single-letter expressions of the error exponents derived, we also present some numerical results and discuss them.

We consider a setting relevant to the design of storage systems in which a list of defective storage blocks, as determined at manufacture time, is stored in a high-speed system controller memory to enable the efficient bypassing of defective blocks in a transparent manner, external to the system. Conventionally, such lists have been compressed losslessly to save on the cost of the controller memory. Under the assumption of a total system cost that is a linear combination of the number of storage blocks and the controller memory size, we study the potential benefits of compressing the defect list using lossy algorithms, with the restriction that the reconstructed defect list not label any defective storage blocks as being non-defective.

In many applications, an uncompressed source stream is systematically encoded by a channel code (which ignores the source redundancy) for transmission over a discrete memoryless channel. The decoder knows the channel and the code but does not know the source statistics. This paper proposes several universal channel decoders that take advantage of the source redundancy without requiring prior knowledge of its statistics.

A new outer bound for the two-user discrete memoryless interference channel is presented. This bound establishes the capacity region of the binary erasure interference channel, whose determination was left open in the work of Vanroose and van der Meulen (1991). The new bound is compared with the best known outer bounds for some additional examples. It is also shown that the new outer bound is tight for a one-sided deterministic interference channel that belongs to the class studied by El Gamal and Costa.

We present a general method for obtaining lower bounds on the capacities of two-dimensional (2-D) constraints. We apply our method to the 2-D (d=2, infin) run-length limited (RLL) constraint and obtain the best known lower bound, .4423, on the capacity of this constraint. Our lower bounds are shown to be achievable by a fixed-rate, polynomial-complexity encoding-decoding algorithm based on enumerative coding with approximate counts.

We consider the filtering problem, where a finite-alphabet individual sequence is corrupted by a discrete memoryless channel, and the goal is to causally estimate each sequence component based on the past and present noisy observations. We establish a correspondence between the filtering problem and the problem of prediction of individual sequences which leads to the following result: Given an arbitrary finite set of filters, there exists a filter which performs, with high probability, essentially as well as the best in the set, regardless of the underlying noiseless individual sequence. We use this relationship between the problems to derive a filter guaranteed of attaining the "finite-state filterability" of any individual sequence by leveraging results from the prediction problem

We consider a set of S independent encoders that must transmit a set of correlated sources through a network of noisy, independent, broadcast channels to T receivers. For the general problem of sending correlated sources through broadcast networks, it is known that the source-channel separation theorem breaks down and the achievable rate region as well as the proper method of coding are unknown. For our scenario, however, we not only establish the optimal rate region, but we show that a type of source-channel separation is possible at the transmitter, provided joint source-channel decoding is used at the receiver. Furthermore, we show that while joint source-channel encoding is unnecessary, not using joint source-channel decoding is suboptimal. Finally, when the optimal input distribution from transmitter i to receiver j is independent of j, our result has a max-flow/min-cut interpretation. Specifically, in this case our result implies that if it is possible to send the sources to each receiver separately while ignoring the others, then it is possible to send to all receivers simultaneously

Following recent work by Weissman et al, in the spirit of results on universal compression, we compare the performance of universal denoisers on discrete memoryless channels to that of the best k-th order omniscient denoiser, namely one that is tuned to the transmitted noiseless sequence. We show that the additional loss incurred in the worst case by any denoiser on a length-n sequence grows like Omega(ck/radicn), where c > 1 is a constant depending on the channel parameters and the loss function. This shows that for fixed k, the additional loss incurred by the discrete universal denoiser (DUDE) is no larger than a constant multiplicative factor of the best possible

A source of random message bits is to be embedded into a covertext modeled as a discrete memoryless source (DMS), resulting in a stegotext from which the embedded bits should be recoverable. A causal code for such a scenario consists of an encoder that generates the stegotext as a causal function of the message bits and the covertext, and a decoder that reproduces the message bits as a causal function of the stegotext. A semicausal code, on the other hand, has an encoder that is causal only with respect to the covertext, and not necessarily with respect to the message, and has a possibly noncausal decoder. We analyze the possible tradeoffs among: a) the distortion between the stegotext and the covertext, b) the compressibility of the stegotext, and c) the rate at which random bits are embedded, that are achievable with causal and semicausal codes, with and without attacks on the stegotext. We also study causal and semicausal codes for the private version of the above scenario in which the decoder has access to the covertext. Connections are made with the causal rate-distortion function of Neuhoff and Gilbert, as well as the problem of channel coding with causal side information at the transmitter analyzed by Shannon.

Let X1,...,Xnbe n independent, symmetric random variables supported on the interval [-1,1] and let Sn=sigmai=1nXibe their sum. We show that the differential entropy of Snis maximized when X1,...,Xn-1are Bernoulli taking on +1 or -1 with equal probability and Xnis uniformly distributed

We consider the problem of optimally recovering a finite-alphabet discrete-time stochastic process {X/sub t/} from its noise-corrupted observation process {Z/sub t/}. In general, the optimal estimate of X/sub t/ will depend on all the components of {Z/sub t/} on which it can be based. We characterize nontrivial situations (i.e., beyond the case where (X/sub t/,Z/sub t/) are independent) for which optimum performance is attained using "symbol-by-symbol" operations (a.k.a. "singlet decoding"), meaning that the optimum estimate of X/sub t/ depends solely on Z/sub t/. For the case where {X/sub t/} is a stationary binary Markov process corrupted by a memoryless channel, we characterize the necessary and sufficient condition for optimality of symbol-by-symbol operations, both for the filtering problem (where the estimate of X/sub t/ is allowed to depend only on {Z/sub t'/}/sub t'/spl les/t/) and the denoising problem (where the estimate of X/sub t/ is allowed dependence on the entire noisy process). It is then illustrated how our approach, which consists of characterizing the support of the conditional distribution of the noise-free symbol given the observations, can be used for characterizing the entropy rate of the binary Markov process corrupted by the binary-symmetric channel (BSC) in various asymptotic regimes. For general noise-free processes (not necessarily Markov), general noise processes (not necessarily memoryless), and general index sets (random fields) we obtain an easily verifiable sufficient condition for the optimality of symbol-by-symbol operations and illustrate its use in a few special cases. For example, for binary processes corrupted by a BSC, we establish, under mild conditions, the existence of a /spl delta//sup */>0 such that the "say-what-you-see" scheme is optimal provided the channel crossover probability is less than /spl delta//sup */. Finally, we show how for the case of a memoryless channel the large deviations (LD) performance of a symbol-by-symbol filter is easy to obtain, thus characterizing the LD behavior of the optimal schemes when these are singlet decoders (and constituting the only known cases where such explicit characterization is available).

This paper discusses the challenges of applying the DUDE framework to continuous tone images and the tools used to address these challenges. As in lossless image compression, a key component of the DUDE framework is the determination of a probability distribution for samples of the input (noisy) image, conditioned on their contexts. Thus, we can leverage from tools developed and tested in the context of lossless compression for determining such distributions, together with tools that are specific to the assumptions of the denoising application. These tools combine with the DUDE principles into a framework that yields powerful and practical denoisers for continuous tone images corrupted by a variety of noise processes.

Recent work by Ordentlich and Weissman put forth a new approach for bounding the entropy rate of a hidden Markov process via the construction of a related Markov process. We use this approach to study the behavior of the filtering error probability and the entropy rate of a hidden Markov process in the rare transitions regime. In this paper, we restrict our attention to the case of a two state Markov chain that is corrupted by a binary symmetric channel. Using this approach we recover the results on the optimal filtering error probability of Khasminskii and Zeitouni. In addition, this approach sheds light on the terms that appear in the expression for the optimal filtering error probability. We then use this approach to obtain tight estimates of the entropy rate of the process in the rare transitions regime. This leads to tight estimates on the capacity of the Gilbert-Elliot channel in the rare transitions regime

Let {Xt} be a stationary finite-alphabet Markov chain and {Zt} denote its noisy version when corrupted by a discrete memoryless channel. We present an approach to bounding the entropy rate of {Zt} by the construction and study of a related measure-valued Markov process. To illustrate its efficacy, we specialize it to the case of a BSC-corrupted binary Markov chain. The bounds obtained are sufficiently tight to characterize the behavior of the entropy rate in asymptotic regimes that exhibit a "concentration of the support". Examples include the 'high SNR', 'low SNR', 'rare spikes', and 'weak dependence' regimes. Our analysis also gives rise to a deterministic algorithm for approximating the entropy rate, achieving the best known precision-complexity tradeoff, for a significant subset of the process parameter space

The classical framework of context-tree models used in sequential decision problems such as compression and prediction is generalized to a setting in which the observations are multi-tracked or multi-directional, and for which it may be beneficial to consider contexts comprised of possibly differing numbers of symbols from each track or direction. Context set definitions, tree representations, and pruning algorithms are all extended from the classical uni-directional setting to the m-directional setting, with an emphasis on the case of m = 2. We provide a simple example suggesting that determining (pruning) the best m-directional context set for m ges 3 is substantially more complex than in the case of m = 2. After briefly describing how the multi-directional framework can be applied to universal data compression, we focus on its application to universal denoising, where we pair the proposed framework with a new technique for estimating the loss of a denoising algorithm based only on noisy observations

Given two probability distributions Q and P, let /spl par/Q-P/spl par//sub 1/ and D(Q/spl par/P), respectively, denote the L/sub 1/ distance and divergence between Q and P. We derive a refinement of Pinsker's inequality of the form D(Q/spl par/P)/spl ges/c(P)/spl par/Q-P/spl par//sub 1//sup 2/ and characterize the best P-dependent factor c(P). We apply the refined inequality to large deviations and measure concentration.

Let X = (X/sub 1/,...) be a stationary ergodic finite-alphabet source, X/sup n/ denote its first n symbols, and Y/sup n/ be the codeword assigned to X/sup n/ by a lossy source code. The empirical kth-order joint distribution Q/spl circ//sup k/[X/sup n/,Y/sup n//spl rceil/(x/sup k/,y/sup k/) is defined as the frequency of appearances of pairs of k-strings (x/sup k/,y/sup k/) along the pair (X/sup n/,Y/sup n/). Our main interest is in the sample behavior of this (random) distribution. Letting I(Q/sup k/) denote the mutual information I(X/sup k/;Y/sup k/) when (X/sup k/,Y/sup k/)/spl sim/Q/sup k/ we show that for any (sequence of) lossy source code(s) of rate /spl les/R lim sup/sub n/spl rarr//spl infin//(1/k)I(Q/spl circ//sup k/[X/sup n/,Y/sup n//spl rfloor/) /spl les/R+(1/k)H (X/sub 1//sup k/)-H~(X) a.s. where H~(X) denotes the entropy rate of X. This is shown to imply, for a large class of sources including all independent and identically distributed (i.i.d.). sources and all sources satisfying the Shannon lower bound with equality, that for any sequence of codes which is good in the sense of asymptotically attaining a point on the rate distortion curve Q/spl circ//sup k/[X/sup n/,Y/sup n//spl rfloor//spl rArr//sup d/P(X/sup k/,Y~/sup k/) a.s. whenever P(/sub X//sup k//sub ,Y//sup k/) is the unique distribution attaining the minimum in the definition of the kth-order rate distortion function. Consequences of these results include a new proof of Kieffer's sample converse to lossy source coding, as well as performance bounds for compression-based denoisers.

A discrete denoising algorithm estimates the input sequence to a discrete memoryless channel (DMC) based on the observation of the entire output sequence. For the case in which the DMC is known and the quality of the reconstruction is evaluated with a given single-letter fidelity criterion, we propose a discrete denoising algorithm that does not assume knowledge of statistical properties of the input sequence. Yet, the algorithm is universal in the sense of asymptotically performing as well as the optimum denoiser that knows the input sequence distribution, which is only assumed to be stationary. Moreover, the algorithm is universal also in a semi-stochastic setting, in which the input is an individual sequence, and the randomness is due solely to the channel noise. The proposed denoising algorithm is practical, requiring a linear number of register-level operations and sublinear working storage size relative to the input data length.

In the discrete filtering problem, a data sequence over a finite alphabet is assumed to be corrupted by a discrete memoryless channel. The goal is to reconstruct the clean sequence, with as high a fidelity as possible, by way of causal processing of the noisy sequence alone, with the reconstruction at time t depending only on noisy observations occurring no later than t. A universal version of this problem in which no assumptions are made about the distribution of the clean data, which may even be nonstochastic is studied. Using techniques from universal data compression, in particular, the incremental parsing rule of LZ78, and derives a practical and efficient algorithms for the universal filtering of discrete sources. A finite-memory filter of order k has the property that the reconstruction at any time t is a time-invariant function only of noisy observations occurring between times t-k and t, inclusive. The universal filtering algorithms perform essentially as well, in an expected sense (with respect to the noise process), as the best finite-memory filter of any fixed order, determined with full knowledge of the actual clean data sequence, for all such data sequences. Also consider more general finite-state filters and show that any such filter is arbitrarily well approximated by a finite-memory filter of growing order, thereby establishing the universality of the proposed algorithms with respect to this larger class. This result can be viewed as the filtering analogue of the well known optimality of LZ78 relative to the class of finite-state compressors.

The problem of joint compression and information embedding under various causality restrictions on the encoder and decoder is studied in this paper. The minimax game between causal encoder-decoder pairs and causal attackers is characterized by a single-letter formula due to the fact that there is a saddle-point where both parties apply memoryless strategies. Main results are based on semicausal codes where causality is required only with respect to the covertext source. Semicausal embedding is related to communication with causal side information at the transmitter. The results are also extended to the case of attacks, and single-letter formulas are provided for the characterization of the minimax game between the communicators and the attacker.

This paper describes the channel decoding of systematically encoded unknown redundant sources. The redundancy of the data is known at the decoder and the channel decoder incorporates the statistics of the data to enhance the performance. The practical decoders are designed which takes the advantage of the source redundancy of systematically encoded for transmission over a discrete memoryless channel (DMC). The performance is achieved by operating discrete universal denoiser (DUDE) and the experiments involving Reed-Solomon codes show that DUDE-enhanced decoding is very effective at high rates.

Given two probability distributions Q and P, let /spl par/Q-P/spl par//sub 1/ and D(Q/spl par/P), respectively, denote the L/sub 1/ distance and divergence between Q and P. We derive a refinement of Pinsker's inequality of the form D(Q/spl par/P)/spl ges//spl phi/(P)/spl par/Q-P/spl par//sub 1//sup 2/ and characterize the best P-dependent factor /spl phi/(P).

This paper describes the optimality of symbol by symbol filtering and denoising and considers the problem of optimally recovering a discrete-time valued stochastic process from a noisy observation process. For binary Markov input process singlet filtering and denoising are optimal. Finally, for the case of memoryless channel, large deviations performance of singlet filtering is obtained.

This paper describes the empirical distribution of rate-constrained source codes. The existence of good code sequences with empirical distributions achieves the minimum mutual information in the rate distortion theory. A source code has entropy of capacity achieving convergence in channel input distributions.

Let {X/sub t/} be a stationary finite-alphabet Markov chain and {Z/sub t/} denote its noisy version when corrupted by a discrete memoryless channel. Let P(X/sub t//spl isin//spl middot/|Z/sub -/spl infin///sup t/) denote the conditional distribution of X/sub t/ given all past and present noisy observations, a simplex-valued random variable. We present a new approach to bounding the entropy rate of {Z/sub t/} by approximating the distribution of this random variable. This approximation is facilitated by the construction and study of a Markov process whose stationary distribution determines the distribution of P(X/sub t//spl isin//spl middot/|Z/sub -/spl infin///sup t/). To illustrate the efficacy of this approach, we specialize it and derive concrete bounds for the case of a binary Markov chain corrupted by a binary symmetric channel (BSC). These bounds are seen to capture the behavior of the entropy rate in various asymptotic regimes.

The classical framework of context-tree models, customary in sequential decision problems such as compression and prediction, is generalized to a setting in which the observations are multi-tracked or multi-directional, and for which it may be beneficial to consider contexts comprised of possibly differing numbers of symbols from each track or direction. The notion of a bi-directional context set is formalized and the generalization of the classical context-tree-based representation for a well defined set of bi-directional contexts is presented, together with an efficient dynamic programming algorithm for determining the best set of bi-directional contexts for a given individual sequence, maximum context depth, and loss function. After briefly describing how this framework can be applied to universal data compression, we focus on its application to universal denoising, where we pair the proposed framework with a new technique for estimating the loss of a denoising algorithm based only on noisy observations.

Let G be a t-uniform s-regular linear hypergraph with r vertices. It is shown that the number of independent sets $\IS(\hgraph)$ in $\hgraph$ satisfies \[ \log_2 \IS(\hgraph) \le \frac{r}{t} \left( 1 + O \biggl( \frac{\log^2(ts)}{s} \biggr) \right) . \] This leads to an improvement of a previous bound by Alon obtained for t = 2 (i.e., for regular ordinary graphs). It is also shown that for the Hamming graph $\Hamming(n,q)$ (with vertices consisting of all n-tuples over an alphabet of size q and edges connecting pairs of vertices with Hamming distance 1), \[ \frac{\log_2 \IS(\Hamming(n,q))}{q^n} = \frac{1}{q} + O \biggl(\frac{\log^2 (q n)}{q n} \biggr). \] The latter result is then applied to show that the Shannon capacity of the n-dimensional $(d,\infty)$-runlength-limited (RLL) constraint converges to 1/(d+1) as n goes to infinity.

This paper describes a discrete universal denoiser for two dimensional data and also presents an experimental results of its application to noisy binary images. A discrete universal denoiser (DUDE) is introduced for recovering a signal with finite-valued components corrupted by finite-valued, uncorrelated noise. The DUDE is asymptotically optimal and universal, in the sense of asymptotically achieving, without access to any information on the statistics of the clean signal, the same performance as the best denoiser that does have access to such information. It is also practical, and can be implemented in low complexity.

We propose a discrete denoising algorithm, that, based on the observation of the output of a known discrete memoryless channel (DMC), estimates the input sequence to minimize a given fidelity criterion. The algorithm is universal in the sense that it requires no knowledge of the input sequence or its statistical properties. Yet, asymptotically it performs as well as the optimum denoiser that knows the input sequence distribution. The proposed denoising algorithm is practical, and can be implemented in O(n log n) time and O(n/sup 2/3/ log n) storage complexity. Extensions to the case of delay-constrained denoising, and to the case of channel uncertainty, are briefly discussed.

The problem of optimal sequential decision for individual sequences, relative to a class of competing off-line reference strategies, is studied for general loss functions with memory. This problem is motivated by applications in which actions may have "long-term" effects, or there is a cost for switching from one action to another. As a first step, we consider the case in which the reference strategies are taken from a finite set of generic "experts." We then focus on finite-state reference strategies, assuming finite action and observation spaces. We show that key properties, that hold for finite-state strategies in the context of memoryless loss functions, do not carry over to the case of loss functions with memory. As a result, an infinite family of randomized finite-state strategies is seen to be the most appropriate reference class for this case, and the problem is basically different from its memoryless counterpart. Based on Vovk's (1990) exponential weighting technique, infinite-horizon on-line decision schemes are devised. For an arbitrary sequence of observations of length n, the excess normalized loss of these schemes relative to the best expert in a corresponding reference class is shown to be upper-bounded by an O(n/sup -1/3/) term in the case of a finite class, or an O([(ln n)/n]/sup 1/3/) term for the class of randomized finite-state strategies. These results parallel the O(n/sup -1/2/) bounds attained by previous schemes for memoryless loss functions. By letting the number of states in the reference class grow, the notion of finite-state predictability is also extended.

Prediction of individual sequences is investigated for cases in which the decision maker observes a delayed version of the sequence, or is forced to issue his/her predictions a number of steps in advance, with incomplete information. For finite action and observation spaces, it is shown that the prediction strategy that minimizes the worst case regret with respect to the Bayes envelope is obtained through subsampling of the sequence of observations. The result extends to the case of logarithmic loss. For finite-state (FS) reference prediction strategies, the delayed FS predictability (DFSP) is defined and related to its nondelayed counterpart. As in the nondelayed case, an efficient on-line decision algorithm, based on the incremental parsing rule, is shown to perform in the long run essentially as well as the best FS strategy determined in hindsight, with full knowledge of the given sequence of observations. An application to adaptive prefetching in computer memory architectures is discussed.

Prefetching in computer memory architectures is formalized as a sequential decision problem in which the instantaneous losses depend not only on the current action-observation pair, as in the traditional formulation, but also on past pairs. Motivated by the prefetching application, we study a class of loss functions that admit an efficient on-line decision algorithm. The algorithm uses the LZ78 parsing rule to dynamically build a tree, different from the classical LZ78 tree, and makes decisions based on the current node in a traversal path, determined by the sequence of observations. The asymptotic performance is essentially as good as that of the best finite-state strategy determined in hindsight, with full knowledge of the given sequence of observations. The related notion of delayed FS predictability is introduced, and its properties are studied.

For a rational /spl alpha//spl isin/(0,1), let /spl Ascr//sub n/spl times/m,/spl alpha// be the set of binary n/spl times/m arrays in which each row has Hamming weight /spl alpha/m and each column has Hamming weight /spl alpha/n, where /spl alpha/m and /spl alpha/n are integers. (The special case of two-dimensional balanced arrays corresponds to /spl alpha/=1/2 and even values for n and m.) The redundancy of /spl Ascr//sub n/spl times/m,/spl alpha// is defined by /spl rho//sub n/spl times/m,/spl alpha//=nmH(/spl alpha/)-log/sub 2/|/spl Ascr//sub n/spl times/m,/spl alpha//| where H(x)=-xlog/sub 2/x-(1-x)log/sub 2/(1-x). Bounds on /spl rho//sub n/spl times/m,/spl alpha// are obtained in terms of the redundancies of the sets /spl Ascr//sub /spl Lscr/,/spl alpha// of all binary /spl Lscr/-vectors with Hamming weight /spl alpha//spl Lscr/, /spl Lscr//spl isin/{n,m}. Specifically, it is shown that /spl rho//sub n/spl times/m,/spl alpha///spl les/n/spl rho//sub m,/spl alpha//+m/spl rho//sub n,/spl alpha// where /spl rho//sub /spl Lscr/,/spl alpha//=/spl Lscr/H(/spl alpha/)-log/sub 2/|/spl Ascr//sub /spl Lscr/,/spl alpha//| and that this bound is tight up to an additive term O(n+log m). A polynomial-time coding algorithm is presented that maps unconstrained input sequences into /spl Ascr//sub n/spl times/m,/spl alpha// at a rate H(/spl alpha/)-(/spl rho//sub m,/spl alpha///m).

We study the problem of memory-efficient scalable image compression and investigate some tradeoffs in the complexity versus coding efficiency space. The focus is on a low-complexity algorithm centered around the use of sub-bit-planes, scan-causal modeling, and a simplified arithmetic coder. This algorithm approaches the lowest possible memory usage for scalable wavelet-based image compression and demonstrates that the generation of a scalable bit-stream is not incompatible with a low-memory architecture.

We present a new low-complexity method for modeling and coding the bitplanes of a wavelet-transformed image in a fully embedded fashion. The scheme uses a simple ordering model for embedding, based on the principle that coefficient bits that are likely to reduce the distortion the most should be described first in the encoded bitstream. The ordering model is tied to a conditioning model in a way that deinterleaves the conditioned subsequences of coefficient bits, making them amenable to coding with a very simple, adaptive elementary Golomb (1966) code. The proposed scheme, without relying on zerotrees or arithmetic coding, attains PSNR vs. bit rate performance superior to that of SPIHT, and competitive with its arithmetic coding variant, SPIHT-AC.

We solve the problem of tracking the best constant rebalanced portfolio computed in hindsight in a max-min optimal sense and relate our results to the pricing of a new derivative security which might be called the hindsight allocation option. This option pays the return of one dollar invested in the best constant rebalanced portfolio computed in hindsight.

We find an on-line portfolio selection strategy which minimizes the worst case regret with respect to the growth rate of wealth achieved by the best off-line optimized constant rebalanced portfolio.The resulting regret is compared to that of the infinite horizon Dirichlet-weighted universal portfolio.Both strategies track the performance of the best off-line constant rebalanced portfolio to first order in the exponent.1

We present a sequential investment algorithm, the /spl mu/-weighted universal portfolio with side information, which achieves, to first order in the exponent, the same wealth as the best side-information dependent investment strategy (the best state-constant rebalanced portfolio) determined in hindsight from observed market and side-information outcomes. This is an individual sequence result which shows the difference between the exponential growth wealth of the best state-constant rebalanced portfolio and the universal portfolio with side information is uniformly less than (d/(2n))log (n+1)+(k/n)log 2 for every stock market and side-information sequence and for all time n. Here d=k(m-1) is the number of degrees of freedom in the state-constant rebalanced portfolio with k states of side information and m stocks. The proof of this result establishes a close connection between universal investment and universal data compression.

It was shown by Pombra and Cover (1994) that the maximum achievable throughput (sum of rates of all users) of a Gaussian multiple-access channel with feedback is at most twice that achievable without feedback. We prove a somewhat stronger result which establishes the factor-of-two bound not only for the total throughput but for the entire capacity region as well. Specifically, we show that the capacity region of a Gaussian multiple-access channel with feedback is contained within twice the capacity region without feedback.

The authors report on the use of the codebook-excited linear-predictive (CELP) algorithm for 32 kb/s low-delay (LD-CELP) coding of wideband speech. The main problem associated with wideband coding, namely, spectral noise weighting, is discussed. The authors propose an enhanced noise weighting technique and demonstrate its efficiency via subjective listening tests. In these tests, involving 20 listeners and 8 test sentences, the average rating for the proposed 32 kb/s LD-CELP was essentially equal to that of the 65 kb/s standard (G.722) CCITT wideband coder.>