We propose a unified optimization framework for designing continuous and discrete noise distributions that ensure differential privacy (DP) by minimizing Rényi DP, a variant of DP, under a cost constraint. Rényi DP has the advantage that by considering different values of the Rényi parameter $\alpha$, we can tailor our optimization for any number of compositions. To solve the optimization problem, we reduce it to a finite-dimensional convex formulation and perform preconditioned gradient descent. The resulting noise distributions are then compared to their Gaussian and Laplace counterparts. Numerical results demonstrate that our optimized distributions are consistently better, with significant improvements in $(\varepsilon, \delta)$-DP guarantees in the moderate composition regimes, compared to Gaussian and Laplace distributions with the same variance.

A mechanism called switched feedback is introduced; under switched feedback, each channel output goes forward to the receiver(s) or back to the transmitter(s) but never both. By studying the capacity of the Multiple-Access Channel (MAC) with switched feedback, this work investigates the benefits of feedback, seeking to maximize that benefit under reliable and unreliable feedback scenarios. The study is used to explore the tradeoffs between cooperation and transmission in the context of communication systems. Results include upper and lower bounds on the capacity region of the MAC with switched feedback.

We introduce a differentially private (DP) algorithm called reveal-or-obscure (ROO) to generate a single representative sample from a dataset of n observations drawn i.i.d. from an unknown discrete distribution P. Unlike methods that add explicit noise to the estimated empirical distribution, ROO achieves ϵ-differential privacy by randomly choosing whether to "reveal" or "obscure" the empirical distribution. While ROO is structurally identical to the algorithm in a recent work by Cheu and Nayak, we prove a strictly better bound on the sampling complexity than that established by them. To further improve the privacy-utility tradeoff, we propose a novel generalized sampling algorithm called Data-Specific ROO (DS-ROO), where the probability of obscuring the empirical distribution of the dataset is chosen adaptively. We prove that DS-ROO satisfies ϵ-DP, and provide empirical evidence that DS-ROO can achieve better utility under the same privacy budget of vanilla ROO.

Differentially private stochastic gradient descent (DP-SGD) is the most widely used method for training machine learning models with provable privacy guarantees. A key challenge in DP-SGD is setting the per-sample gradient clipping threshold, which significantly affects the trade-off between privacy and utility. While recent adaptive methods improve performance by adjusting this threshold during training, they operate in the standard coordinate system and fail to account for correlations across the coordinates of the gradient. We propose GeoClip, a geometry-aware framework that clips and perturbs gradients in a transformed basis aligned with the geometry of the gradient distribution. GeoClip adaptively estimates this transformation using only previously released noisy gradients, incurring no additional privacy cost. We provide convergence guarantees for GeoClip and derive a closed-form solution for the optimal transformation that minimizes the amount of noise added while keeping the probability of gradient clipping under control. Experiments on both tabular and image datasets demonstrate that GeoClip consistently outperforms existing adaptive clipping methods under the same privacy budget.

We formulate a fundamental limit in differential privacy under growing composition. We introduce the universal composition curve: the best privacy guarantee under repeated composition of a given privacy mechanism given only the sensitivity of the query. We define privacy capacity as the slowest growth rate of this universal composition curve among all privacy mechanisms. We show that, in the limit of large compositions, privacy capacity “single-letterizes” as a minimax KL-divergence term. Our privacy capacity formula extends previous literature results that connect differential privacy and KL-divergence via concentration theorems.

We introduce the paradigm of validated decentralized learning for undirected networks with heterogeneous data and possible adversarial infiltration. We require ($a$) convergence to a global empirical loss minimizer when adversaries are absent, and$(\boldsymbol{b})$either detection of adversarial presence or convergence to an admissible consensus model in their presence. This contrasts sharply with the traditional byzantine-robustness requirement of convergence to an admissible consensus irrespective of the adversarial configuration. To this end, we propose the Valid protocol which, to the best of our knowledge, is the first to achieve a validated learning guarantee. Moreover, Valid offers an$O(1/T)$convergence rate (under pertinent regularity assumptions), and computational and communication complexities comparable to non-adversarial distributed stochastic gradient descent. Remarkably, Valid retains optimal performance metrics in adversary-free environments, sidestepping the robustness penalties observed in prior byzantine-robust methods. A distinctive aspect of our study is a heterogeneity metric based on the norms of individual agents' gradients computed at the global empirical loss minimizer. This not only provides a natural statistic for detecting significant byzantine disruptions but also allows us to prove the optimality of Valid in wide generality. Lastly, our numerical results reveal that, in the absence of adversaries, Validcon-verges faster than state-of-the-art byzantine robust algorithms, while when adversaries are present, Valid terminates with each honest agent either converging to an admissible consensus or declaring adversarial presence in the network.

Motivated by the design of low-complexity low-power coding solutions for the Gaussian relay channel, this work presents an upper bound on the minimum energy-per-bit achiev-able on the Gaussian relay channel using rank-1 linear relaying. Our study addresses high-dimensional relay codes and presents bounds that outperform prior known bounds using 2-dimensional schemes. A novelty of our analysis ties the optimization problem at hand to the solution of a certain differential equation which, in turn, leads to a low energy-per-bit achievable scheme.

We introduce a family of information leakage measures calledmaximal(α, β)-leakage(MαbeL), parameterized by real numbers α and β greater than or equal to 1. The measure is formalized via an operational definition involving an adversary guessing an unknown (randomized) function of the data given the released data. We obtain a simplified computable expression for the measure and show that it satisfies several basic properties such as monotonicity in β for a fixed α, non-negativity, data processing inequalities, and additivity over independent releases. We highlight the relevance of this family by showing that it bridges several known leakage measures, including maximal α-leakage (β = 1), maximal leakage (α = ∞, β = 1), local differential privacy (LDP) (α = ∞, β = ∞), and local Rényi differential privacy (LRDP) (α = β), thereby giving an operational interpretation to local Rényi differential privacy. We also study a conditional version of MαbeL on leveraging which we recover differential privacy and Rényi differential privacy. A new variant of LRDP, which we callmaximal Rényi leakage, appears as a special case of MαbeL for α = ∞ that smoothly tunes between maximal leakage (β = 1) and LDP (β = ∞). Finally, we show that a vector form of the maximal Rényi leakage relaxes differential privacy under Gaussian and Laplacian mechanisms.

We introduce a gain function viewpoint of information leakage by proposing maximal$g$-leakage, a rich class of operationally meaningful leakage measures that subsumes recently introduced leakage measures — maximal leakage and maximal$\alpha $-leakage. In maximal$g$-leakage, the gain of an adversary in guessing an unknown random variable is measured using a gain function applied to the probability of correctly guessing. In particular, maximal$g$-leakage captures the multiplicative increase, upon observing$Y$, in the expected gain of an adversary in guessing a randomized function of$X$, maximized over all such randomized functions. We also consider the scenario where an adversary can make multiple attempts to guess the randomized function of interest. We show that maximal leakage is an upper bound on maximal$g$-leakage under multiple guesses, for any non-negative gain function$g$. We obtain a closed-form expression for maximal$g$-leakage under multiple guesses for a class of concave gain functions. We also study maximal$g$-leakage measure for a specific class of gain functions related to the$\alpha $-loss, that interpolates log-loss ($\alpha =1$) and (soft) 0–1 loss ($\alpha =\infty $). In particular, we first completely characterize the minimal expected$\alpha $-loss under multiple guesses and analyze how the corresponding leakage measure is affected with the number of guesses. We show that a new measure of divergence that belongs to the class of Bregman divergences captures the relative performance of an arbitrary adversarial strategy with respect to an optimal strategy in minimizing the expected$\alpha $-loss. Finally, we study two variants of maximal$g$-leakage depending on the type of adversary and obtain closed-form expressions for them, which do not depend on the particular gain function considered as long as it satisfies some mild regularity conditions. We do this by developing a variational characterization for the Rényi divergence of order infinity which naturally generalizes the definition of pointwise maximal leakage to incorporate arbitrary gain functions.

We characterize the differential privacy guarantees of privacy mechanisms in the large-composition regime, i.e., when a privacy mechanism is sequentially applied a large number of times to sensitive data. Via exponentially tilting the privacy loss random variable, we derive a new formula for the privacy curve expressing it as a contour integral over an integration path that runs parallel to the imaginary axis with a free real-axis intercept. Then, using the method of steepest descent from mathematical physics, we demonstrate that the choice of saddle-point as the real-axis intercept yields closed-form accurate approximations of the desired contour integral. This procedure---dubbed the saddle-point accountant (SPA)---yields a constant-time accurate approximation of the privacy curve. Theoretically, our results can be viewed as a refinement of both Gaussian Differential Privacy and the moments accountant method found in Rényi Differential Privacy. In practice, we demonstrate through numerical experiments that the SPA provides a precise approximation of privacy guarantees competitive with purely numerical-based methods (such as FFT-based accountants), while enjoying closed-form mathematical expressions.

We construct vector differentially-private (DP) mechanisms that are asymptotically optimal in the limit of the number of compositions growing without bound. First, we derive via the central limit theorem a reduction from DP to a KL-divergence minimization problem. Second, we formulate the general theory of spherically-symmetric DP mechanisms in the large-composition regime. Specifically, we show that additive, continuous, spherically-symmetric DP mechanisms are optimal if one considers a spherically-symmetric cost (e.g., bounded noise variance) and an ℓ2sensitivity metric. We then formulate a finite-dimensional problem that produces noise distributions that can get arbitrarily close to optimal among monotone mechanisms. Finally, we demonstrate numerically that our proposed mechanism achieves better DP parameters than the vector Gaussian mechanism for the same variance constraint.

We consider the problem of designing optimal differential privacy mechanisms with a favorable privacy-utility tradeoff in the limit of a large number n of compositions (i.e., sequential queries). Here, utility is measured by the average distance between the mechanism's input and output, evaluated by a cost function c. We show that if n is sufficiently large and the sensitivities of all queries are small, then the optimal additive noise mechanism has probability density function fully characterized by the ground-state eigenfunction of the Schrödinger operator with potential c. This leads to a family of optimal mechanisms, dubbed the Schrödinger mechanisms, depending on the choice of the cost function. Instantiating this result, we demonstrate that for c(x) = x2the Gaussian mechanism is optimal, and for c(x) = |x|, the optimal mechanism is obtained by the Airy function, thereby leading to the Airy mechanism.

We consider the problem of authenticated communication over a discrete arbitrarily varying channel where the legitimate parties are unaware of whether or not an adversary is present. When there is no adversary, the channel state always takes a default value ∅. When the adversary is present, they may choose the channel state sequence based on a non-causal noisy view of the transmitted codewords and the encoding and decoding scheme. We require that the decoder output the correct message with a high probability when there is no adversary, and either output the correct message or reject the transmission when the adversary is present. Further, we allow the transmitter to employ private randomness during encoding that is known neither to the receiver nor the adversary. Our first result proves a dichotomy property for the capacity for this problem – the capacity either equals zero or it equals the non-adversarial capacity of the channel. Next, we give a sufficient condition for the capacity for this problem to be positive even when the non-adversarial channel to the receiver is stochastically degraded with respect to the channel to the adversary. Our proofs rely on a connection to a standalone authentication problem, where the goal is to accept or reject a candidate message that is already available to the decoder. Finally, we give examples and compare our sufficient condition with other related conditions known in the literature.

The multiple access channel (MAC) capacity with feedback is considered under feedback models designed to tease out which factors contribute to the MAC feedback capacity benefit. Comparing the capacity of a MAC with "perfect" feedback, which causally delivers to the transmitters the true channel output, to that of a MAC with "independent" feedback, which causally delivers to the transmitters an independent instance of that same channel output, allows separation of effects like cooperation from alternative feedback benefits such as knowledge of the channel instance. Proving that the Cover-Leung (CL) achievability bound, which is known to be loose for some channels, is achievable also under (shared or distinct) independent feedback at the transmitters shows that the CL bound does not require transmitter knowledge of the channel instance. Proving that each transmitter’s maximal rate under independent feedback exceeds that under perfect feedback highlights the potential power of an independent look at the channel output.

This work establishes that the physical layer can be used to perform information-theoretic authentication in additive white Gaussian noise (AWGN) channels, as long as the adversary is not omniscient. The model considered consists of an encoder, decoder, and adversary, where the adversary knows the message given to the encoder, has a non-causal noisy observation of the encoder’s transmission and may use unlimited transmission power, while the decoder observes a noisy version of the sum of the encoder and adversary’s outputs. A method to modify a generic existing channel code to enable authentication is presented. This method relies on injecting message-dependent noise into the transmission and accepting the transmission as authentic only if the correct noise levels for the decoded message are observed. One drawback to this method is that the encoder must still transmit a low-power signal in the case where there is no message to send. It is shown that this modification costs an asymptotically negligible amount of the coding rate, while still enabling authentication as long as the adversary’s observation is not noiseless. Also notable is that this modification is not (asymptotically) a function of the statistical characterization of the adversary’s channel and no secret key is required. We believe these features will pave the way for a robust practical implementation. Using these results, the channel-authenticated capacity is calculated and shown to be equal to the non-adversarial channel capacity. As our results will show, information-theoretic authentication in AWGN channels is possible without the need for the legitimate party to have a model-based advantage over the adversary. While this modular scheme is designed for use in the given channel model, it is applicable to a wide range of settings.

Most differential privacy mechanisms are applied (i.e., composed) numerous times on sensitive data. We study the design of optimal differential privacy mechanisms in the limit of a large number of compositions. As a consequence of the law of large numbers, in this regime the best privacy mechanism is the one that minimizes the Kullback-Leibler divergence between the conditional output distributions of the mechanism given two different inputs. We formulate an optimization problem to minimize this divergence subject to a cost constraint on the noise. We first prove that additive mechanisms are optimal. Since the optimization problem is infinite dimensional, it cannot be solved directly; nevertheless, we quantize the problem to derive nearoptimal additive mechanisms that we call "cactus mechanisms" due to their shape. We show that our quantization approach can be arbitrarily close to an optimal mechanism. Surprisingly, for quadratic cost, the Gaussian mechanism is strictly suboptimal compared to this cactus mechanism. Finally, we provide numerical results which indicate that cactus mechanisms outperform Gaussian and Laplace mechanisms for a finite number of compositions.The full proofs can be found in the extended version at [1]. This paper is Part I in a pair of papers, where Part II is [2].

This work addresses the cooperation facilitator (CF) model, in which network nodes coordinate through a rate limited communication device. For multiple-access channel (MAC) encoders, the CF model is known to show significant rate benefits, even when the rate of cooperation is negligible. Specifically, the benefit in MAC sum-rate, as a function of the cooperation rate CCF, sometimes has an infinite slope at CCF= 0 when the CF enables transmitter dependence where none was possible otherwise. This work asks whether cooperation through a CF can yield similar infinite-slope benefits when dependence among MAC transmitters has no benefit or when it can be established without the help of the CF. Specifically, this work studies the CF model when applied to relay nodes of a single-source, single-terminal, diamond network comprising a broadcast channel followed by a MAC. In the relay channel with orthogonal receiver components, careful generalization of the partial-decode-forward/compress-forward lower bound to the CF model yields sufficient conditions for an infinite-slope benefit. Additional results include derivation of a family of diamond networks for which the infinite-slope rate-benefit derives directly from the properties of the corresponding MAC studied in isolation.

We present a variational characterization for the Rényi divergence of order infinity. Our characterization is related´ to guessing: the objective functional is a ratio of maximal expected values of a gain function applied to the probability of correctly guessing an unknown random variable. An important aspect of our variational characterization is that it remains agnostic to the particular gain function considered, as long as it satisfies some regularity conditions. Also, we define two variants of a tunable measure of information leakage, the maximal αleakage, and obtain closed-form expressions for these information measures by leveraging our variational characterization.

We introduce a family of information leakage measures called maximal α, β-leakage, parameterized by real numbers α and β. The measure is formalized via an operational definition involving an adversary guessing an unknown function of the data given the released data. We obtain a simple, computable expression for the measure and show that it satisfies several basic properties such as monotonicity in β for a fixed α, non-negativity, data processing inequalities, and additivity over independent releases. Finally, we highlight the relevance of this family by showing that it bridges several known leakage measures, including maximal α-leakage (β = 1), maximal leakage (α = ∞, β = 1), local differential privacy (α = ∞, β = ∞), and local Rényi differential privacy (α = β).

A new converse bound is presented for the two-user multiple-access channel under the average probability of error constraint. This bound shows that for most channels of interest, the second-order coding rate—that is, the difference between the best achievable rates and the asymptotic capacity region as a function of blocklength$n$with fixed probability of error—is$O(1/\sqrt {n})$bits per channel use. The principal tool behind this converse proof is a new measure of dependence between two random variables called wringing dependence, as it is inspired by Ahlswede’s wringing technique. The$O(1/\sqrt {n})$gap is shown to hold for any channel satisfying certain regularity conditions, which includes all discrete-memoryless channels and the Gaussian multiple-access channel. Exact upper bounds as a function of the probability of error are proved for the coefficient in the$O(1/\sqrt {n})$term, although for most channels they do not match existing achievable bounds.

The work at hand presents a finite-blocklength analysis of the multiple access channel (MAC) sum-rate under the cooperation facilitator (CF) model. The CF model, in which independent encoders coordinate through an intermediary node, is known to show significant rate benefits, even when the rate of cooperation is limited. We continue this line of study for cooperation rates which are sub-linear in the blocklength n. Roughly speaking, our results show that if the facilitator transmits log$K$bits, then there is a sum-rate benefit of order √log K/n compared to the best-known achievable rate. This result extends across a wide range of K: even a single bit of cooperation is shown to provide a sum-rate benefit of order 1/√n.

We consider a problem of guessing, wherein an adversary is interested in knowing the value of the realization of a discrete random variable$X$on observing another correlated random variable Y. The adversary can make multiple (say, k) guesses. The adversary's guessing strategy is assumed to minimize a-loss, a class of tunable loss functions parameterized by a. It has been shown before that this loss function captures well known loss functions including the exponential loss (a = 1/2), the log-loss (a = 1) and the 0–1 loss (a = ∞). We completely characterize the optimal adversarial strategy and the resulting expected α-loss, thereby recovering known results for a = ∞. We define an information leakage measure from the k-guesses setup and derive a condition under which the leakage is unchanged from a single guess.

The second-order converse bound of multiple access channels is an intriguing problem in information theory. In this work, in the setting of the two-user discrete memoryless multiple access channel (DM-MAC) under the maximal error probability criterion, we investigate the gap between the best achievable rates and the asymptotic capacity region. With “wringing techniques” and meta-converse arguments, we show that gap at blocklength$n$is upper bounded by$O(1/\sqrt{n})$.

We derive the optimal differential privacy (DP) parameters of a mechanism that satisfies a given level of Renyí differential privacy (RDP). Our result is based on the joint range of two f-divergences that underlie the approximate and the Renyi variations of differential privacy. We apply our result tó the moments accountant framework for characterizing privacy guarantees of stochastic gradient descent. When compared to the state-of-the-art, our bounds may lead to about 100 more stochastic gradient descent iterations for training deep learning models for the same privacy budget.

In unsecured communications settings, ascertaining the trustworthiness of received information, called authentication, is paramount. We consider keyless authentication over an arbitrarily-varying channel, where channel states are chosen by a malicious adversary with access to noisy versions of transmitted sequences. We have shown previously that a channel condition termed U-overwritability is a sufficient condition for zero authentication capacity over such a channel, and also that with a deterministic encoder, a sufficiently clear-eyed adversary is essentially omniscient. In this paper, we show that even if the authentication capacity with a deterministic encoder and an essentially omniscient adversary is zero, allowing a stochastic encoder can result in a positive authentication capacity. Furthermore, the authentication capacity with a stochastic encoder can be equal to the no-adversary capacity of the underlying channel in this case. We illustrate this for a binary channel model, which provides insight into the more general case.

This paper considers the two-user Gaussian arbitrarily-varying broadcast channel, wherein a power-limited transmitter wishes to send a message to each of two receivers. Each receiver sees a superposition of the transmitter's sequence, Gaussian noise, and a signal from a power-limited malicious jammer. The jammer is assumed to know the code, but is oblivious to real-time transmissions. The exact capacity region of this setting is determined to be the capacity region of the standard Gaussian broadcast channel, but with the noise variance increased by the power of the jammer, as long as the received power of the jammer at each receiver is less than that of the legitimate transmitter. A key aspect of the achievable scheme involves sharing randomness from the transmitter to the receivers by breaking the transmitted sequence into segments, and either transmitting at full power in a segment, or sending zero. By coding over the on/off signal, a small shared randomness can be established without corruption by the jammer, and without interfering with the standard superposition coding strategy for the Gaussian broadcast channel.

Authentication in the presence of a malicious adversary consists of either recovering the legitimate transmission or declaring that the adversary has interfered with the transmission. In this work, we present a structured coding scheme for keyless authentication over a discrete memoryless binary-input, symmetric adversarial channel. Our scheme allows for coding rates up to the non-adversarial capacity of the underlying channel, as well as bounded-complexity decoding.

Maximal α-leakage is a tunable measure of information leakage based on the accuracy of guessing an arbitrary function of private data based on public data. The parameter α determines the loss function used to measure the accuracy of a belief, ranging from log-loss at α = 1 to the probability of error at α = ∞. To study the effect of side information on this measure, we introduce and define conditional maximal α-leakage. We show that, for a chosen mapping (channel) from the actual (viewed as private) data to the released (public) data and some side information, the conditional maximal α-leakage is the supremum (over all side information) of the conditional Arimoto channel capacity where the conditioning is on the side information. We prove that if the side information is conditionally independent of the public data given the private data, the side information cannot increase the information leakage.

Universal source coding at short blocklengths is considered for an i.i.d. exponential family of distributions. TheType Sizecode has previously been shown to be optimal up to the third-order rate for universal compression of all memoryless sources over finite alphabets. The Type Size code assigns sequences ordered based on their type class sizes to binary strings ordered lexicographically. To generalize this type class approach for parametric sources, a natural scheme is to define two sequences to be in the same type class if and only if they are equiprobable under any model in the parametric class. This natural approach, however, is shown to be suboptimal. A variation of the Type Size code is introduced, where type classes are defined based on neighborhoods of minimal sufficient statistics. The asymptotics of the overflow rate of this variation are derived and a converse result establishes its optimality up to the third-order term.

In The paper above[1], published in the April 2019 issue of the IEEE Transactions on Information Theory, the following change is noted. The affiliations of the authors were listed incorrectly due to a production error. The affiliations should be listed as follows: The authors were with the School of Electrical, Computer and Energy Engineering, Arizona State University, Tempe, AZ 85287 USA, while performing their research for[1].

This paper explores the relationship between two ideas in the network information theory: edge removal and strong converses. Edge removal properties state that if an edge of small capacity is removed from a network, the capacity region does not change too much. Strong converses state that, for rates outside the capacity region, the probability of error converges to 1 as the blocklength goes to infinity. Various notions of edge removal and strong converse are defined, depending on how edge capacity and error probability scale with blocklength, and relations between them are proved. Each class of strong converse implies a specific class of edge removal. The opposite directions are proved for deterministic networks. Furthermore, a technique based on a novel, causal version of the blowing-up lemma is used to prove that for discrete memoryless networks, the weak edge removal property-that the capacity region changes continuously as the capacity of an edge vanishes-is equivalent to the exponentially strong converse-that outside the capacity region, the probability of error goes to 1 exponentially fast. This result is used to prove exponentially strong converses for several examples, including the discrete two-user interference channel with strong interference, with only a small variation from traditional weak converse proofs.

We introduce a tunable measure for information leakage calledmaximal$\alpha $-leakage. This measure quantifies the maximal gain of an adversary in inferring any (potentially random) function of a dataset from a release of the data. The inferential capability of the adversary is, in turn, quantified by a class of adversarial loss functions that we introduce as$\alpha $-loss,$\alpha \in [1,\infty) \cup \{\infty \}$. The choice of$\alpha $determines the specific adversarial action and ranges from refining a belief (about any function of the data) for$\alpha =1$to guessing the most likely value for$\alpha = \infty $while refining the$\alpha ^{\text {th}}$moment of the belief for$\alpha $in between. Maximal$\alpha $-leakage then quantifies the adversarial gain under$\alpha $-loss over all possible functions of the data. In particular, for the extremal values of$\alpha =1$and$\alpha =\infty $, maximal$\alpha $-leakage simplifies to mutual information and maximal leakage, respectively. For$\alpha \in (1,\infty)$this measure is shown to be the Arimoto channel capacity of order$\alpha $. We show that maximal$\alpha $-leakage satisfies data processing inequalities and a sub-additivity property thereby allowing for a weak composition result. Building upon these properties, we use maximal$\alpha $-leakage as the privacy measure and study the problem of data publishing with privacy guarantees, wherein the utility of the released data is ensured via ahard distortionconstraint. Unlike average distortion, hard distortion provides a deterministic guarantee of fidelity. We show that under a hard distortion constraint, for$\alpha >1$the optimal mechanism is independent of$\alpha $, and therefore, the resulting optimal tradeoff is the same for all values of$\alpha >1$. Finally, the tunability of maximal$\alpha $-leakage as a privacy measure is also illustrated for binary data with average Hamming distortion as the utility measure.

This paper considers list-decoding for the Gaussian arbitrarily-varying channel under the average probability of error criterion, where both the legitimate transmission and the state (or adversarial signal) are power limited. For list size L, the capacity is equivalent to the capacity of a standard Gaussian with the noise power raised by the adversary power, if the ratio of the adversary power to the transmitter power is less than L; otherwise, the capacity is zero. The converse proof involves showing that with enough power, an adversary can confuse the decoder by transmitting a superposition of several codewords while satisfying its power constraint with positive probability. The achievability proof uses a novel variant of the Csiszar-Narayan method for the arbitrarily-varying channel.

Finite blocklength and second-order (dispersion) results are presented for the arbitrarily-varying channel (AVC), a classical model wherein an adversary can transmit arbitrary signals into the channel. A novel finite blocklength achievability bound is presented, roughly analogous to the random coding union bound for non-adversarial channels. This finite blocklength bound, along with a known converse bound, is used to derive bounds on the dispersion of discrete memoryless AVCs without shared randomness, and with cost constraints on the input and the state. These bounds are tight for many channels of interest, including the binary symmetric AVC. However, the bounds are not tight if the deterministic and random code capacities differ.

A tunable measure for information leakage called maximal a-leakage is introduced. This measure quantifies the maximal gain of an adversary in refining a tilted version of its prior belief of any (potentially random) function of a dataset conditioning on a disclosed dataset. The choice of α determines the specific adversarial action ranging from refining a belief for α = 1 to guessing the best posterior for α = ∞, and for these extremal values this measure simplifies to mutual information (MI) and maximal leakage (MaxL), respectively. For all other α this measure is shown to be the Arimoto channel capacity. Several properties of this measure are proven including: (i) quasi-convexity in the mapping between the original and disclosed datasets; (ii) data processing inequalities; and (iii) a composition property. A full version of this paper is in [1].

Keyless authentication is considered in an adversarial point-to-point channel. Namely, a legitimate transmitter and receiver aim to communicate over a noisy channel that may or may not also contain an active adversary, capable of transmitting an arbitrary signal into the channel. If the adversary is not present, then the receiver must successfully decode the message with high probability; if it is present, then the receiver must either decode the message or detect the adversary's presence. Thus, whenever the receiver decodes, it can be certain that the decoded message is authentic. The exact authentication capacity is characterized for discrete-memoryless adversary channels, where the adversary is assumed to know the code but not the message. The authentication capacity is shown to be either zero or equal to the no-adversary capacity, depending on whether the channel satisfies a condition termed overwritability.

We study the problem of data disclosure with privacy guarantees, wherein the utility of the disclosed data is ensured via a hard distortion constraint. Unlike average distortion, hard distortion provides a deterministic guarantee of fidelity. For the privacy measure, we use a tunable information leakage measure, namely maximal α-leakage (α ∈ [1, ∞]), and formulate the privacy-utility tradeoff problem. The resulting solution highlights that under a hard distortion constraint, the nature of the solution remains unchanged for both local and nonlocal privacy requirements. More precisely, we show that both the optimal mechanism and the optimal tradeoff are invariant for any α > 1; i.e., the tunable leakage measure only behaves as either of the two extrema, i.e., mutual information for α = 1 and maximal leakage for α = ∞.

We consider a problem in which a source is encoded into N packets, an unknown number of which are subject to adversarial errors en route to the decoder. We seek code designs for which the decoder is guaranteed to be able to reproduce the source subject to a certain distortion constraint when there are no packets errors, subject to a less stringent distortion constraint when there is one error, and so on. Focusing on the special case of the erasure distortion measure, we introduce a code design based on the polytope codes of Kosut et al.. The resulting designs are also applied to a separate problem in distributed storage.

The privacy-utility tradeoff problem is formulated as determining the privacy mechanism (random mapping) that minimizes the mutual information (a metric for privacy leakage) between the private features of the original dataset and a released version. The minimization is subject to a constraint on the average distortion cost defined as a function f evaluated for every distortion d between the public features and the released version of dataset. The asymptotic optimal leakage is derived both for general and stationary memoryless privacy mechanisms. It is shown that for convex cost functions there is no asymptotic loss in using stationary memoryless mechanisms. Of independent interest are the proof techniques developed here for arbitrary cost functions.

The second-order behavior of the discrete memoryless arbitrarily-varying channel is considered in the fixed error regime when the encoder and decoder share randomness that is independent from the adversarial choice of state. The dispersion (coefficient of the second-order term) is exactly characterized for most channels of interest when infinite shared randomness is allowed, and it is shown that precisely the same dispersion is achievable with only O (log n) bits of shared randomness. We also show that the dispersion is identical to that of the non-adversarial channel induced by the adversary simply choosing an i.i.d. state sequence according to the correct distribution. Further, we present some remarks on the connection to the compound channel, as well as on cost constraints for input and state sequences.

Universal fixed-to-variable lossless source coding for memoryless sources is studied in the finite blocklength and higher order asymptotic regimes. Optimal three-term fixed-error asymptotic expressions are derived for general fixed-to-variable codes and for prefix codes. It is shown that the non-prefix Type Size code, in which codeword lengths are chosen in ascending order of type class size, achieves the optimal third-order term, and outperforms classical two-stage codes. Converse results are proved making use of a result on the distribution of the empirical entropy and Laplace's approximation. Finally, the fixed-to-variable coding problem without a prefix constraint is shown to be essentially the same as the universal guessing problem.

We consider universal source coding of an exponential family of i.i.d. distributions for short blocklengths. We present a variation of the previously introduced Type Size code, in which type classes are characterized based on the neighborhoods of the minimal sufficient statistics. We show that, there is no loss in dispersion compared to the non-universal setup. We identify the third-order coding rate of this variation of the Type Size code for compression of such parametric sources.

This paper explores the relationship between two ideas in network information theory: edge removal and strong converses. Edge removal properties state that if an edge of small capacity is removed from a network, the capacity region does not change too much. Strong converses state that, for rates outside the capacity region, the probability of error converges to 1. Various notions of edge removal and strong converse are defined, depending on how edge capacity and residual error probability scale with blocklength, and relations between them are proved. In particular, each class of strong converse implies a specific class of edge removal. The opposite direction is proved for deterministic networks, and some discussion is given for the noisy case.

We consider the problem of communication over a network containing a hidden and malicious adversary that can control a subset of network resources, and aims to disrupt communications. We focus on omniscient node-based adversary, i.e., the adversary can control a subset of nodes, and knows the message, network code and packets on all links. Characterizing information-theoretically optimal communication rates as a function of network parameters and bounds on the adversarially controlled network is in general open, even for unicast (single source, single destination) problems. In this work we characterize the information-theoretically optimal randomized capacity of such problems, i.e., under the assumption that the source node shares (an asymptotically negligible amount of) independent common randomness with each network node a priori. We propose a novel computationally-efficient communication scheme whose rate matches a natural information-theoretically “erasure outer bound” on the optimal rate. Our schemes require no prior knowledge of network topology, and can be implemented in a distributed manner as an overlay on top of classical distributed linear network coding.

We consider the problem of finding the capacity of noisy networks under the presence of Byzantine adversaries, modeled by a joint compound channel and arbitrarily varying channel (AVC) model. This extends our earlier work which considers these models only in isolation. The motivation for this setup is that typically the adversary first selects an arbitrary subset of edges from the network and then specifies adversarial transmissions to each of the selected edges. We show that in some cases equivalence between this network and another network holds in the sense that for a fixed selection of adversarial edges the noisy links can be replaced by noiseless bit-pipes with a capacity equal to the random coding capacity of the corresponding AVC. In particular, the capacity region for the noisy network can be outer bounded by the intersection of the individual capacity regions for the noiseless case, for each adversarial edge selection. Moreover, if the network is fully connected, we also show that this upper bound is equivalent to the capacity of the noisy network. We also provide necessary and sufficient condition for full connectivity, making use of a new condition for an AVC termed overwritability.

We consider the universal source coding problem for first-order stationary, irreducible and aperiodic Markov sources for short blocklengths. Achievability is derived based on the previously introduced algorithm for universal compression of memoryless sources in the finite blocklengths, the Type Size Code, which encodes strings based on type class size. We derive the third-order asymptotic coding rate of the Type Size code for this model class. We also present a converse on the third-order coding rate for the general class of fixed-to-variable codes and show the optimality of Type Size codes for such Markov sources.

Point-to-point channel coding is studied in the finite blocklength regime. Many existing converse bounds involve an optimization over a distribution on the channel output. This paper provides a method for generating good, if not optimal, output distributions. In particular, given any candidate output distribution, a “boosting” procedure is given that constructs a new distribution which improves the converse bound derived from the divergence spectrum. For discrete memoryless channels, it is shown that using the i.i.d. capacity-achieving output distribution as an initial guess in this procedure results in an output distribution that is good enough to derive the third-order coding rate for most channels. The finite blocklengths bounds are then applied to the Z channel.

We address the problem of finding the capacity of noisy networks with either independent point-to-point compound channels (CC) or arbitrarily varying channels (AVC). These channels model the presence of a Byzantine adversary, which controls a subset of links or nodes in the network. We derive equivalence results showing that these point-to-point channels with state can be replaced by noiseless bit-pipes without changing the network capacity region. Exact equivalence results are found for the CC model, and for some instances of the AVC, including all nonsymmetrizable AVCs. These results show that a feedback path between the output and input of a CC can increase the equivalent capacity, and that if common randomness can be established between the terminals of an AVC (either by a feedback, a forward path, or via a third-party node), then again the equivalent capacity can increase. This leads to an observation that deleting an edge of arbitrarily small capacity can cause a significant change in network capacity. We also analyze an example involving an AVC for which no fixed-capacity bit-pipe is equivalent.

Converse and achievable results on the third-order coding rate for universal fixed-to-variable source coding are presented. The authors previously introduced a new universal non-prefix Type Size code for memoryless sources in which codewords lengths are chosen in ascending order of type class sizes. This paper presents a converse on the third-order coding rate for general class of fixed-to-variable codes and proves the optimality of Type Size codes for memoryless sources. The third-order coding rate for a two-stage code is also developed and is shown to be strictly larger than achieved by the Type Size code.

Active attacks are studied on noise-free graphical multicast networks. A malicious adversary may enter the network and arbitrarily corrupt transmissions. A very general model is adopted for the scope of attack: a collection of sets of edges is specified, and the adversary may control any one set of edges in this collection. The adversary is assumed to be omniscient but causal, such that the adversary is forced to decide on transmissions before knowing random choices by the honest nodes. Four main results are presented. First, a precise characterization of whether any positive rate can be achieved. Second, a simple erasure upper bound. Third, an achievable bound wherein random hashes are generated and distributed, so that nodes in the network can filter out adversarial corruption. Finally, an example network is presented that has capacity strictly between the general upper and lower bounds.

This paper investigates a network coding problem wherein an adversary controls a subset of nodes in the network of limited quantity but unknown location. This problem is shown to be more difficult than that of an adversary controlling a given number of edges in the network, in that linear codes are insufficient. To solve the node problem, the class of polytope codes is introduced. Polytope codes are constant composition codes operating over bounded polytopes in integer vector fields. The polytope structure creates additional complexity, but it induces properties on marginal distributions of code vectors so that validities of codewords can be checked by internal nodes of the network. It is shown that polytope codes achieve a cut-set bound for a class of planar networks. It is also shown that this cut-set bound is not always tight, and a tighter bound is given for an example network.

We analyze the dispersions of distributed lossless source coding (the Slepian-Wolf problem), the multiple-access channel, and the asymmetric broadcast channel. For the two-encoder Slepian-Wolf problem, we introduce a quantity known as the entropy dispersion matrix, which is analogous to the scalar dispersions that have gained interest recently. We prove a global dispersion result that can be expressed in terms of this entropy dispersion matrix and provides intuition on the approximate rate losses at a given blocklength and error probability. To gain better intuition about the rate at which the nonasymptotic rate region converges to the Slepian-Wolf boundary, we define and characterize two operational dispersions: 1) the local dispersion and 2) the weighted sum-rate dispersion. The former represents the rate of convergence to a point on the Slepian-Wolf boundary, whereas the latter represents the fastest rate for which a weighted sum of the two rates converges to its asymptotic fundamental limit. Interestingly, when we approach either of the two corner points, the local dispersion is characterized not by a univariate Gaussian, but a bivariate one as well as a subset of off-diagonal elements of the aforementioned entropy dispersion matrix. Finally, we demonstrate the versatility of our achievability proof technique by providing inner bounds for the multiple-access channel and the asymmetric broadcast channel in terms of dispersion matrices. All our proofs are unified by a so-called vector rate redundancy theorem, which is proved using the multidimensional Berry-Esséen theorem.

Distributed storage systems are studied in the presence of an active omniscient adversary. The adversary is able to control several storage nodes in the system and alter their behavior. A Polytope code is proposed to handle such an adversary, and it is used to prove a lower bound on the overall storage capacity. Polytope codes have been shown to outperform linear codes over a finite field in defeating active adversaries. In a Polytope code, linear operations are performed over the integers rather than a finite field. This allows examinations of cross-covariances as a sort of parity check, which can improve error detection and correction without sacrificing asymptotic rate.

A universal source coding problem is considered in the finite blocklength regime for stationary memoryless sources. A new coding scheme is presented that encodes based on the type class size and the empirical support set of the sequence. It is shown that there is no loss in dispersion relative to the case when the source distribution is known. A new bound is obtained on the third order asymptotic coding rate. Numerical results are presented for finite blocklengths comparing the proposed coding scheme with a variety of coding schemes including Lempel-Ziv.

The problem of efficiently drawing samples from a Gaussian graphical model or Gaussian Markov random field is studied. We introduce the subgraph perturbation sampling algorithm, which makes use of any pre-existing tractable inference algorithm for a subgraph by perturbing this algorithm so as to yield asymptotically exact samples for the intended distribution. The subgraph can have any structure for which efficient inference algorithms exist: for example, tree-structured, low tree-width, or having a small feedback vertex set. The experimental results demonstrate that this subgraph perturbation algorithm efficiently yields accurate samples for many graph topologies.

We characterize second-order coding rates (or dispersions) for distributed lossless source coding (the Slepian-Wolf problem). We introduce a fundamental quantity known as the entropy dispersion matrix, which is analogous to scalar dispersion quantities. We show that if this matrix is positive-definite, the optimal rate region under the constraint of a fixed blocklength and non-zero error probability has a curved boundary compared to being polyhedral for the Slepian-Wolf case. In addition, the entropy dispersion matrix governs the rate of convergence of the non-asymptotic region to the asymptotic one. As a by-product of our analyses, we develop a general universal achievability procedure for dispersion analysis of some other network information theory problems such as the multiple-access channel. Numerical examples show how the region given by Gaussian approximations compares to the Slepian-Wolf region.

Network coding is studied when an unknown subset of nodes in the network is controlled by an adversary. To solve this problem, a new class of codes called Polytope Codes is introduced. Polytope Codes are linear codes operating over bounded polytopes in real vector fields. The polytope structure creates additional complexity, but it induces properties on marginal distributions of code vectors so that validities of codewords can be checked by internal nodes of the network. It is shown that a cut-set bound for a class planar networks can be achieved using Polytope Codes. It is also shown that this cut-set bound is not always tight, and a tighter bound is given for an example network.

The quadratic Gaussian CEO problem is studied when the agents are under Byzantine attack. That is, an unknown subset of agents is controlled by an adversary that attempts to damage the quality of the estimate at the central estimation officer, or CEO. Inner and outer bounds are presented for the achievable rate region as a function of the fraction of adversarial agents. The inner bound is derived from a generalization of the Berger-Tung quantize-and-bin strategy, which has been shown to be tight in the non-Byzantine case. The outer bound has similarities to the singleton bound in that the traitorous agents must be prevented from allowing two sources to result in the same transmitted codewords if their values are too far apart for the distortion constraint to be satisfied with a single estimate. The inner and outer bounds on the rate regions are used to find bounds on the asymptotic proportionality constant in the limit of a large number of agents and high sum-rate. These bounds on the proportionality constant differ at most by a factor of 4.

The CEO Problem is considered when a subset of the agents are under Byzantine attack; that is, they have been taken over and reprogrammed by a malicious intruder. Inner and outer bounds are given for the error exponent with respect to the sum rate, as a function of the fraction of reprogrammed, or traitor, agents. The inner bound is proved by introducing a coding scheme that takes advantage of the fact that the set of honest (non-traitor) agents will report jointly typical information. The CEO looks for a group with the same size as the set of honest agents that appear to do so. Even if not all the agents in this group are honest, the fact that they all agree keeps the probability of error in check. The outer bound is given in two parts, based on two different possible attacks by the traitors. The first is a black hole attack, in which the traitors simply transmit no information at all. The second is one in which they fabricate false data such that the CEO cannot determine which of two possibilities is the truth.

The distributed source coding problem is considered when the sensors, or encoders, are under Byzantine attack; that is, an unknown group of sensors have been reprogrammed by a malicious intruder to undermine the reconstruction at the fusion center. Three different forms of the problem are considered. The first is a variable-rate setup, in which the decoder adaptively chooses the rates at which the sensors transmit. An explicit characterization of the variable-rate achievable sum rates is given for any number of sensors and any groups of traitors. The converse is proved constructively by letting the traitors simulate a fake distribution and report the generated values as the true ones. This fake distribution is chosen so that the decoder cannot determine which sensors are traitors while maximizing the required rate to decode every value. Achievability is proved using a scheme in which the decoder receives small packets of information from a sensor until its message can be decoded, before moving on to the next sensor. The sensors use randomization to choose from a set of coding functions, which makes it probabilistically impossible for the traitors to cause the decoder to make an error. Two forms of the fixed-rate problem are considered, one with deterministic coding and one with randomized coding. The achievable rate regions are given for both these problems, and it is shown that lower rates can be achieved with randomized coding.

The distributed source coding problem is considered when the sensors, or encoders, are under Byzantine attack; that is, an unknown number of sensors have been reprogrammed by a malicious intruder to undermine the reconstruction at the fusion center. Three different forms of the problem are considered. The first is a variable-rate setup, in which the decoder adaptively chooses the rates at which the sensors transmit. An explicit characterization of the variable-rate minimum achievable sum rate is stated, given by the maximum entropy over the set of distributions indistinguishable from the true source distribution by the decoder. In addition, two forms of the fixed-rate problem are considered, one with deterministic coding and one with randomized coding. The achievable rate regions are given for both these problems, with a larger region achievable using randomized coding, though both are suboptimal compared to variable-rate coding.