Local differential privacy (LDP) is increasingly employed in privacy-preserving machine learning to protect user data before sharing it with an untrusted aggregator. Most LDP methods assume that users possess only a single data record, which is a significant limitation since users often gather extensive datasets (e.g., images, text, time-series data) and frequently have access to public datasets. To address this limitation, we propose a locally private sampling framework that leverages both the private and public datasets of each user. Specifically, we assume each user has two distributions: $p$ and $q$ that represent their private and public datasets, respectively. The objective is to design a mechanism that generates a private sample approximating $p$ while simultaneously preserving $q$. We frame this objective as a minimax optimization problem using $f$-divergence as the utility measure. We fully characterize the minimax optimal mechanisms for general $f$-divergences provided that $p$ and $q$ are discrete distributions. Remarkably, we demonstrate that this optimal mechanism is universal across all $f$-divergences. Experiments validate the effectiveness of our minimax optimal mechanism compared to the state-of-the-art private sampler.

In many applications across statistics, differential privacy, and machine learning, it is necessary to evaluate or bound an$f$-divergence between distributions of random vectors with independent components. Even for relatively simple distributions, such computations can become intractable unless the chosen$f$-divergence tensorizes. In this work, we introduce a formal definition of tensorization for$f$-divergences and present a necessary condition under which such tensorization can occur. Moreover, we demonstrate—under certain assumptions—that the only$f$-divergences admitting a polynomial tensorization formula of degree at most two are, essentially, the KL divergence, cross-entropy, and Hellinger divergences of order$\alpha$. Taken together, our findings represent an initial step toward a complete characterization of$f$-divergences that tensorize.

We propose a privacy auditing framework using minimum mean-squared error (MMSE) estimation and linear auditing models. Our approach provides theoretical lower bounds on the true MMSE of inferring sensitive features from noisy observations of other correlated features. The bounds are in terms of the empirical MMSE under a restricted hypothesis class and a decomposable error term capturing finite sample and approximation effects. For linear auditing models, we derive order-optimal closed-form bounds for classes of relationships between the private and non-private features, including linear mappings, binary symmetric channels, and class-conditional Gaussian models. Through empirical evaluation, we demonstrate that our linear model-based auditing framework serves as a powerful yet tractable tool for MMSE-based privacy auditing that balances theoretical guarantees with practical efficiency.

Differentially Private Stochastic Gradient Descent (DP-SGD) is a widely adopted algorithm for privately training machine learning models. An inherent feature of this algorithm is the incorporation of gradient clipping to counteract the influence of individual samples during training. Nevertheless, the introduction of gradient clipping also introduces non-convexity into the problem, rendering it challenging to derive upper bounds on the privacy loss. In this paper, we establish effective upper bounds for the privacy loss of both projected DP-SGD and regularized DP-SGD, without relying on convexity or smoothness assumptions regarding the loss function. Our approach involves a direct analysis of the hockey-stick divergence between coupled stochastic processes through the application of nonlinear data processing inequalities.

We investigate the mixing times of Markov kernels under$\mathsf{E}_{\gamma}-\mathbf{divergence}$. We demonstrate that the zero-error$\mathsf{E}_{\gamma}- \mathbf{mixing}$time, for any$\gamma > 1$, of irreducible and aperiodic Markov chains, is bounded, a property that is not shared by the TV-mixing time. We further obtain upper bounds on the$\mathsf{E}_{\gamma}-\mathbf{mixing}$times for a broad family of contractive Markov kernels via a new non-linear strong data processing inequality for the$\mathsf{E}_{\gamma}- \mathbf{divergence}$. We apply our results to derive new bounds for the local differential privacy guarantees offered by the sequential application of a privacy mechanism to data.

The Rashomon effect in machine learning (ML) occurs when multiple distinct models achieve similar average loss on a given learning task. The set of all models with expected loss smaller than ϵ is called the Rashomon set. The characterization of this set for a given learning task allows searching for models that satisfy additional constraints (e.g., interpretability, fairness) without compromising accuracy. Though folklore treats the Rashomon set as the collection of all indistinguishable "good" models, there are no established theoretical guarantees that models in this set are statistically indistinguishable. We fill this gap by proposing a hypothesis test framework to choose the best-performing model between two elements in the Rashomon set and derive lower and upper bounds for its probability of error. Specifically, we prove that for any ϵ > 0 if the data set has less than $O\left( {{{[\varepsilon {\text{log}}\left( {\varepsilon /\left( {1 - \varepsilon } \right)} \right)]}^{ - 1}}} \right)$ instances, models in the Rashomon set are statistically indistinguishable and the Rashomon effect is inevitable. Additionally, our bounds can guide data scientists to choose an ϵ that generates a Rashomon set so that any two models in it are indistinguishable.

We introduce a tunable loss function called$\alpha $-loss, parameterized by$\alpha \in (0,\infty]$, which interpolates between the exponential loss ($\alpha = 1/2$), the log-loss ($\alpha = 1$), and the 0–1 loss ($\alpha = \infty $), for the machine learning setting of classification. Theoretically, we illustrate a fundamental connection between$\alpha $-loss and Arimoto conditional entropy, verify the classification-calibration of$\alpha $-loss in order to demonstrate asymptotic optimality via Rademacher complexity generalization techniques, and build-upon a notion called strictly local quasi-convexity in order to quantitatively characterize the optimization landscape of$\alpha $-loss. Practically, we perform class imbalance, robustness, and classification experiments on benchmark image datasets using convolutional-neural-networks. Our main practical conclusion is that certain tasks may benefit from tuning$\alpha $-loss away from log-loss ($\alpha = 1$), and to this end we provide simple heuristics for the practitioner. In particular, navigating the$\alpha $hyperparameter can readily provide superior model robustness to label flips ($\alpha > 1$) and sensitivity to imbalanced classes ($\alpha < 1$).

Disparate treatment occurs when a machine learning model produces different decisions for groups of individuals based on a sensitive attribute (e.g., age, sex). In domains where prediction accuracy is paramount, it could potentially be acceptable to fit a model which exhibits disparate treatment. To evaluate the effect of disparate treatment, we compare the performance of split classifiers (i.e., classifiers trained and deployed separately on each group) with group-blind classifiers (i.e., classifiers which do not use a sensitive attribute). We introduce the benefit-of-splitting for quantifying the performance improvement by splitting classifiers when the underlying data distribution is known. Computing the benefit-of-splitting directly from its definition involves solving optimization problems over an infinite-dimensional functional space. Under different performance measures, we (i) prove an equivalent expression for the benefit-of-splitting which can be efficiently computed by solving small-scale convex programs; (ii) provide sharp upper and lower bounds for the benefit-of-splitting which reveal precise conditions where a group-blind classifier will always suffer from a non-trivial performance gap from the split classifiers. A full version of this paper is accessible at [1].

The minimum mean-square error (MMSE) achievable by optimal estimation of a random variable$S$given another random variable$T$is of much interest in a variety of statistical contexts. Motivated by a growing interest in auditing machine learning models for unintended information leakage, we propose a neural network-based estimator of this MMSE. We derive a lower bound for the MMSE based on the proposed estimator and the Barron constant associated with the conditional expectation of$S$given$T$. Since the latter is typically unknown in practice, we derive a general bound for the Barron constant that produces order optimal estimates for canonical distribution models.

Disparate treatment occurs when a machine learning model produces different decisions for individuals based on a legally protected or sensitive attribute (e.g., age, sex). In domains where prediction accuracy is paramount, it could potentially be acceptable to fit a model which exhibits disparate treatment. To evaluate the effect of disparate treatment, we compare the performance of split classifiers (i.e., classifiers trained and deployed separately on each group) with group-blind classifiers (i.e., classifiers which do not use a sensitive attribute). We introduce the benefit-of-splitting for quantifying the performance improvement by splitting classifiers. Computing the benefit-of-splitting directly from its definition could be intractable since it involves solving optimization problems over an infinite-dimensional functional space. Under different performance measures, we (i) prove an equivalent expression for the benefit-of-splitting which can be efficiently computed by solving small-scale convex programs; (ii) provide sharp upper and lower bounds for the benefit-of-splitting which reveal precise conditions where a group-blind classifier will always suffer from a non-trivial performance gap from the split classifiers. In the finite sample regime, splitting is not necessarily beneficial and we provide data-dependent bounds to understand this effect. Finally, we validate our theoretical results through numerical experiments on both synthetic and real-world datasets.

We investigate the framework of privacy amplification by iteration, recently proposed by Feldman et al., from an information-theoretic lens. We demonstrate that differential privacy guarantees of iterative mappings can be determined by a direct application of contraction coefficients derived from strong data processing inequalities for f-divergences. In particular, by generalizing the Dobrushin's contraction coefficient for total variation distance to an f-divergence known as Eγ-divergence, we derive tighter bounds on the differential privacy parameters of the projected noisy stochastic gradient descent algorithm with hidden intermediate updates.

We analyze the optimization landscape of a recently introduced tunable class of loss functions called α-loss, α ∈ (0, ∞], in the logistic model. This family encapsulates the exponential loss (α = 1/2), the log-loss (α = 1), and the 0-1 loss (α = ∞) and contains compelling properties that enable the practitioner to discern among a host of operating conditions relevant to emerging learning methods. Specifically, we study the evolution of the optimization landscape of α-loss with respect to α using tools drawn from the study of strictly-locally-quasi-convex functions in addition to geometric techniques. We interpret these results in terms of optimization complexity via normalized gradient descent.

Consider a data publishing setting for a dataset composed by both private and non-private features. The publisher uses an empirical distribution, estimated from n i.i.d. samples, to design a privacy mechanism which is applied to new fresh samples afterward. In this paper, we study the discrepancy between the privacy-utility guarantees for the empirical distribution, used to design the privacy mechanism, and those for the true distribution, experienced by the privacy mechanism in practice. We first show that, for any privacy mechanism, these discrepancies vanish at speed O(1/√n) with high probability. These bounds follow from our main technical results regarding the Lipschitz continuity of the considered information leakage measures. Then we prove that the optimal privacy mechanisms for the empirical distribution approach the corresponding mechanisms for the true distribution as the sample size n increases, thereby establishing the statistical consistency of the optimal privacy mechanisms. Finally, we introduce and study uniform privacy mechanisms which, by construction, provide privacy to all the distributions within a neighborhood of the estimated distribution and, thereby, guarantee privacy for the true distribution with high probability.

We present α-loss, α ∈ [1, ∞], a tunable loss function for binary classification that bridges log-loss (α = 1) and 0-1 loss (α = ∞). We prove that α-loss has an equivalent margin-based form and is classification-calibrated, two desirable properties for a good surrogate loss function for the ideal yet intractable 0-1 loss. For logistic regression-based classification, we provide an upper bound on the difference between the empirical and expected risk for α-loss at the critical points of the empirical risk by exploiting its Lipschitzianity along with recent results on the landscape features of empirical risk functions. Finally, we show that α-loss with α = 2 performs better than log-loss on MNIST for logistic regression.

We capitalize on the Wasserstein distance to obtain two information-theoretic bounds on the generalization error of learning algorithms. First, we specialize the Wasserstein distance into total variation, by using the discrete metric. In this case we derive a generalization bound and, from a strong data-processing inequality, show how to narrow the bound by adding Gaussian noise to the output hypothesis. Second, we consider the Wasserstein distance under a generic metric. In this case we derive a generalization bound by exploiting the geometric nature of the Kantorovich-Rubinstein duality theorem. We illustrate the use of these bounds with examples. Our bounds can handle certain cases in which existing bounds via mutual information fail.

We investigate the problem of estimating a random variable Y under a privacy constraint dictated by another correlated random variable X. When X and Y are discrete, we express the underlying privacy-utility tradeoff in terms of the privacy-constrained guessing probability (PXY, ε), and the maximum probability Pc(Y|Z) of correctly guessing Y given an auxiliary random variable Z, where the maximization is taken over all PZ|Yensuring that Pc(X|Z) ≤ ε for a given privacy threshold ε ≥ 0. We prove that ħ (PXY, ·) is concave and piecewise linear, which allows us to derive its expression in closed form for any ε when X and Y are binary. In the non-binary case, we derive (PXY, ε) in the high-utility regime (i.e., for sufficiently large, but nontrivial, values of ε) under the assumption that Y and Z have the same alphabets. We also analyze the privacy-constrained guessing probability for two scenarios in which X, Y, and Z are binary vectors. When X and Y are continuous random variables, we formulate the corresponding privacy-utility tradeoff in terms of sENSR(PXY, ε), the smallest normalized minimum mean squared-error (mmse) incurred in estimating Y from a Gaussian perturbation Z. Here, the minimization is taken over a family of Gaussian perturbations Z for which the mmse of f (X) given Z is within a factor 1-ε from the variance of f (X) for any non-constant real-valued function f . We derive tight upper and lower bounds for sENSR when Y is Gaussian. For general absolutely continuous random variables, we obtain a tight lower bound for sENSR(PXY, ε) in the high privacy regime, i.e., for small ε.

Consider a data publishing setting for a data set with public and private features. The objective of the publisher is to maximize the amount of information about the public features in a revealed data set, while keeping the information leaked about the private features bounded. The goal of this paper is to analyze the performance of privacy mechanisms that are constructed to match the distribution learned from the data set. Two distinct scenarios are considered: (i) mechanisms are designed to provide a privacy guarantee for the learned distribution; and (ii) mechanisms are designed to provide a privacy guarantee for every distribution in a given neighborhood of the learned distribution. For the first scenario, given any privacy mechanism, upper bounds on the difference between the privacy-utility guarantees for the learned and true distributions are presented. In the second scenario, upper bounds on the reduction in utility incurred by providing a uniform privacy guarantee are developed.

We investigate the problem of guessing a discrete random variable Y under a privacy constraint dictated by another correlated discrete random variable X, where both guessing efficiency and privacy are assessed in terms of the probability of correct guessing. We define h(PXY,ε) as the maximum probability of correctly guessing Y given an auxiliary random variable Z, where the maximization is taken over all PZ|Yensuring that the probability of correctly guessing X given Z does not exceed ε. We show that the map ε → h(PXY,ε) is strictly increasing, concave, and piecewise linear, which allows us to derive a closed form expression for h(PxY,ε) when X and Y are connected via a binary-input binary-output channel. For {(Xi, Yi)}ni=1being pairs of independent and identically distributed binary random vectors, we similarly define h_n(PX n Y n, ε) under the assumption that Znis also a binary vector. Then we obtain a closed form expression for h_n(PX n Y n, ε) for sufficiently large, but nontrivial values of ε.

In this paper, we consider point-to-point multiantenna channels with certain block distributional symmetries that do not require the entries of the channel matrix to be either Gaussian, or independent, or identically distributed. A main contribution is a capacity theorem for these channels, which might be regarded as a generalization of Telatar's theorem (1999), which reduces the numerical optimization domain in the capacity computation. With this information theoretic result and some free probability arguments, we prove an asymptotic capacity theorem that, in addition to reducing the optimization domain, does not depend on the dimension of the channel matrix. This theorem allows us to apply free probability techniques to numerically compute the asymptotic capacity of the channels under consideration. These theorems provide a very efficient method for numerically approximating both the capacity and a capacity achieving input covariance matrix of certain channels.

In this paper we study the capacity achieving input covariance matrices of a single user multiantenna channel based solely on the group of symmetries of its matrix of propagation coefficients. Our main result, which unifies and improves the techniques used in a variety of classical capacity theorems, uses the Haar (uniform) measure on the group of symmetries to establish the existence of a capacity achieving input covariance matrix in a very particular subset of the covariance matrices. This result allows us to provide simple proofs for old and new capacity theorems. Among other results, we show that for channels with two or more standard symmetries, the isotropic input is optimal. Overall, this paper provides a precise explanation of why the capacity achieving input covariance matrices of a channel depend more on the symmetries of the matrix of propagation coefficients than any other distributional assumption.