Federated Learning is expected to provide strong privacy guarantees, as only gradients or model parameters but no plain text training data is ever exchanged either between the clients or between the clients and the central server. In this paper, we challenge this claim by introducing a simple but still very effective membership inference attack algorithm, which relies only on a single training step. In contrast to the popular honest-but-curious model, we investigate a framework with a dishonest central server. Our strategy is applicable to models with ReLU activations and uses the properties of this activation function to achieve perfect accuracy. Empirical evaluation on visual classification tasks with MNIST, CIFAR10, CIFAR100 and CelebA datasets show that our method provides perfect accuracy in identifying one sample in a training set with thousands of samples. Occasional failures of our method lead us to discover duplicate images in the CIFAR100 and CelebA datasets

We consider a distributed detection problem with statistically spatial dependent measurements in a sensor network, when there is not a fusion center. Thus, each node takes some measurements, does some processing, exchanges messages with its neighbors and finally makes a decision (typically the same for all nodes) about the phenomenon of interest. A cooperative algorithm is proposed for reducing the number of communications between sensors and thus make an efficient use of the energy budget of a wireless sensor network (WSN). The problem is formulated as a composite hypothesis test using a general probability density function with unknown parameters leading naturally to the use of the generalized likelihood ratio (GLR) test. As the sensors observe statistically spatial dependent samples, which makes difficult the implementation of fully distributed detection procedures, we propose a simpler algorithm for making a decision about the true hypothesis. We also compute its asymptotic distribution to characterize its performance. Interestingly, despite the fact that our proposal is more simple and efficient to implement than the GLR test, we find relevant scenarios for which it outperforms the latter, even in finite length regimes.

We consider the problem of distributed detection over a network of sensors using energy measurements. Assuming that the problem is to detect the presence of stochastic signal with arbitrary temporal correlation, we exactly compute the characteristic function of the energy measurement at each sensing node. As it is shown, when the time-bandwidth product W T is large enough, the exact joint probability density function of the measurements at the sensor sites can be closely approximated by a product of non-trivial density functions that depend of some unknown parameters of the source signal and the sensor network. As those parameters can be easily estimated at each sensor node, we propose a modified generalized likelihood ratio test (GLRT). Through numerical experiments we show that this proposal has performance close to the optimal one which use the exact joint density function and with a significant reduction in the consumption of network resources as bandwidth and transmission power.

Recent works suggests an interesting interplay between the information flow between inputs features and hidden representations of a learning and the ability of the algorithm to generalize from trained samples to unobserved data. For instance, some of regularization techniques used to control generalization are expected to impact the corresponding information metrics. In this work, we study mutual information in Restricted Boltzmann Machines (RBM) and its relationship with the different regularization techniques. Our results show some evidence on interesting connections between the mutual information (inputs and its representations) with relevant parameters such as: network dimension, matrix norms and dropout probability, which are known to influence the generalization ability of the network. Results are empirically corroborated with a numerical study.

One of the main challenges in Internet of Things (IoT) is providing communication support to an increasing number of connected devices. In recent years, the narrowband radio technology has emerged to address this situation: narrowband IoT (NB-IoT), which is now part of 5G. Supporting massive connectivity becomes particularly demanding in extreme coverage scenarios, such as underground or deep inside building sites. We propose a novel strategy for these situations focused on optimizing NB-IoT shared channels through the selection of link parameters: modulation and coding scheme, as well as the number of repetitions. These parameters are established by the base station (BS) for each block transmitted until reaching a target block error rate (BLERt). A wrong selection of these magnitudes leads to radio resource waste and a decrease in the number of possible concurrent connections. Specifically, our strategy is based on a lookup table (LUT) scheme which is used for rapidly delivering the optimal link parameters given a target QoS. To validate our proposal, we compare with alternative strategies using an open-source NB-IoT uplink simulator. The experiments are based on transmitting blocks of 256 bits using an AWGN channel over the NPUSCH. Results show that, especially under extreme conditions, only a few options for link parameters are available, favoring robustness against measurement uncertainties. Our strategy minimizes resource usage in all scenarios of the acknowledged mode and remarkably reduces losses in the unacknowledged mode, presenting also substantial gains in performance. We expect to influence the future BS software design and implementation, favoring connection support under extreme environments.

This paper investigates a multi-terminal source coding problem under a logarithmic loss fidelity which does not necessarily lead to an additive distortion measure. The problem is motivated by an extension of the information bottleneck method to a multi-source scenario where several encoders have to build cooperatively rate-limited descriptions of their sources in order to maximize information with respect to other unobserved (hidden) sources. More precisely, we study fundamental information-theoretic limits of the so-called: 1) two-way collaborative information bottleneck (TW-CIB) and 2) the collaborative distributed information bottleneck (CDIB) problems. The TW-CIB problem consists of two distant encoders that separately observe marginal (dependent) components X1and X2and can cooperate through multiple exchanges of limited information with the aim of extracting information about hidden variables (Y1, Y2), which can be arbitrarily dependent on (X1, X2). On the other hand, in CDIB, there are two cooperating encoders which separately observe X1and X2and a third node which can listen to the exchanges between the two encoders in order to obtain information about a hidden variable Y. The relevance (figure-of-merit) is measured in terms of a normalized (per-sample) multi-letter mutual information metric (log-loss fidelity), and an interesting tradeoff arises by constraining the complexity of descriptions, measured in terms of the rates needed for the exchanges between the encoders and decoders involved. Inner and outer bounds to the complexity-relevance region of these problems are derived from which optimality is characterized for several cases of interest. Our resulting theoretical complexity-relevance regions are finally evaluated for binary symmetric and Gaussian statistical models, showing theoretical tradeoffs between the complexity-constrained descriptions and their relevance with respect to the hidden variables.

This paper investigates, from information theoretic principles, a learning problem based on the principle that any regularity in a given dataset can be exploited to extract compact features from data, in order to build meaningful representations of a relevant content. We begin by introducing the fundamental tradeoff between the average risk and the model complexity. Interestingly, our formulation allows an information theoretic formulation of the multi-task learning (MTL) problem. Then, we present an iterative algorithm for computing the optimal tradeoffs. Remarkably, empirical results illustrate that there exists an optimal information rate minimizing the excess risk which depends on the nature and the amount of available training data. An application to hierarchical text categorization is also investigated, extending previous works.

A grand challenge in representation learning is the development of computational algorithms that learn the different explanatory factors of variation behind high-dimensional data. Encoder models are usually determined to optimize performance on training data when the real objective is to generalize well to other (unseen) data. Although numerical evidence suggests that noise injection at the level of representations might improve the generalization ability of the resulting encoders, an information-theoretic justification of this principle remains elusive. In this work, we derive an upper bound to the so-called generalization gap corresponding to the cross-entropy loss and show that when this bound times a suitable multiplier and the empirical risk are minimized jointly, the problem is equivalent to optimizing the Information Bottleneck objective with respect to the empirical data-distribution. We specialize our general conclusions to analyze the dropout regularization method in deep neural networks, explaining how this regularizer helps to decrease the generalization gap.

This paper investigates a scenario where two distant nodes separately observe memoryless process, namely X1and X2, and can cooperate through multiple exchanges of messages with the goal of enabling a third node to learn “relevant information” (measured in terms of a multi-letter mutual information) about some hidden memoryless process Y, which is arbitrarily dependent on (X1, X2). These interactive exchanges yield an explicit cooperation that helps the third node to identify, from the distributed observations X1and X2, useful features for the inference of Y. An inner and an outer bound to the rate-relevance region of this problem is derived. Optimal characterization of the rate-relevance region under two different conditions on the dependence structures of the involved variables is showed. Also, two examples for Gaussian sources are studied.

In this letter, we use the Berry-Esseen theorem and the method of tilted distributions to derive a simple tight closed-form approximation for the tail probabilities of a sum of independent but not necessarily identically distributed random variables. We also provide lower and upper bounds. The expression can also be used for computing the cumulative distribution function. We illustrate the accuracy of the method by analyzing some convergence properties of the theoretical approximation and comparing it with previous results in the literature when available and/or numerical results.

In this paper, we explore the three-node multiterminal lossy source coding problem, which seems to offer a formidable mathematical complexity. We derive an inner bound to the general rate-distortion region of this problem, which is a natural extension of the seminal work by Kaspi on the interactive two-terminal source coding problem. It is shown that this (rather involved) inner bound contains several rate-distortion regions of some relevant source coding settings. In this way, besides the non-trivial extension of the interactive two terminal problem, our results can be seen as a generalization and hence unification of several previous works in the field. By specializing the inner bound to particular cases, we obtain some novel rate-distortion regions for several multi-terminal lossy source coding problems.

The rate-distortion regions of cooperative and interactive source coding is studied and characterized in several important special cases. For the general cooperative and interactive three node source coding problem, we present an improved inner bound with respect to the bound derived in [1]. This inner bound is used to characterize the optimal rate-distortion regions of several special cases of three terminal information sharing protocols.

The two-way Information Bottleneck problem, where two nodes exchange information iteratively about two arbitrarily dependent memoryless sources, is considered. Based on the observations and the information exchange, each node is required to extract “relevant information”, measured in terms of the normalized mutual information, from two arbitrarily dependent hidden sources. The optimal trade-off between rates of relevance and complexity, and the number of exchange rounds, is obtained through a single-letter characterization. We further extend the results to the Gaussian case. Applications of our setup arise in the development of collaborative clustering algorithms.

This paper studies the gains, in terms of served requests, attainable through out-of-band device-to-device (D2D) video exchanges in large cellular networks. A stochastic framework, in which users are clustered to exchange videos, is introduced, considering several aspects of this problem, i.e., the video caching policy, user matching for exchanges, and aspects regarding scheduling and transmissions. A family of admissible protocols is introduced: in each protocol the users are clustered by means of a hard-core point process and, within the clusters, video exchanges take place. Two metrics, quantifying the “local” and “global” fractions of video requests served through D2D are defined, and relevant trade-off regions involving these metrics, as well as quality-of-service constraints, are identified. A simple communication strategy is proposed and analyzed to obtain inner bounds to the trade-off regions and to draw conclusions on the performance attainable through D2D. To this end, the analysis of the time-varying interference that the nodes experience and the tight approximations of its Laplace transform are derived.

The Generalized Markov Lemma has been used in the proofs of several multiterminal source coding theorems for finite alphabets. An alternative approach to extend this result to countable infinite sources is proposed. We establish sufficient conditions to guarantee the joint typicality of reproduction sequences of random descriptions that have not been necessarily generated from the product of probability measures. Compared to existing proofs for finite alphabets, our technique is simpler and self-contained. It also offers bounds on the asymptotic tail probability of the typicality event providing a scaling law for a large number of source encoders.

The three-node multiterminal lossy source coding problem is investigated. We derive an inner bound to the general rate-distortion region of this problem which appears to be the natural extension of the seminal work by Kaspi [1] on the interactive two-terminal source coding problem. It is shown that this -rather involved- inner bound contains several rate-distortion regions of some relevant source coding settings. In this way, besides the non-trivial extension of the interactive two terminal problem, our results can be seen as a generalization and hence unification of several previous works in the field.

In this paper, we study the performance, in terms of the asymptotic error probability, of a user that communicates with a destination with the aid of a full-duplex in-band relay. We consider that the network is interference-limited, and interfering users are distributed as a Poisson point process. In this case, the asymptotic error probability is upper bounded by the outage probability (OP). We investigate the outage behavior for well-known cooperative schemes, namely, decode-and-forward (DF) and compress-and-forward (CF) considering fading and path loss. For DF, we determine the exact OP and develop upper bounds that are tight in typical operating conditions. Also, we find the correlation coefficient between source and relay signals that minimizes the OP when the density of interferers is small. For CF, the achievable rates are determined by the spatial correlation of the interferences, and a straightforward analysis is not possible. To handle this issue, we show that the rate with correlated noises is at most one bit worse than with uncorrelated noises and thus find an upper bound on the performance of CF. These results are useful to evaluate the performance and to optimize relaying schemes in the context of full-duplex wireless networks.

This paper investigates the benefits of cooperation and proposes a relay activation strategy for a large wireless network with multiple transmitters. In this framework, some nodes cooperate with a nearby node that acts as a relay, using the decode-and-forward protocol, and others use direct transmission. The network is modeled as an independently marked Poisson point process, and the source nodes may choose their relays from the set of inactive nodes. Although cooperation can potentially lead to significant improvements in the performance of a communication pair, relaying causes additional interference in the network, increasing the average noise that other nodes see. We investigate how source nodes should balance cooperation versus interference to obtain reliable transmissions, and for this purpose, we study and optimize a relay activation strategy with respect to the outage probability. Surprisingly, in the high reliability regime, the optimized strategy consists on the activation of all the relays or none at all, depending on network parameters. We provide a simple closed-form expression that indicates when the relays should be active, and we introduce closed-form expressions that quantify the performance gains of this scheme with respect to a network that only uses direct transmission.

This paper investigates the potential gains of half-duplex unicast strategies in a large wireless network consisting of clusters in which a source node attempts to transmit a message with the aid of other inactive-potential relays- nodes within the cluster. The network is modeled as an independently marked Poisson point process and a slow-fading scenario is considered. A two-phase transmission protocol is studied and an achievable (upper) bound on the asymptotic error probability of any cluster in the network is derived in terms of the outage probability (OP). The proposed scheme takes advantage of the spatial distribution of the cooperating nodes to create a distributed virtual antenna array, thus improving the OP over direct transmission (DT) through cooperative diversity. Finally, we obtain a converse (lower) bound on the OP of any unicast strategy when restricting all admissible protocols to belong to the same-still quite general- class of half-duplex codes that includes the one studied.

This paper investigates the potential gain of cooperation in large wireless networks with multiple sources and relays, where the nodes form an homogeneous Poisson point process. The source nodes may choose their nearest neighbor from the set of inactive nodes as their relay. Although cooperation can potentially lead to significant improvements on the asymptotic error probability of a communication pair, relaying causes additional interference in the network, increasing the average noise. We address the basic question: how should source nodes optimally balance cooperation vs. interference to guarantee reliability in all communication pairs. Based on the decode-and-forward (DF) scheme at the relays, we derive closed-form approximations to the upper bounds on the error probability, averaging over all node positions. Surprisingly, in the small outage probability regime, there is an almost binary behavior that dictates - depending on network parameters - the activation or not of all relay nodes.

Consider the communication of a single user aided by a nearby relay involved in a large and dense wireless network where the nodes form an homogeneous Poisson point process. We assume that the network is working in the interference-limited regime. In this case the asymptotic error probability is bounded from above by the outage probability experienced by the user. We investigate the outage behavior for the well-known cooperative schemes, namely, decode-and-forward (DF) and compress-and-forward (CF). In this setting, the outage events are induced by both fading and the spatial proximity of neighbor nodes who generate the strongest interference and hence the worst communication case. Upper and lower bounds on the asymptotic error probability which are tight in some cases are derived. It is shown that there exists a clear trade-off between the network density and the benefits of user cooperation. These results are useful to evaluate performance and to optimize relaying schemes in the context of large wireless networks.

We introduce a new family of algorithms to exploit sparsity in adaptive filters. It is based on a recently introduced new framework for designing robust adaptive filters. It results from minimizing a certain cost function subject to a time-dependent constraint on the norm of the filter update. Although in general this problem does not have a closed-form solution, we propose an approximate one which is very close to the optimal solution. We take a particular algorithm from this family and provide some theoretical results regarding the asymptotic behavior of the algorithm. Finally, we test it in different environments for system identification and acoustic echo cancellation applications.

A new framework for designing robust adaptive filters is introduced. It is based on the optimization of a certain cost function subject to a time-dependent constraint on the norm of the filter update. Particularly, we will derive a robust variable step-size NLMS algorithm which optimizes the square norm of the a posteriori error subject to the constraint on the norm of the filter change. We also show the link between the proposed algorithm and another one derived using a robust statistics approach. The algorithm is then tested in different environments for system identification and acoustic echo cancelation applications.

A variable regularized affine projection algorithm (VR-APA) is introduced, which does not require the classical step size. Its use is supported from different points of view. First, it has the property of being Hinfinoptimal, providing robust behavior against perturbations and model uncertainties. Second, the time varying regularization parameter is obtained by maximizing the speed of convergence of the algorithm. At each time step, it needs knowledge of the power of the estimation error vector, which can be estimated by averaging observable quantities. Although we first derive it for a linear time invariant (LTI) system, we show that the same expression holds if we consider a time varying system following a first order Markov model. Simulation results are presented to test the performance of the proposed algorithm and to compare it with other schemes under different situations

The aim of a variable step size normalized least-mean-square (VSS-NLMS) algorithm is to try to solve the conflicting requirement of fast convergence and low misadjustment of the NLMS algorithm. Numerous VSS-NLMS algorithms can be found in the literature with a common point for most of them: they may not work very reliably since they depend on several parameters that are not simple to tune in practice. The objective of this letter is twofold. First, we explain a simple and elegant way to derive VSS-NLMS-type algorithms. Second, a new nonparametric VSS-NLMS is proposed that is easy to control and gives good performances in the context of acoustic echo cancellation