In this paper, we consider the parallel implementation of an already-trained deep model on multiple processing nodes (a.k.a. workers). Specifically, we investigate as to how a deep model should be divided into several parallel sub-models, each of which is executed efficiently by a worker. Since latency due to synchronization and data transfer among workers negatively impacts the performance of the parallel implementation, it is desirable to have minimum interdependency among parallel sub-models. To achieve this goal, we propose to rearrange the neurons in the neural network, partition them (without changing the general topology of the neural network), and modify the weights such that the interdependency among sub-models is minimized under the computations and communications constraints of the workers while minimizing its impact on the performance of the model. We propose RePurpose, a layer-wise model restructuring and pruning technique that guarantees the performance of the overall parallelized model. To efficiently apply RePurpose, we propose an approach based on L0 optimization and the Munkres assignment algorithm. We show that, compared to the existing methods, RePurpose significantly improves the efficiency of the distributed inference via parallel implementation, both in terms of communication and computational complexity.

The highly directional nature of mmWave channels results in a mutlipath incoming signal, often with varying power levels. To exploit the complete diversity of this channel, beamformer design should incorporate this multipath. This increases pilot overhead for initial access. However, low latency mmWave signalling protocols require minimal pilot transmission. Additionally, practical system implementations of beamformers use low-complexity phase-shifter (PS) beamformers. Balancing performance, latency and hardware constraints, active learning has proven to be an extremely effective strategy for initial channel access. However, modern active learning algorithms are tailored to line-of-sight (LoS) angular estimation and tracking. We address multipath beamformer design via deep learning, which is increasingly used for channel estimation and end-to-end communication. We develop two novel deep neural networks (DNN) for multipath beamformer learning: (i) Deep Unfolded Beamformer Learning and (ii) Deep Recurrent Beamformer Learning. Our approach improves on active learning for LoS paths by utilizing multipath diversity for increased reference signal received power (RSRP).

In many diverse fields, distributed IoT devices perform collaborative inference by communicating with an edge router. Often sensory data contains sensitive attributes that should not be revealed to the router. To address this, we develop, to the best of our knowledge, the first privacy-aware machine learning framework for distributed functional compression over AWGN channels. The key feature of our approach to privacy is that we focus only on sensitive attributes of data rather than paying a high cost to protect everything. Employing a mutual information based privacy constraint, we first propose a novel approximate upper bound to protect sensitive attributes in the compressed representations of the sensory data. Next, in conjunction with the upper bound, we propose an adversarial lower bound to enhance the protection further. Thirdly, we propose novel decompositions to these bounds such distributed edge devices can ensure overall privacy by independently privatizing their components. This allows us to propose an enhanced privacy-aware algorithm that protects sensitive information during training and inference. Our experiments show that the privacy-utility trade-off from our proposed methods is significantly better than existing mechanisms.

In this paper, we study Joint Source-Channel Coding (JSCC) for distributed analog functional compression over both Gaussian Multiple Access Channel (MAC) and AWGN channels. Notably, we propose a deep neural network based solution for learning encoders and decoders. We propose three methods of increasing performance. The first one frames the problem as an autoencoder; the second one incorporates the power constraint in the objective by using a Lagrange multiplier; the third method derives the objective from the information bottleneck principle. We show that all proposed methods are variational approximations to upper bounds on the indirect rate-distortion problem's minimization objective. Further, we show that the third method is the variational approximation of a tighter upper bound compared to the other two. Finally, we show empirical performance results for image classification. We compare with existing work and showcase the performance improvement yielded by the proposed methods.

This paper proposes a general framework to design a sparse sensing matrix ${\mathbf {A}} \in \mathbb{R}^{m\times n}$, in a linear measurement system ${\mathbf {y = Ax}}^{\sharp } + {\mathbf {w}}$, where ${\mathbf {y}} \in \mathbb{R}^{n}, {\mathbf {x}}^{\sharp } \in \mathbb{R}^{n}$, and w denote the measurements, the signal with certain structures, and the measurement noise, respectively. By viewing the signal reconstruction from the measurements as a message passing algorithm over a graphical model, we leverage tools from coding theory in the design of low density parity check codes, namely the density evolution, and provide a framework for the design of matrix A. Particularly, compared to the previous methods, our proposed framework enjoys the following desirable properties: (i) Universality: the design supports both regular sensing and preferential sensing, and incorporates them in a single frame-work; (ii) Flexibility: the framework can easily adapt the design of A to a signal $x^{\sharp }$ with different underlying structures. As an illustration, we consider the $\ell_{1}$ regularizer, which correspond to Lasso, for both the regular sensing and preferential sensing scheme. Noteworthy, our framework can reproduce the classical result of Lasso, i.e., $m \geq c_{0}k\log (n/k)$ (the regular sensing) with regular design after proper distribution approximation, where $c_{0}\gt 0$ is some fixed constant. We also provide numerical experiments to confirm the analytical results and demonstrate the superiority of our framework whenever a preferential treatment of a sub-block of vector $x^{\sharp }$ is required.

In this paper, we present a learning scheme for Joint Source-Channel Coding (JSCC) over analog independent additive noise channels. We formulate the learning problem by showing that the minimization loss function from rate-distortion theory, is upper bounded by the loss function of the Variational Autoencoder (VAE). We show that when the source dimension is greater than the channel dimension, the encoding of two source samples in the neighborhood of each other need not be near each other. Such discontinuous projection needs to be accounted for by using multiple encoders and selecting an encoder to encode samples on a particular side of the discontinuity. We explore two selection methodologies, one based on an intuitive rule and the other where it is posed as a learning task in a Mixture-of-Experts (MoE) setup. We analyze the gradients of these methods and reason why the latter is better at avoiding local optima. We show the efficacy of the proposed methodology by simulating the performance of the system for JSCC of Gaussian sources over AWGN channels and showing that the learned solutions are close to or better than the ones proposed earlier. The proposed methodology is also naturally capable of generalizing to other source distributions which we showcase by simulating for Laplace sources. The learned systems are also robust to changes in channel conditions. Further, a single system can be trained to generalize over a range of channel conditions provided the channel conditions are known at both the transmitter and the receiver. Finally, we evaluate our proposed methodology on three different image datasets and showcase consistent improvement over existing methods due to the VAE formulation.

In distributed training of deep models, the transmission volume of stochastic gradients (SG) imposes a bottleneck in scaling up the number of processing nodes. On the other hand, the existing methods for compression of SGs have two major drawbacks. First, due to the increase in the overall variance of the compressed SG, the hyperparameters of the learning algorithm must be readjusted to ensure the convergence of the training. Further, the convergence rate of the resulting algorithm still would be adversely affected. Second, for those approaches for which the compressed SG values are biased, there is no guarantee for the learning convergence and thus an error feedback is often required. We propose Quantized Compressive Sampling (QCS) of SG that addresses the above two issues while achieving an arbitrarily large compression gain. We introduce two variants of the algorithm: Unbiased-QCS and MMSE-QCS and show their superior performance w.r.t. other approaches. Specifically, we show that for the same number of communication bits, the convergence rate is improved by a factor of 2 relative to state of the art. Next, we propose to improve the convergence rate of the distributed training algorithm via a weighted error feedback. Specifically, we develop and analyze a method to both control the overall variance of the compressed SG and prevent the staleness of the updates. Finally, through simulations, we validate our theoretical results and establish the superior performance of the proposed SG compression in the distributed training of deep models. Our simulations also demonstrate that our proposed compression method expands substantially the region of step-size values for which the learning algorithm converges.

Transmitting the gradients or model parameters is a critical bottleneck in distributed training of large models. To mitigate this issue, we propose an indirect quantization and compression of stochastic gradients (SG) via factorization. The gist of the idea is that, in contrast to the direct compression methods, we focus on the factors in SGs, i.e., the forward and backward signals in the backpropagation algorithm. We observe that these factors are correlated and generally sparse in most deep models. This gives rise to rethinking of the approaches for quantization and compression of gradients with the ultimate goal of minimizing the error in the final computed gradients subject to the desired communication constraints. We have proposed and theoretically analyzed different indirect SG quantization (ISGQ) methods. The proposed ISGQ reduces the reconstruction error in SGs compared to the direct quantization methods with the same number of quantization bits. Moreover, it can achieve compression gains of more than 100, while the existing traditional quantization schemes can achieve compression ratio of at most 32 (quantizing to 1 bit). Further, for a fixed total batch-size, the required transmission bit-rate per worker decreases in ISGQ as the number of workers increases.

In this paper, we propose a deep neural network framework for Joint Source-Channel Coding of an m dimensional i.i.d. Gaussian source for transmission over a single additive white Gaussian noise channel with no delay. The framework employs two neural encoder-decoder pairs that learn to split the input signal space into two disjoint support sets. The encoder and the decoder are jointly trained to minimize the mean square error subject to a power constraint on the signal transmitted across the channel. The proposed method achieves results as good as the state of the art for m=3,4 and is easily extendable to higher dimensions. The trained model, we discovered, assigns almost equal probability to the disjoint support sets. The results show that the scheme performance is within 1.9dB of the Shannon optimal limit over a wide range of Channel Signal to Noise Ratios (CSNR) from 0dB to 30dB for various values of m. The method is also robust, i.e. employing a model trained at CSNR+/-5dB is only 0.6dB worse than a model trained specifically for that CSNR.

Performance of the reconstruction algorithms in compressed sensing largely depends on the characteristics of measurement matrices. As such, the construction and analysis of the measurement matrix is of paramount interest. In this paper, for the first time, we focus on the class of sparse sensing matrices with (non-negative) integer entries. This problem, among other applications, is particularly motivated by the constraint of measuring gene regulatory expressions. We study randomly generated matrices from the integer family and analyze their properties in terms of the covariance and RIP constant. We derive bounds for the coherence and RIP constant of such measurement matrices. Further, apart from the coherence, we find that the RIP constant is closely related to the minimum non-diagonal entry ρnin the covariance matrix, which is rarely studied before.

In this paper, we study joint source-channel coding of gaussian sources over multiple AWGN channels where the source dimension is greater than the number of channels. We model our system as a Variational Autoencoder and show that its loss function takes up a form that is an upper bound on the optimization function got from rate-distortion theory. The constructed system employs two encoders that learn to split the source input space into almost half with no constraints. The system is jointly trained in a data-driven manner, end-to-end. We achieve state of the art results for certain configurations, some of which are 0.7dB better than previous works. We also showcase that the trained encoder/decoder is robust, i.e., even if the channel conditions change by +/-5dB, the performance of the system does not vary by more than 0.7dB w.r.t. a system trained at that channel condition. The trained system, to an extent, has the ability to generalize when a single input dimension is dropped and for some scenarios it is less than 1dB away from the system trained for that reduced dimension.

In this paper, we study a general framework for compressive sensing assuming the existence of the prior knowledge that x* belongs to the union of multiple convex sets, x* ε υi ℒi. In fact, by proper choices of these convex sets in the above framework, the problem can be transformed to well known CS problems such as the phase retrieval, quantized compressive sensing, and model-based CS. First we analyze the impact of this prior knowledge on the minimum number of measurements M to guarantee the uniqueness of the solution. Then we formulate a universal objective function for signal recovery, which is both computationally inexpensive and flexible. Then, an algorithm based on multiplicative weight update and proximal gradient descent is proposed and analyzed for signal reconstruction. Finally, we investigate as to how we can improve the signal recovery by introducing regularizers into the objective function.

In general, uncertainties in the sensing matrix weakens the system performance and reduces the reliability of recovered signals. In some applications, the sign values of signals instead of their exact values may be needed. In this paper, we show that as long as the uncertainty in the sensing matrix is sparse, a thresholding mechanism can be developed to recover the sign vector. In particular, provided that the true signal satisfies certain conditions, the exact sign vector can be recovered with high probability even under uncertain sensing matrices. Simulations are also presented to verify our theoretical results.

We consider a non-stationary data stream in which the data statistics may change abruptly from one sample to another, i.e. each sample might be generated from a different (unknown) source in a mixture of K sources. The problem of identifying the models and parameters of K sources, as well as the source switching model is investigated. We proposed an algorithm based on Bayesian Information Criterion and Expectation Maximization to determine the models and estimate the mixture parameters. The estimated data generation model can be used in memory-assisted universal compression to decrease the coding rate further. Simulation results confirmed that using the proposed algorithm for source identification and universal compression can significantly decrease the compression redundancy.

We consider the problem of learning dictionaries for data compression. Different from ordinary learning methods, the objective is to design a dictionary such that the signal has a low entropy representation in the basis of the dictionary, rather than giving a sparse or low-energy representation. To achieve this goal, we need to consider the effect of quantization on the rate-distortion curve as well as an estimation of the distributions of the coefficients. Based on this probability estimation, the coefficients are computed, quantized and then entropy-coded. As such, we have developed algorithms for different classes of dictionaries; orthonormal, union of orthonormals and general dictionaries with unit-norm atoms, to iteratively learn the dictionary and the distribution models of the coefficients. A mixture of Gaussians is adopted to estimate the probability and is updated using the expectation maximization algorithm together with the dictionary learning. Simulation results on the real seismic data show the effectiveness of the proposed algorithm compared to ordinary dictionary learning methods.

The sensor selection problem arises in many applications ranging from sensor networks for event detection to determining concentrations of bio-markers for disease detection. In this paper, we assume that in addition to noise, there exist interference signals (which can be correlated with the desired signals) corrupting the measurements. We consider two different criteria to measure the performance of the selected sensors; average error and minimax analysis. For each case, the cost function is defined over the reconstruction algorithm (or matrix in the linear case), which in turn, explicitly determines the selected sensors. Therefore, minimizing the cost function with some sparsity constraints on the reconstruction algorithm results in the best subset of sensors and as to how we recover the desired signals from the selected measurements. In this paper, we consider the problem for the linear measurement system in various settings and derive the optimization problems. Finally, we propose various methods to solve these problems, and show the effectiveness of the proposed algorithms through simulations.

In this paper, we propose using stochastic framework for computations by biological cells. The key observation is that the input molecules activate receptors of a biological cell independently with probability p that is dependent on the molecule's concentration. Hence, the (active/inactive) states of the receptors can be viewed as a stochastic number representing the input concentration or probability p. We construct the addition operation via a cell having two different types of receptors. We also develop the multiplication by using a receptor that is active (with some probability) when both types of input molecules present. We analyze the computing accuracy of a cell w.r.t. parameters such as number of receptors and the input sensitivity of the cell.

In certain applications, recovering the signs of values may be more critical than the values themselves. Inspired by advances of sparse recovery of signals with fewer measurements, we would like to study the sign recovery problem and generalize it from a linear case to a non-linear setup. We focus on the sign values in quadratic measurement systems and provide theorems for the consistency condition, which ensures the signs are recovered correctly with probability close to 1. In deriving the consistency condition, we adopt a new penalty term using the trace operation and transform the optimization problem to the widely known Lasso problem. We also present simulation results to verify the correctness of our theorems.

In many sparse sensing applications, it is desirable to limit not only the number of non-zero variables but also the amplitude or energy of the signal, i.e., the non-zero variables are expected to be in a certain range. One approach to incorporate the energy constraint is using objective functions such as IIxII22+ λ||χ||ο In this paper, we consider minimizing this objective function, given the linear sensing system y = Ax. As this optimization problem is not convex, we first find the convex envelope of the objective function and then analyze the relation between the uniqueness of the solution and the required number of sensors. Further, we show that the sparsity of the measurement matrix A has negative effects on the required number of sensors.

We introduce and analyze a new technique for model reduction for deep neural networks. While large networks are theoretically capable of learning arbitrarily complex models, overfitting and model redundancy negatively affects the prediction accuracy and model variance. Our Net-Trim algorithm prunes (sparsifies) a trained network layer-wise, removing connections at each layer by solving a convex optimization program. This program seeks a sparse set of weights at each layer that keeps the layer inputs and outputs consistent with the originally trained model. The algorithms and associated analysis are applicable to neural networks operating with the rectified linear unit (ReLU) as the nonlinear activation. We present both parallel and cascade versions of the algorithm. While the latter can achieve slightly simpler models with the same generalization performance, the former can be computed in a distributed manner. In both cases, Net-Trim significantly reduces the number of connections in the network, while also providing enough regularization to slightly reduce the generalization error. We also provide a mathematical analysis of the consistency between the initial network and the retrained model. To analyze the model sample complexity, we derive the general sufficient conditions for the recovery of a sparse transform matrix. For a single layer taking independent Gaussian random vectors of length $N$ as inputs, we show that if the network response can be described using a maximum number of $s$ non-zero weights per node, these weights can be learned from $\mathcal{O}(s\log N)$ samples.

The problem of designing optimum filter banks for different applications is a popular research subject. It has also been shown that principal component filter banks (PCFB) are the optimum filter bank for many application. Existing methods to design FIR PCFBs are based on designing energy compaction filters. In this work we concentrate on designing FIR PCFBs with the same frequency response as the ideal one. The presented approach results in filter banks with a very good approximation of the ideal PCFB, as verified by simulations.