Feature selection is a critical step in data-driven applications, reducing input dimensionality to enhance learning accuracy, computational efficiency, and interpretability. Existing state-of-the-art methods often require post-selection retraining and extensive hyperparameter tuning, complicating their adoption. We introduce a novel, non-intrusive feature selection layer that, given a target feature count $k$, automatically identifies and selects the $k$ most informative features during neural network training. Our method is uniquely simple, requiring no alterations to the loss function, network architecture, or post-selection retraining. The layer is mathematically elegant and can be fully described by: \begin{align} \nonumber \tilde{x}_i = a_i x_i + (1-a_i)z_i \end{align} where $x_i$ is the input feature, $\tilde{x}_i$ the output, $z_i$ a Gaussian noise, and $a_i$ trainable gain such that $\sum_i{a_i^2}=k$. This formulation induces an automatic clustering effect, driving $k$ of the $a_i$ gains to $1$ (selecting informative features) and the rest to $0$ (discarding redundant ones) via weighted noise distortion and gain normalization. Despite its extreme simplicity, our method achieves competitive performance on standard benchmark datasets and a novel real-world dataset, often matching or exceeding existing approaches without requiring hyperparameter search for $k$ or retraining. Theoretical analysis in the context of linear regression further validates its efficacy. Our work demonstrates that simplicity and performance are not mutually exclusive, offering a powerful yet straightforward tool for feature selection in machine learning.

The training phase of deep neural networks requires substantial resources and as such is often performed on cloud servers. However, this raises privacy concerns when the training dataset contains sensitive content, e.g., facial or medical images. In this work, we propose a method to perform the training phase of a deep learning model on both an edge device and a cloud server that prevents sensitive content being transmitted to the cloud while retaining the desired information. The proposed privacy-preserving method uses adversarial early exits to suppress the sensitive content at the edge and transmits the task-relevant information to the cloud. This approach incorporates noise addition during the training phase to provide a differential privacy guarantee. We extensively test our method on different facial and medical datasets with diverse attributes using various deep learning architectures, showcasing its outstanding performance. We also demonstrate the effectiveness of privacy preservation through successful defenses against different white-box, deep and GAN-based reconstruction attacks. This approach is designed for resource-constrained edge devices, ensuring minimal memory usage and computational overhead.

Deep neural networks (DNNs) provide state-of-the-art accuracy for vision tasks, but they require significant resources for training. Thus, they are trained on cloud servers far from the edge devices that acquire the data. This issue increases communication cost, runtime, and privacy concerns. In this study, a novel hierarchical training method for DNNs is proposed that uses early exits in a divided architecture between edge and cloud workers to reduce the communication cost, training runtime, and privacy concerns. The method proposes a brand-new use case for early exits to separate the backward pass of neural networks between the edge and the cloud during the training phase. We address the issues of most available methods that, due to the sequential nature of the training phase, cannot train the levels of hierarchy simultaneously or they do it with the cost of compromising privacy. In contrast, our method can use both edge and cloud workers simultaneously, does not share the raw input data with the cloud, and does not require communication during the backward pass. Several simulations and on-device experiments for different neural network architectures demonstrate the effectiveness of this method. It is shown that the proposed method reduces the training runtime for VGG-16 and ResNet-18 architectures by 29% and 61% in CIFAR-10 classification and by 25% and 81% in Tiny ImageNet classification, respectively, when the communication with the cloud is done over a low bit rate channel. This gain in the runtime is achieved, while the accuracy drop is negligible. This method is advantageous for online learning of high-accuracy DNNs on sensor-holding low-resource devices such as mobile phones or robots as a part of an edge-cloud system, making them more flexible in facing new tasks and classes of data.

Preserving privacy is a growing concern in our society where cameras are ubiquitous. In this work, we propose a trainable image acquisition method that removes the sensitive information in the optical domain before it reaches the image sensor. The method benefits from a trainable optical convolution kernel, which transmits the desired information whilst filtering out the sensitive information, making it irretrievable against different privacy attacks in the digital domain. This is in contrast with the current digital privacy-preserving methods that are all vulnerable to direct access attacks. Also, in contrast with most of the previous optical privacy-preserving methods that cannot be trained, our method is data-driven and optimized for the specific application at hand. Moreover, there is no additional computation or power burden on the acquisition system since it works passively in the optical domain and can be even used in conjunction with other privacy-preserving techniques in the digital domain. We demonstrate our new, generic method in several scenarios such as smile or open-mouth detection as the desired attribute while the gender or wearing make-up is filtered out as the sensitive content. Through several experiments, we show that this method is able to reduce around$\mathbf {65}\%$of sensitive content while causing a negligible reduction in the desired information. Moreover, we tested our method by deep reconstruction attack and confirmed the ineffectiveness of this attack to reconstruct the original sensitive content. This new method has different use cases such as feedback systems for smart TV content or outdoor advertising.

Despite the substantial progress made in deep learning in recent years, advanced approaches remain computationally intensive. The trade-off between accuracy and computation time and energy limits their use in real-time applications on low power and other resource-constrained systems. In this paper, we tackle this fundamental challenge by introducing a hybrid optical-digital implementation of a convolutional neural network (CNN) based on engineering of the point spread function (PSF) of an optical imaging system. This is done by coding an imaging aperture such that its PSF replicates a large convolution kernel of the first layer of a pre-trained CNN. As the convolution takes place in the optical domain, it has zero cost in terms of energy consumption and has zero latency independent of the kernel size. Experimental results on two datasets demonstrate that our approach yields more than two orders of magnitude reduction in the computational cost while achieving near-state-of-the-art accuracy, or equivalently, better accuracy at the same computational cost.

We propose a new statistical dictionary learning algorithm for sparse signals that is based on an $\alpha$-stable innovation model. The parameters of the underlying model—that is, the atoms of the dictionary, the sparsity index $\alpha$ and the dispersion of the transform-domain coefficients—are recovered using a new type of probability distribution tomography. Specifically, we drive our estimator with a series of random projections of the data, which results in an efficient algorithm. Moreover, since the projections are achieved using linear combinations, we can invoke the generalized central limit theorem to justify the use of our method for sparse signals that are not necessarily $\alpha$-stable. We evaluate our algorithm by performing two types of experiments: image in-painting and image denoising. In both cases, we find that our approach is competitive with state-of-the-art dictionary learning techniques. Beyond the algorithm itself, two aspects of this study are interesting in their own right. The first is our statistical formulation of the problem, which unifies the topics of dictionary learning and independent component analysis. The second is a generalization of a classical theorem about isometries of $\ell_p$-norms that constitutes the foundation of our approach.

We investigate the performance of wavelet shrinkage methods for the denoising of symmetric-α-stable (SαS) self-similar stochastic processes corrupted by additive white Gaussian noise (AWGN), where α is tied to the sparsity of the process. The wavelet transform is assumed to be orthonormal and the shrinkage function minimizes the mean-square approximation error (MMSE estimator). We derive the corresponding formula for the expected value of the averaged estimation error. We show that the predicted MMSE is a monotone function of a simple criterion that depends on the wavelet and the statistical parameters of the process. Using the calculus of variations, we then optimize this criterion to find the best performing wavelet within the extended family of Meyer wavelets, which are bandlimited. These are compared with the Daubechies wavelets, which are compactly supported in time. We find that the wavelets that are shorter in time (in particular, the Haar basis) are better suited to denoise the sparser processes (say, α1.6, the limit corresponding to the Gaussian case (fBm) with α = 2.

We propose two minimum-mean-square-error (MMSE) estimation methods for denoising non-Gaussian first-order autoregressive (AR(1)) processes. The first one is based on the message passing framework and gives the exact theoretic MMSE estimator. The second is an iterative algorithm that combines standard wavelet-based thresholding with an optimized non-linearity and cycle-spinning. This method is more computationally efficient than the former and appears to provide the same optimal denoising results in practice. We illustrate the superior performance of both methods through numerical simulations by comparing them with other well-known denoising schemes.

A crucial component of steerable wavelets is the radial profile of the generating function in the frequency domain. In this paper, we present an infinite-dimensional optimization scheme that helps us find the optimal profile for a given criterion over the space of tight frames. We consider two classes of criteria that measure the localization of the wavelet. The first class specifies the spatial localization of the wavelet profile, and the second that of the resulting wavelet coefficients. From these metrics and the proposed algorithm, we construct tight wavelet frames that are optimally localized and provide their analytical expression. In particular, one of the considered criterion helps us finding back the popular Simoncelli wavelet profile. Finally, the investigation of local orientation estimation, image reconstruction from detected contours in the wavelet domain, and denoising indicate that optimizing wavelet localization improves the performance of steerable wavelets, since our new wavelets outperform the traditional ones.

In this letter, we aim to identify the optimal isotropic mother wavelet for a given spatial dimension based on a localization criterion. Within the framework of the calculus of variations, we specify an Euler-Lagrange equation for this problem, and we find the unique analytic solutions. In the one- and two-dimensional cases, the derived wavelets are well known.

We study the issue of localization in the context of isotropic wavelet frames. We define a variance-type measure of localization and propose an algorithm based on calculus of variations to minimize this criterion under the constraint of a tight wavelet frame. Based on these calculations, we design the variance-optimal wavelet (VOW). Finally, we demonstrate the advantage of better localization in a practical image-processing task.

Sinusoidal transforms such as the DCT are known to be optimal-that is, asymptotically equivalent to the Karhunen-Loève transform (KLT)-for the representation of Gaussian stationary processes, including the classical AR(1) processes. While the KLT remains applicable for non-Gaussian signals, it loses optimality and, is outperformed by the independent-component analysis (ICA), which aims at producing the most-decoupled representation. In this paper, we consider an extension of the classical AR(1) model that is driven by symmetric-alpha-stable (SαS) noise which is either Gaussian (α = 2) or sparse (0 <; α <; 2). For the sparse (non-Gaussian) regime, we prove that an expansion in a proper wavelet basis (including the Haar transform) is much closer to the optimal orthogonal ICA solution than the classical Fourier-type representations. Our criterion for optimality, which favors independence, is the Kullback-Leibler divergence between the joint pdf of the original signal and the product of the marginals in the transformed domain. We also observe that, for very sparse AR(1) processes (α ≤ 1), the operator-like wavelet transform is indistinguishable from the ICA solution that is determined through numerical optimization.

In this paper, we prove the existence of capacity achieving linear codes with random binary sparse generating matrices over the Binary Symmetric Channel (BSC). The results on the existence of capacity achieving linear codes in the literature are limited to the random binary codes with equal probability generating matrix elements and sparse parity-check matrices. Moreover, the codes with sparse generating matrices reported in the literature are not proved to be capacity achieving for channels other than Binary Erasure Channel. As opposed to the existing results in the literature, which are based on optimal maximum a posteriori decoders, the proposed approach is based on a different decoder and consequently is suboptimal. We also demonstrate an interesting trade-off between the sparsity of the generating matrix and the error exponent (a constant which determines how exponentially fast the probability of error decays as block length tends to infinity). Based on our results, we also propose a channel coding rate achievable by linear codes at a given block length and error probability. Moreover, we prove the existence of capacity achieving linear codes with a given (arbitrarily low) density of ones on rows of the generating matrix. In addition to proving the existence of capacity achieving sparse codes, an important conclusion of our paper is to prove that any arbitrarily selected sequence of sparse generating matrices is capacity achieving with high probability.

In this paper, we explore some of the fundamentals of synchronous Code Division Multiple Access (CDMA) as applied to wireless and optical communication systems under very general settings (of any size) for the user symbols and the signature matrix entries. The channel is modeled by real/complex additive noise of arbitrary distribution. Two problems are addressed. The first problem concerns whether uniquely detectable overloaded matrices exist in the absence of additive noise under these general settings, and if so, whether there are any practical optimum detection algorithms. The second one is about the bounds for the sum channel capacity when user data and signature matrices employ any real or complex alphabets (finite or infinite). In response to the first problem, we have developed practical maximum likelihood detection algorithms for overloaded CDMA systems for a large class of alphabets. In response to the second problem, a general theorem has been developed in which the sum capacity lower bounds with respect to the number of users, spreading gain, and signal-to-noise ratio can be derived. To show the power and utility of the main theorem, a number of sum capacity bounds for special cases are evaluated. An important conclusion of this paper is that the lower and upper bounds of the sum capacity for small/medium-size CDMA systems depend on both the input and the signature symbols; this is contrary to the asymptotic results for large-scale systems reported in the literature (also confirmed in this paper) where the signature symbols and statistics disappear for signature matrices and input vectors with i.i.d. entries. Furthermore, upper and asymptotic bounds are derived and compared to other derivations.

In this paper we propose a new model for the near-far effect in a CDMA system. We derive upper and lower bounds for the maximum near-far effect for errorless transmission. Using these bounds, we propose some near-far resistant codes. Also a very low complexity ML decoder for a subclass of the proposed codes is suggested.

Lower and upper bounds are derived for the sum capacity of synchronous CDMA where the signature matrix and input alphabets are binary or (2p + 1)-ary, in two cases of noiseless and noisy channels. The bounds are very tight in some regions. Interestingly, simulations show that the formulas for noisy systems tend to the ones for noiseless system as noise tends to 0 while it cannot be deduced easily from the formulas. The results give good insights about the extent of the number of users in which errorless communication is possible for a system with a given chip rate.

In this paper we introduce a new class of codes for over-loaded synchronous wireless CDMA systems which increases the number of users for a fixed number of chips without introducing any errors. In addition these codes support active user detection. We derive an upper bound on the number of users with a fixed spreading factor. Also we propose an ML decoder for a subclass of these codes that is computationally implementable. Although for our simulations we consider a scenario that is worse than what occurs in practice, simulation results indicate that this coding/decoding scheme is robust against additive noise. As an example, for 64 chips and 88 users we propose a coding/decoding scheme that can obtain an arbitrary small probability of error which is computationally feasible and can detect active users. Furthermore, we prove that for this to be possible the number of users cannot be beyond 230.

In this paper, we obtain a family of lower bounds for the sum capacity of code-division multiple-access (CDMA) channels assuming binary inputs and binary signature codes in the presence of additive noise with an arbitrary distribution. The envelope of this family gives a relatively tight lower bound in terms of the number of users, spreading gain, and the noise distribution. The derivation methods for the noiseless and the noisy channels are different but when the noise variance goes to zero, the noisy channel bound approaches the noiseless case. The behavior of the lower bound shows that for small noise power, the number of users can be much more than the spreading gain without any significant loss of information (overloaded CDMA). A conjectured upper bound is also derived under the usual assumption that the users send out equally likely binary bits in the presence of additive noise with an arbitrary distribution. As the noise level increases, and/or, the ratio of the number of users and the spreading gain increases, the conjectured upper bound approaches the lower bound. We have also derived asymptotic limits of our bounds that can be compared to a formula that Tanaka obtained using techniques from statistical physics; his bound is close to that of our conjectured upper bound for large scale systems.

In this paper, we introduce a new class of codes for overloaded synchronous wireless and optical code-division multiple-access (CDMA) systems which increases the number of users for fixed number of chips without introducing any errors. Equivalently, the chip rate can be reduced for a given number of users, which implies bandwidth reduction for downlink wireless systems. An upper bound for the maximum number of users for a given number of chips is derived. Also, lower and upper bounds for the sum channel capacity of a binary overloaded CDMA are derived that can predict the existence of such overloaded codes. We also propose a simplified maximum likelihood method for decoding these types of overloaded codes. Although a high percentage of the overloading factor degrades the system performance in noisy channels, simulation results show that this degradation is not significant. More importantly, for moderate values ofEb/N0(in the range of 6-10 dB) or higher, the proposed codes perform much better than the binary Welch bound equality sequences.

In this paper we introduce a new class of codes for over-loaded synchronous wireless and optical CDMA systems which increases the number of users for fixed number of chips without introducing any errors. Equivalently, the chip rate can be reduced for a given number of users, which implies bandwidth reduction for downlink wireless systems. An upper bound for the maximum number of users for a given number of chips is derived. Also, lower and upper bounds for the sum channel capacity of an overloaded CDMA are derived that can predict the existence of such overloaded codes. Although a high percentage of the overloading factor degrades the system performance in noisy channels, simulation results show that this degradation is not significant.