It is widely known that state-of-the-art machine learning models, including vision and language models, can be seriously compromised by adversarial perturbations. It is therefore increasingly relevant to develop capabilities to certify their performance in the presence of the most effective adversarial attacks. Our paper offers a new approach to certify the performance of machine learning models in the presence of adversarial attacks with population level risk guarantees. In particular, we introduce the notion of (α,ζ)-safe machine learning model. We propose a hypothesis testing procedure, based on the availability of a calibration set, to derive statistical guarantees providing that the probability of declaring that the adversarial (population) risk of a machine learning model is less than α (i.e. the model is safe), while the model is in fact unsafe (i.e. the model adversarial population risk is higher than α), is less than ζ. We also propose Bayesian optimization algorithms to determine efficiently whether a machine learning model is (α,ζ)-safe in the presence of an adversarial attack, along with statistical guarantees. We apply our framework to a range of machine learning models - including various sizes of vision Transformer (ViT) and ResNet models - impaired by a variety of adversarial attacks, such as PGDAttack, MomentumAttack, GenAttack and BanditAttack, to illustrate the operation of our approach. Importantly, we show that ViT's are generally more robust to adversarial attacks than ResNets, and large models are generally more robust than smaller models. Our approach goes beyond existing empirical adversarial risk-based certification guarantees. It formulates rigorous (and provable) performance guarantees that can be used to satisfy regulatory requirements mandating the use of state-of-the-art technical tools.

The deployment of large language models (LLMs) raises concerns regarding their cultural misalignment and potential ramifications on individuals and societies with diverse cultural backgrounds. While the discourse has focused mainly on political and social biases, our research proposes a Cultural Alignment Test (Hoftede’s CAT) to quantify cultural alignment using Hofstede’s cultural dimension framework, which offers an explanatory cross-cultural comparison through the latent variable analysis. We apply our approach to quantitatively evaluate LLMs—namely Llama 2, GPT-3.5, and GPT-4—against the cultural dimensions of regions like the United States, China, and Arab countries, using different prompting styles and exploring the effects of language-specific fine-tuning on the models’ behavioural tendencies and cultural values. Our results quantify the cultural alignment of LLMs and reveal the difference between LLMs in explanatory cultural dimensions. Our study demonstrates that while all LLMs struggle to grasp cultural values, GPT-4 shows a unique capability to adapt to cultural nuances, particularly in Chinese settings. However, it faces challenges with American and Arab cultures. The research also highlights that fine-tuning LLama 2 models with different languages changes their responses to cultural questions, emphasizing the need for culturally diverse development in AI for worldwide acceptance and ethical use. For more details or to contribute to this research, visit our GitHub page https://github.com/reemim/Hofstedes_CAT

Learning with Noisy Labels (LNL) reduces reliance on high-quality labeled data but often overlooks open-set noise, where noisy samples belong to unknown classes, unlike closed-set noise within known categories.This paper advances LNL by reformulating the problem to incorporate open-set noise through a complete noise transition matrix, enabling a theoretical comparison of its impact on classification error rates against closed-set noise. Our analysis reveals that open-set noise induces smaller error increases, with distinct effects from 'hard' (semantically similar to inliers) and 'easy' (dissimilar) variants. We evaluate entropy-based detection, finding it effective only for easy open-set noise, and propose solutions leveraging vision-language models and self-supervised learning to address hard noise challenges. For empirical validation, we introduce CIFAR100-O, ImageNet-O, and a WebVision open-set test set, enabling robust benchmarking of LNL methods under open-set noise conditions. Recognizing classification accuracy's limitations in capturing model robustness, we advocate out-of-distribution (OOD) detection as a complementary metric. Our theoretical and empirical results highlight the unique challenges of open-set noise, offering new tools and evaluation frameworks to enhance LNL robustness in real-world scenarios.

Multimodal machine learning with missing modalities is an increasingly relevant challenge arising in various applications such as healthcare. This paper extends the current research into missing modalities to the low-data regime, i.e., a downstream task has both missing modalities and limited sample size issues. This problem setting is particularly challenging and also practical as it is often expensive to get full-modality data and sufficient annotated training samples. We propose to use retrieval-augmented in-context learning to address these two crucial issues by unleashing the potential of a transformer’s in-context learning ability. Diverging from existing methods, which primarily belong to the parametric paradigm and often require sufficient training samples, our work exploits the value of the available full-modality data, offering a novel perspective on resolving the challenge. The proposed data-dependent framework exhibits a higher degree of sample efficiency and is empirically demonstrated to enhance the classification model’s performance on both full- and missing-modality data in the low-data regime across various multimodal learning tasks. When only 1% of the training data are available, our proposed ICL-CA method outperforms the best baseline by 5.9%, 5.9%, 5.3% and 10.8% on four datasets across various missing states. Notably, our method also reduces the performance gap between full-modality and missing-modality data compared with the baseline. Code is available 1 1 GitHub repository . . • We address data scarcity in missing-modality tasks, highlighting the limitations of existing parametric approaches, where adequate sample sizes are essential. Our study reveals that focusing adaptively on both missing- and full-modality data improves performance. • We introduce a novel, data-dependent in-context learning method to enhance sample efficiency and optimize learning across missing- and full-modality data. • Experiments on four datasets (medical and vision-language tasks) show that our method boosts performance by 5.9%, 5.9%, 5.3% and 10.8% over the best baseline in the low-data regime.

High-resolution simulations, such as the ICOsahedral Non-hydrostatic Large-Eddy Model (ICON-LEM), provide valuable insights into the complex interactions among aerosols, clouds, and precipitation, which are the major contributors to climate change uncertainty. However, due to their exorbitant computational costs, they can only be employed for a limited period and geographical area. To address this, we propose a more cost-effective method powered by an emerging machine learning approach to better understand the intricate dynamics of the climate system. Our approach involves active learning techniques by leveraging high-resolution climate simulation as an oracle that is queried based on an abundant amount of unlabeled data drawn from satellite observations. In particular, we aim to predict autoconversion rates, a crucial step in precipitation formation, while significantly reducing the need for a large number of labeled instances. In this study, we present novel methods: custom fusion query strategies for labeling instances – weight fusion (WiFi) and merge fusion (MeFi) – along with active feature selection based on SHAP. These methods are designed to tackle real-world challenges – in this case, climate change, with a specific focus on the prediction of autoconversion rates – due to their simplicity and practicality in application.

Various approaches have been developed to upper bound the generalization error of a supervised learning algorithm. However, existing bounds are often loose and even vacuous when evaluated in practice. As a result, they may fail to characterize the exact generalization ability of a learning algorithm. Our main contributions are exact characterizations of the expected generalization error of the well-known Gibbs algorithm (a.k.a. Gibbs posterior) using different information measures, in particular, the symmetrized KL information between the input training samples and the output hypothesis. Our result can be applied to tighten existing expected generalization errors and PAC-Bayesian bounds. Our information-theoretic approach is versatile, as it also characterizes the generalization error of the Gibbs algorithm with a data-dependent regularizer and that of the Gibbs algorithm in the asymptotic regime, where it converges to the standard empirical risk minimization algorithm. Of particular relevance, our results highlight the role the symmetrized KL information plays in controlling the generalization error of the Gibbs algorithm.

With the rise of machine learning, hyperspectral image (HSI) unmixing problems have been tackled using learning-based methods. However, physically meaningful unmixing results are not guaranteed without proper guidance. In this work, we propose an unsupervised framework inspired by deep image prior (DIP) that can be used for both linear and nonlinear blind unmixing models. The framework consists of three modules: 1) an Endmember estimation module using DIP (EDIP); 2) an Abundance estimation module using DIP (ADIP); and 3) a mixing module (MM). EDIP and ADIP modules generate endmembers and abundances, respectively, while MM produces a reconstruction of the HSI observations based on the postulated unmixing model. We introduce a composite loss function that applies to both linear and nonlinear unmixing models to generate meaningful unmixing results. In addition, we propose an adaptive loss weight strategy for better unmixing results in nonlinear mixing scenarios. The proposed methods outperform state-of-the-art unmixing algorithms in extensive experiments conducted on both synthetic and real datasets.

We provide an exact characterization of the expected generalization error (gen-error) for semi-supervised learning (SSL) with pseudo-labeling via the Gibbs algorithm. The gen-error is expressed in terms of the symmetrized KL information between the output hypothesis, the pseudo-labeled dataset, and the labeled dataset. Distribution-free upper and lower bounds on the gen-error can also be obtained. Our findings offer new insights that the generalization performance of SSL with pseudo-labeling is affected not only by the information between the output hypothesis and input training data but also by the information shared between the labeled and pseudo-labeled data samples. This serves as a guideline to choose an appropriate pseudo-labeling method from a given family of methods. To deepen our understanding, we further explore two examples—mean estimation and logistic regression. In particular, we analyze how the ratio of the number of unlabeled to labeled data $\lambda$ affects the gen-error under both scenarios. As $\lambda$ increases, the gen-error for mean estimation decreases and then saturates at a value larger than when all the samples are labeled, and the gap can be quantified exactly with our analysis, and is dependent on the cross-covariance between the labeled and pseudo-labeled data samples. For logistic regression, the gen-error and the variance component of the excess risk also decrease as $\lambda$ increases.

This research presents a new approach for blind single-image transparency separation, a significant challenge in image processing. The proposed framework divides the task into two parallel processes: feature separation and image reconstruction. The feature separation task leverages two deep image prior (DIP) networks to recover two distinct layers. An exclusion loss and deep feature separation loss are used to decompose features. For the image reconstruction task, we minimize the difference between the mixed image and the re-mixed image while also incorporating a regularizer to impose natural priors on each layer. Our results indicate that our method performs comparably or outperforms state-of-the-art approaches when tested on various image datasets.

We analyze the generalization ability of joint-training meta learning algorithms via the Gibbs algorithm. Our exact characterization of the expected meta generalization error for the meta Gibbs algorithm is based on symmetrized KL information, which measures the dependence between all meta-training datasets and the output parameters, including task-specific and meta parameters. Additionally, we derive an exact characterization of the meta generalization error for the super-task Gibbs algorithm, in terms of conditional symmetrized KL information within the super-sample and super-task framework introduced in [1] and [2], respectively. Our results also enable us to provide novel distribution-free generalization error upper bounds for these Gibbs algorithms applicable to meta learning.

Multiple Instance Learning (MIL) is a crucial weakly supervised learning method applied across various domains, e.g., medical diagnosis based on whole slide images (WSIs). Recent advancements in MIL algorithms have yielded exceptional performance when the training and test data originate from the same domain, such as WSIs obtained from the same hospital. However, this paper reveals a performance deterioration of MIL models when tested on an out-of-domain test set, exemplified by WSIs sourced from a novel hospital. To address this challenge, this paper introduces the Retrieval-AugMented MIL (RAM-MIL) framework, which integrates Optimal Transport (OT) as the distance metric for nearest neighbor retrieval. The development of RAM-MIL is driven by two key insights. First, a theoretical discovery indicates that reducing the input's intrinsic dimension can minimize the approximation error in attention-based MIL. Second, previous studies highlight a link between input intrinsic dimension and the feature merging process with the retrieved data. Empirical evaluations conducted on WSI classification demonstrate that the proposed RAM-MIL framework achieves state-of-the-art performance in both in-domain scenarios, where the training and retrieval data are in the same domain, and more crucially, in out-of-domain scenarios, where the (unlabeled) retrieval data originates from a different domain. Furthermore, the use of the transportation matrix derived from OT renders the retrieval results interpretable at the instance level, in contrast to the vanilla $l_2$ distance, and allows for visualization for human experts. *Code can be found at \url{https://github.com/ralphc1212/ram-mil*.

X-radiography (X-ray imaging) is a widely used imaging technique in art investigation. It can provide information about the condition of a painting as well as insights into an artist's techniques and working methods, often revealing hidden information invisible to the naked eye. X-radiograpy of double-sided paintings results in a mixed X-ray image and this paper deals with the problem of separating this mixed image. Using the visible color images (RGB images) from each side of the painting, we propose a new Neural Network architecture, based upon 'connected' auto-encoders, designed to separate the mixed X-ray image into two simulated X-ray images corresponding to each side. This connected auto-encoders architecture is such that the encoders are based on convolutional learned iterative shrinkage thresholding algorithms (CLISTA) designed using algorithm unrolling techniques, whereas the decoders consist of simple linear convolutional layers; the encoders extract sparse codes from the visible image of the front and rear paintings and mixed X-ray image, whereas the decoders reproduce both the original RGB images and the mixed X-ray image. The learning algorithm operates in a totally self-supervised fashion without requiring a sample set that contains both the mixed X-ray images and the separated ones. The methodology was tested on images from the double-sided wing panels of the Ghent Altarpiece, painted in 1432 by the brothers Hubert and Jan van Eyck. These tests show that the proposed approach outperforms other state-of-the-art X-ray image separation methods for art investigation applications.

A common assumption in semi-supervised learning is that the labeled, unlabeled, and test data are drawn from the same distribution. However, this assumption is not satisfied in many applications. In many scenarios, the data is collected sequentially (e.g., healthcare) and the distribution of the data may change over time often exhibiting so-called covariate shifts. In this paper, we propose an approach for semi-supervised learning algorithms that is capable of addressing this issue. Our framework also recovers some popular methods, including entropy minimization and pseudo-labeling. We provide new information-theoretical based generalization error upper bounds inspired by our novel framework. Our bounds are applicable to both general semi-supervised learning and the covariate-shift scenario. Finally, we show numerically that our method outperforms previous approaches proposed for semi-supervised learning under the covariate shift.

We provide an information-theoretic analysis of the generalization ability of Gibbs-based transfer learning algorithms by focusing on two popular empirical risk minimization (ERM) approaches for transfer learning, $\alpha$-weighted-ERM and two-stage-ERM. Our key result is an exact characterization of the generalization behavior using the conditional symmetrized Kullback-Leibler (KL) information between the output hypothesis and the target training samples given the source training samples. Our results can also be applied to provide novel distribution-free generalization error upper bounds on these two aforementioned Gibbs algorithms. Our approach is versatile, as it also characterizes the generalization errors and excess risks of these two Gibbs algorithms in the asymptotic regime, where they converge to the $\alpha$-weighted-ERM and two-stage-ERM, respectively. Based on our theoretical results, we show that the benefits of transfer learning can be viewed as a bias-variance trade-off, with the bias induced by the source distribution and the variance induced by the lack of target samples. We believe this viewpoint can guide the choice of transfer learning algorithms in practice.

Recently, unfolding techniques have been widely utilized to solve the inverse problems in various applications. In this paper, we study optimization guarantees for two popular unfolded networks, i.e., unfolded networks derived from iterative soft thresholding algorithms (ISTA) and derived from Alternating Direction Method of Multipliers (ADMM). Our guarantees–leveraging the Polyak-Lojasiewicz* (PL*) condition–state that the training (empirical) loss decreases to zero with the increase in the number of gradient descent epochs provided that the number of training samples is less than some threshold that depends on various quantities underlying the desired information processing task. Our guarantees also show that this threshold is larger for unfolded ISTA in comparison to unfolded ADMM, suggesting that there are certain regimes of number of training samples where the training error of unfolded ADMM does not converge to zero whereas the training error of unfolded ISTA does. A number of numerical results are provided backing up our theoretical findings.

In this paper, we propose a novel network structure to solve the blind hyperspectral unmixing problem using a double Deep Image Prior (DIP). In particular, the blind unmixing problem involves two sub-problems: endmember estimation and abundance estimation. We, therefore, propose two sub-networks, endmember estimation DIP (EDIP) and abundance estimation DIP (ADIP), to generate the estimation of endmembers and estimation of corresponding abundances respectively. The overall network is then constructed by assembling these two sub-networks. The network is trained in an end-to-end manner by minimizing a novel composite loss function. The experiments on synthetic and real datasets show the effectiveness of the proposed method over state-of-art unmixing methods.

Generalization error bounds are essential to understanding machine learning algorithms. This paper presents novel expected generalization error upper bounds based on the average joint distribution between the output hypothesis and each input training sample. Multiple generalization error upper bounds based on different information measures are provided, including Wasserstein distance, total variation distance, KL divergence, and Jensen-Shannon divergence. Due to the convexity of the information measures, the proposed bounds in terms of Wasserstein distance and total variation distance are shown to be tighter than their counterparts based on individual samples in the literature. An example is provided to demonstrate the tightness of the proposed generalization error bounds.

Neural networks (NNs) have demonstrated their potential in a variety of domains ranging from computer vision (CV) to natural language processing. Among various NNs, two-dimensional (2-D) and three-dimensional (3-D) convolutional NNs (CNNs) have been widely adopted for a broad spectrum of applications, such as image classification and video recognition, due to their excellent capabilities in extracting 2-D and 3-D features. However, standard 2-D and 3-D CNNs are not able to capture their model uncertainty which is crucial for many safety-critical applications, including healthcare and autonomous driving. In contrast, Bayesian CNNs (BayesCNNs), as a variant of CNNs, have demonstrated their ability to express uncertainty in their prediction via a mathematical grounding. Nevertheless, BayesCNNs have not been widely used in industrial practice due to their compute requirements stemming from sampling and subsequent forward passes through the whole network multiple times. As a result, these requirements significantly increase the amount of computation and memory consumption in comparison to standard CNNs. This article proposes a novel field-programmable gate array (FPGA)-based hardware architecture to accelerate both 2-D and 3-D BayesCNNs based on Monte Carlo dropout (MCD). Compared with other state-of-the-art accelerators for BayesCNNs, the proposed design can achieve up to four times higher energy efficiency and nine times better compute efficiency. An automatic framework capable of supporting partial Bayesian inference is proposed to explore the tradeoff between algorithm and hardware performance. Extensive experiments are conducted to demonstrate that our framework can effectively find the optimal implementations in the design space.

Recent research on joint source channel coding (JSCC) for wireless communications has achieved great success owing to the employment of deep learning (DL). However, the existing work on DL based JSCC usually trains the designed network to operate under a specific signal-to-noise ratio (SNR) regime, without taking into account that the SNR level during the deployment stage may differ from that during the training stage. A number of networks are required to cover the scenario with a broad range of SNRs, which is computational inefficiency (in the training stage) and requires large storage. To overcome these drawbacks our paper proposes a novel method called Attention DL based JSCC (ADJSCC) that can successfully operate with different SNR levels during transmission. This design is inspired by the resource assignment strategy in traditional JSCC, which dynamically adjusts the compression ratio in source coding and the channel coding rate according to the channel SNR. This is achieved by resorting to attention mechanisms because these are able to allocate computing resources to more critical tasks. Instead of applying the resource allocation strategy in traditional JSCC, the ADJSCC uses the channel-wise soft attention to scaling features according to SNR conditions. We compare the ADJSCC method with the state-of-the-art DL based JSCC method through extensive experiments to demonstrate its adaptability, robustness and versatility. Compared with the existing methods, the proposed method takes less storage and is more robust in the presence of channel mismatch.

Hyperspectral image (HSI) unmixing is an increasingly studied problem in various areas, including remote sensing. It has been tackled using both physical model-based approaches and more recently machine learning-based ones. In this article, we propose a new HSI unmixing algorithm combining both model- and learning-based techniques, based on algorithm unrolling approaches, delivering improved unmixing performance. Our approach unrolls the alternating direction method of multipliers (ADMMs) solver of a constrained sparse regression problem underlying a linear mixture model. We then propose a neural network structure for abundance estimation that can be trained using supervised learning techniques based on a new composite loss function. We also propose another neural network structure for blind unmixing that can be trained using unsupervised learning techniques. Our proposed networks are also shown to possess a lighter and richer structure containing less learnable parameters and more skip connections compared with other competing architectures. Extensive experiments show that the proposed methods can achieve much faster convergence and better performance even with a very small training dataset size when compared with other unmixing methods, such as model-inspired neural network for abundance estimation (MNN-AE), model-inspired neural network for blind unmixing (MNN-BU), unmixing using deep image prior (UnDIP), and endmember-guided unmixing network (EGU-Net).

In this paper, we focus on X-ray images (X-radiographs) of paintings with concealed sub-surface designs (e.g., deriving from reuse of the painting support or revision of a composition by the artist), which therefore include contributions from both the surface painting and the concealed features. In particular, we propose a self-supervised deep learning-based image separation approach that can be applied to the X-ray images from such paintings to separate them into two hypothetical X-ray images. One of these reconstructed images is related to the X-ray image of the concealed painting, while the second one contains only information related to the X-ray image of the visible painting. The proposed separation network consists of two components: the analysis and the synthesis sub-networks. The analysis sub-network is based on learned coupled iterative shrinkage thresholding algorithms (LCISTA) designed using algorithm unrolling techniques, and the synthesis sub-network consists of several linear mappings. The learning algorithm operates in a totally self-supervised fashion without requiring a sample set that contains both the mixed X-ray images and the separated ones. The proposed method is demonstrated on a real painting with concealed content, Do na Isabel de Porcel by Francisco de Goya, to show its effectiveness.

Bayesian neural networks (BayesNNs) have demonstrated their advantages in various safety-critical applications, such as autonomous driving or healthcare, due to their ability to capture and represent model uncertainty. However, standard BayesNNs require to be repeatedly run because of Monte Carlo sampling to quantify their uncertainty, which puts a burden on their real-world hardware performance. To address this performance issue, this article systematically exploits the extensive structured sparsity and redundant computation in BayesNNs. Different from the unstructured or structured sparsity in standard convolutional NNs, the structured sparsity of BayesNNs is introduced by Monte Carlo Dropout and its associated sampling required during uncertainty estimation and prediction, which can be exploited through both algorithmic and hardware optimizations. We first classify the observed sparsity patterns into three categories: channel sparsity, layer sparsity and sample sparsity. On the algorithmic side, a framework is proposed to automatically explore these three sparsity categories without sacrificing algorithmic performance. We demonstrated that structured sparsity can be exploited to accelerate CPU designs by up to 49 times, and GPU designs by up to 40 times. On the hardware side, a novel hardware architecture is proposed to accelerate BayesNNs, which achieves a high hardware performance using the runtime adaptable hardware engines and the intelligent skipping support. Upon implementing the proposed hardware design on an FPGA, our experiments demonstrated that the algorithm-optimized BayesNNs can achieve up to 56 times speedup when compared with unoptimized Bayesian nets. Comparing with the optimized GPU implementation, our FPGA design achieved up to 7.6 times speedup and up to 39.3 times higher energy efficiency.

Neural networks (NNs) have demonstrated their potential in a wide range of applications such as image recognition, decision making or recommendation systems. However, standard NNs are unable to capture their model uncertainty which is crucial for many safety-critical applications including healthcare and autonomous vehicles. In comparison, Bayesian neural networks (BNNs) are able to express uncertainty in their prediction via a mathematical grounding. Nevertheless, BNNs have not been as widely used in industrial practice, mainly because of their expensive computational cost and limited hardware performance. This work proposes a novel FPGA based hardware architecture to accelerate BNNs inferred through Monte Carlo Dropout. Compared with other state-of-the-art BNN accelerators, the proposed accelerator can achieve up to 4 times higher energy efficiency and 9 times better compute efficiency. Considering partial Bayesian inference, an automatic framework is proposed, which explores the trade-off between hardware and algorithmic performance. Extensive experiments are conducted to demonstrate that our proposed framework can effectively find the optimal points in the design space.

Neural networks have demonstrated their outstanding performance in a wide range of tasks. Specifically recurrent architectures based on long-short term memory (LSTM) cells have manifested excellent capability to model time dependencies in real-world data. However, standard recurrent architectures cannot estimate their uncertainty which is essential for safety-critical applications such as in medicine. In contrast, Bayesian recurrent neural networks (RNNs) are able to provide uncertainty estimation with improved accuracy. Nonetheless, Bayesian RNNs are computationally and memory demanding, which limits their practicality despite their advantages. To address this issue, we propose an FPGA-based hardware design to accelerate Bayesian LSTM-based RNNs. To further improve the overall algorithmic-hardware performance, a co-design framework is proposed to explore the most fitting algorithmic-hardware configurations for Bayesian RNNs. We conduct extensive experiments on healthcare applications to demonstrate the improvement of our design and the effectiveness of our framework. Compared with GPU implementation, our FPGA-based design can achieve up to 10 times speedup with nearly 106 times higher energy efficiency. To the best of our knowledge, this is the first work targeting acceleration of Bayesian RNNs on FPGAs.

The use of large-scale antenna arrays poses great difficulties in obtaining perfect channel state information (CSI) in multi-antenna communication systems, which is essential for precoding optimization. To tackle this issue, in this paper we propose a probabilistic-learning based approach (PLA), aiming at alleviating the requirement of perfect CSI. The rationale is that the existing precoding algorithms that output a single precoder are often overconfident in their abilities and the obtained CSI. To avoid overconfidence, we incorporate the idea of regularization in machine learning (ML) into precoding models, so as to limit representative abilities of the precoding models. Compared to the state-of-the-art robust precoding designs, an important advantage of PLA is that CSI uncertainty models are not required. As a specific application of PLA, we design an efficient robust symbol-level hybrid precoding algorithm for the millimeter wave system and confirm the effectiveness of PLA via simulations.

Linear inverse problems appear in many applications, where different algorithms are typically employed to solve each inverse problem. Nowadays, the rapid development of deep learning (DL) provides a fresh perspective for solving the linear inverse problem: a number of well-designed network architectures results in state-of-the-art performance in many applications. In this overview paper, we present the combination of the DL and the Plug-and-Play priors (PPP) framework, showcasing how it allows solving various inverse problems by leveraging the impressive capabilities of existing DL based denoising algorithms. Open challenges and potential future directions along this line of research are also discussed.

We consider compressive sensing problems with model mismatch where one wishes to recover a sparse high-dimensional vector from low-dimensional observations subject to uncertainty in the measurement operator. In particular, we design a new robust deep neural network architecture by applying algorithm unfolding techniques to a robust version of the underlying recovery problem. Our proposed network –named Robust lErned Shrinkage-Thresholding (REST) –exhibits additional features including enlarged number of parameters and normalization processing compared to state-of-the-art deep architecture Learned Iterative Shrinkage-Thresholding Algorithm (LISTA), leading to the reliable recovery of the signal under sample-wise varying model mismatch. Our proposed network is also shown to outperform LISTA in compressive sensing problems under sample-wise varying model mismatch.

In this paper, we use algorithm unrolling approaches in order to design a new neural network structure applicable to hyperspectral unmixing challenges. In particular, building upon a constrained sparse regression formulation of the underlying unmixing problem, we unroll an ADMM solver onto a neural network architecture that can be used to deliver the abundances of different (known) endmembers given a reflectance spectrum. Our proposed network – which can be readily trained using standard supervised learning procedures – is shown to possess a richer structure consisting of various skip connections and shortcuts than other competing architectures. Moreover, our proposed network also delivers state-of-the-art unmixing performance compared to competing methods.

Much of the work in the field of group fairness addresses disparities between predefined groups based on protected features such as gender, age, and race, which need to be available at train, and often also at test, time. These approaches are static and retrospective, since algorithms designed to protect groups identified a priori cannot anticipate and protect the needs of different at-risk groups in the future. In this work we analyze the space of solutions for worst-case fairness beyond demographics, and propose Blind Pareto Fairness (BPF), a method that leverages no-regret dynamics to recover a fair minimax classifier that reduces worst-case risk of any potential subgroup of sufficient size, and guarantees that the remaining population receives the best possible level of service. BPF addresses fairness beyond demographics, that is, it does not rely on predefined notions of at-risk groups, neither at train nor at test time. Our experimental results show that the proposed framework improves worst-case risk in multiple standard datasets, while simultaneously providing better levels of service for the remaining population. The code is available at github.com/natalialmg/BlindParetoFairness

Generalization error bounds are critical to understanding the performance of machine learning models. In this work, building upon a new bound of the expected value of an arbitrary function of the population and empirical risk of a learning algorithm, we offer a more refined analysis of the generalization behaviour of a machine learning models based on a characterization of (bounds) to their generalization error moments. We discuss how the proposed bounds - which also encompass new bounds to the expected generalization error - relate to existing bounds in the literature. We also discuss how the proposed generalization error moment bounds can be used to construct new generalization error high-probability bounds.

Various approaches have been developed to upper bound the generalization error of a supervised learning algorithm. However, existing bounds are often loose and lack of guarantees. As a result, they may fail to characterize the exact generalization ability of a learning algorithm.Our main contribution is an exact characterization of the expected generalization error of the well-known Gibbs algorithm (a.k.a. Gibbs posterior) using symmetrized KL information between the input training samples and the output hypothesis. Our result can be applied to tighten existing expected generalization error and PAC-Bayesian bounds. Our approach is versatile, as it also characterizes the generalization error of the Gibbs algorithm with data-dependent regularizer and that of the Gibbs algorithm in the asymptotic regime, where it converges to the empirical risk minimization algorithm. Of particular relevance, our results highlight the role the symmetrized KL information plays in controlling the generalization error of the Gibbs algorithm.

Bayesian neural networks (BNNs) are making significant progress in many research areas where decision-making needs to be accompanied by uncertainty estimation. Being able to quantify uncertainty while making decisions is essential for understanding when the model is over-/under-confident, and hence BNNs are attracting interest in safety-critical applications, such as autonomous driving, healthcare, and robotics. Nevertheless, BNNs have not been as widely used in industrial practice, mainly because of their increased memory and compute costs. In this work, we investigate quantisation of BNNs by compressing 32-bit floating-point weights and activations to their integer counterparts, that has already been successful in reducing the compute demand in standard pointwise neural networks. We study three types of quantised BNNs, we evaluate them under a wide range of different settings, and we empirically demonstrate that a uniform quantisation scheme applied to BNNs does not substantially decrease their quality of uncertainty estimation.

X-radiography is a widely used imaging technique in art investigation, whether to investigate the condition of a painting or provide insights into artists' techniques and working methods. In this paper, we propose a new architecture based on the use of `connected' auto-encoders in order to separate mixed X-ray images acquired from double-sided paintings, where in addition to the mixed X-ray image one can also exploit the two RGB images associated with the front and back of the painting. This proposed architecture uses convolutional auto-encoders that extract features from the RGB images that can be employed to (1) reproduce both of the original RGB images, (2) reconstruct the associated separated X-ray images, and (3) regenerate the mixed X-ray image. It operates in a totally self-supervised fashion without the need for examples containing both the mixed X-ray images and the separated ones. Based on images from the double-sided wing panels from the famous Ghent Altarpiece, painted in 1432 by the brothers Hubert and Jan Van Eyck, the proposed algorithm has been experimentally verified to outperform state-of-the-art X-ray separation methods in art investigation applications.

Generalization error bounds are critical to understanding the performance of machine learning models. In this work, we propose a new information-theoretic based generalization error upper bound applicable to supervised learning scenarios. We show that our general bound can specialize in various previous bounds. We also show that our general bound can be specialized under some conditions to a new bound involving the Jensen-Shannon information between a random variable modelling the set of training samples and another random variable modelling the hypothesis. We also prove that our bound can be tighter than mutual information-based bounds under some conditions.

Depth image super-resolution is a challenging problem, since normally high upscaling factors are required (e.g., 16×), and depth images are often noisy. In order to achieve large upscaling factors and resilience to noise, we propose a Robust Algorithm for Depth imAge super Resolution (RADAR) that combines the power of finite rate of innovation (FRI) theory with multimodal dictionary learning. Given a low-resolution (LR) depth image, we first model its rows and columns as piece-wise polynomials and propose an FRI-based depth upscaling (FDU) algorithm to super-resolve the image. Then, the upscaled moderate quality (MQ) depth image is further enhanced with the guidance of a registered high-resolution (HR) intensity image. This is achieved by learning multimodal mappings from the joint MQ depth and HR intensity pairs to the HR depth, through a recently proposed triple dictionary learning (TDL) algorithm. Moreover, to speed up the super-resolution process, we introduce a new projection-based rapid upscaling (PRU) technique that pre-calculates the projections from the joint MQ depth and HR intensity pairs to the HR depth. Compared with the state-of-the-art deep learning-based methods, our approach has two distinct advantages: we need a fraction of training data but can achieve the best performance, and we are resilient to mismatches between training and testing datasets. The extensive numerical results show that the proposed method outperforms other state-of-the-art methods on either noise-free or noisy datasets with large upscaling factors up to 16× and can handle unknown blurring kernels well.

Magnetic resonance (MR) imaging tasks often involve multiple contrasts, such as T1-weighted, T2-weighted and fluid-attenuated inversion recovery (FLAIR) data. These contrasts capture information associated with the same underlying anatomy and thus exhibit similarities in either structure level or gray level. In this paper, we propose a coupled dictionary learning based multi-contrast MRI reconstruction (CDLMRI) approach to leverage the dependency correlation between different contrasts for guided or joint reconstruction from their under-sampled k -space data. Our approach iterates between three stages: coupled dictionary learning, coupled sparse denoising, and enforcing k -space consistency. The first stage learns a set of dictionaries that not only are adaptive to the contrasts, but also capture correlations among multiple contrasts in a sparse transform domain. By capitalizing on the learned dictionaries, the second stage performs coupled sparse coding to remove the aliasing and noise in the corrupted contrasts. The third stage enforces consistency between the denoised contrasts and the measurements in the k -space domain. Numerical experiments, consisting of retrospective under-sampling of various MRI contrasts with a variety of sampling schemes, demonstrate that CDLMRI is capable of capturing structural dependencies between different contrasts. The learned priors indicate notable advantages in multi-contrast MR imaging and promising applications in quantitative MR imaging such as MR fingerprinting.

Massive multiple-input multiple-output (MIMO) systems are the focus of increasing research attention. In such setups, there is an urgent need to utilize simple low-resolution quantizers, due to power and memory constraints. In this work we study massive MIMO channel estimation with quantized measurements, when the quantization system is designed to minimize the channel estimation error, as opposed to the quantization distortion. We first consider vector quantization, and characterize the minimal error achievable. Next, we focus on practical systems utilizing scalar uniform quantizers, and design the analog and digital processing as well as the quantization dynamic range to optimize the channel estimation accuracy. Our results demonstrate that the resulting massive MIMO system which utilizes low-resolution scalar quantizers can approach the minimal estimation error dictated by rate-distortion theory, achievable using vector quantizers.

Conventional dictionary matching based MR Fingerprinting (MRF) reconstruction approaches suffer from time-consuming operations that map temporal MRF signals to quantitative tissue parameters. In this paper, we design a 1-D residual convolutional neural network to perform the signature-to-parameter mapping in order to improve inference speed and accuracy. In particular, a 1-D convolutional neural network with shortcuts, a.k.a skip connections, for residual learning is developed using a TensorFlow platform. To avoid the requirement for a large amount of MRF data, the designed network is trained on synthesized MRF data simulated with the Bloch equations and fast imaging with steady state precession (FISP) sequences. The proposed approach was validated on both synthetic data and phantom data generated from a healthy subject. The reconstruction performance demonstrates a significantly improved speed - only 1.6s for reconstructing a pair of T1/T2 maps of size 128 × 128 - 50× faster than the original dictionary matching based method. The better performance was also confirmed by improved signal to noise ratio (SNR) and reduced root mean square error (RMSE). Furthermore, it is more compact to store a network instead of a large dictionary.

Data collection and sharing are pervasive aspects of modern society. This process can either be voluntary, as in the case of a person taking a facial image to unlock his/her phone, or incidental, such as traffic cameras collecting videos on pedestrians. An undesirable side effect of these processes is that shared data can carry information about attributes that users might consider as sensitive, even when such information is of limited use for the task. It is therefore desirable for both data collectors and users to design procedures that minimize sensitive information leakage. Balancing the competing objectives of providing meaningful individualized service levels and inference while obfuscating sensitive information is still an open problem. In this work, we take an information theoretic approach that is implemented as an unconstrained adversarial game between Deep Neural Networks in a principled, data-driven manner. This approach enables us to learn domain-preserving stochastic transformations that maintain performance on existing algorithms while minimizing sensitive information leakage.

In this paper, we propose a new multimodal image denoising approach to attenuate white Gaussian additive noise in a given image modality under the aid of a guidance image modality. The proposed coupled image denoising approach consists of two stages: coupled sparse coding and reconstruction. The first stage performs joint sparse transform for multimodal images with respect to a group of learned coupled dictionaries, followed by a shrinkage operation on the sparse representations. Then, in the second stage, the shrunken representations, together with coupled dictionaries, contribute to the reconstruction of the denoised image via an inverse transform. The proposed denoising scheme demonstrates the capability to capture both the common and distinct features of different data modalities. This capability makes our approach more robust to inconsistencies between the guidance and the target images, thereby overcoming drawbacks such as the texture copying artifacts. Experiments on real multimodal images demonstrate that the proposed approach is able to better employ guidance information to bring notable benefits in the image denoising task with respect to the state-of-the-art.

Medical imaging tasks often involve multiple contrasts, such as T1-and T2-weighted magnetic resonance imaging (MRI) data. These contrasts capture information associated with the same underlying anatomy and thus exhibit similarities. In this paper, we propose a Coupled Dictionary Learning based multi-contrast MRI reconstruction (CDLMRI) approach to leverage an available guidance contrast to restore the target contrast. Our approach consists of three stages: coupled dictionary learning, coupled sparse denoising, and k-space consistency enforcing. The first stage learns a group of dictionaries that capture correlations among multiple contrasts. By capitalizing on the learned adaptive dictionaries, the second stage performs joint sparse coding to denoise the corrupted target image with the aid of a guidance contrast. The third stage enforces consistency between the denoised image and the measurements in the k-space domain. Numerical experiments on the retrospective under-sampling of clinical MR images demonstrate that incorporating additional guidance contrast via our design improves MRI reconstruction, compared to state-of-the-art approaches.

Large-scale wireless sensor networks (WSNs) and Internet-of-Things (IoT) applications involve diverse sensing devices collecting and transmitting massive amounts of heterogeneous data. In this paper, we propose a novel compressive data aggregation and recovery mechanism that reduces the global communication cost without introducing computational overhead at the network nodes. Following the principles of compressive demixing, each node of the network collects measurement readings from multiple sources and mixes them with readings from other nodes into a single low-dimensional measurement vector, which is then relayed to other nodes; the constituent signals are recovered at the sink using convex optimization. Our design achieves significant reduction in the overall network data rates compared to prior schemes based on (distributed) compressed sensing or compressed sensing with (multiple) side information. Experiments using real large-scale air-quality data demonstrate the superior performance of the proposed framework against state-of-the-art solutions, with and without the presence of measurement and transmission noise.

This paper studies the generalization error of invariant classifiers. In particular, we consider the common scenario where the classification task is invariant to certain transformations of the input, and that the classifier is constructed (or learned) to be invariant to these transformations. Our approach relies on factoring the input space into a product of a base space and a set of transformations. We show that whereas the generalization error of a non-invariant classifier is proportional to the complexity of the input space, the generalization error of an invariant classifier is proportional to the complexity of the base space. We also derive a set of sufficient conditions on the geometry of the base space and the set of transformations that ensure that the complexity of the base space is much smaller than the complexity of the input space. Our analysis applies to general classifiers such as convolutional neural networks. We demonstrate the implications of the developed theory for such classifiers with experiments on the MNIST and CIFAR-10 datasets.

We consider problems where one wishes to represent a parameter associated with a signal source - subject to a certain rate and distortion - based on the observation of a number of realizations of the source signal. By reducing these indirect vector quantization problems to a standard vector quantization one, we provide a bound to the fundamental interplay between the rate and distortion in the large-rate setting. We specialize this characterization to two particular quantization scenarios: i) the representation of the mean of a multivariate Gaussian source; and ii) the representation of the eigen-spectrum of a multivariate Gaussian source. Numerical results compare our quantization approach to an approach where one recovers the parameters from the representation of the source signals itself: in addition to revealing that the characterization is sharp in the large-rate setting, the results also show that our approach offers considerable gains.

An information-theoretic projection design framework is proposed, of interest for feature design and compressive measurements. Both Gaussian and Poisson measurement models are considered. The gradient of a proposed information-theoretic metric (ITM) is derived, and a gradient-descent algorithm is applied in design; connections are made to the information bottleneck. The fundamental solution structure of such design is revealed in the case of a Gaussian measurement model and arbitrary input statistics. This new theoretical result reveals how ITM parameter settings impact the number of needed projection measurements, with this verified experimentally. The ITM achieves promising results on real data, for both signal recovery and classification.

Large-scale data collection by means of wireless sensor network and Internet-of-Things technology poses various challenges in view of the limitations in transmission, computation, and energy resources of the associated wireless devices. Compressive data gathering based on compressed sensing has been proven a well-suited solution to the problem. Existing designs exploit the spatiotemporal correlations among data collected by a specific sensing modality. However, many applications, such as environmental monitoring, involve collecting heterogeneous data that are intrinsically correlated. In this paper, we propose to leverage the correlation from multiple heterogeneous signals when recovering the data from compressive measurements. To this end, we propose a novel recovery algorithm-built upon belief-propagation principles-that leverages correlated information from multiple heterogeneous signals. To efficiently capture the statistical dependencies among diverse sensor data, the proposed algorithm uses the statistical model of copula functions. Experiments with heterogeneous air-pollution sensor measurements show that the proposed design provides significant performance improvements against the state-of-the-art compressive data gathering and recovery schemes that use classical compressed sensing, compressed sensing with side information, and distributed compressed sensing.

In support of art investigation, we propose a new source separation method that unmixes a single X-ray scan acquired from double-sided paintings. In this problem, the X-ray signals to be separated have similar morphological characteristics, which brings previous source separation methods to their limits. Our solution is to use photographs taken from the front-and back-side of the panel to drive the separation process. The crux of our approach relies on the coupling of the two imaging modalities (photographs and X-rays) using a novel coupled dictionary learning framework able to capture both common and disparate features across the modalities using parsimonious representations; the common component captures features shared by the multi-modal images, whereas the innovation component captures modality-specific information. As such, our model enables the formulation of appropriately regularized convex optimization procedures that lead to the accurate separation of the X-rays. Our dictionary learning framework can be tailored both to a single- and a multi-scale framework, with the latter leading to a significant performance improvement. Moreover, to improve further on the visual quality of the separated images, we propose to train coupled dictionaries that ignore certain parts of the painting corresponding to craquelure. Experimentation on synthetic and real data - taken from digital acquisition of the Ghent Altarpiece (1432) - confirms the superiority of our method against the state-of-the-art morphological component analysis technique that uses either fixed or trained dictionaries to perform image separation.

We address the problem of compressed sensing (CS) with prior information: reconstruct a target CS signal with the aid of a similar signal that is known beforehand, our prior information. We integrate the additional knowledge of the similar signal into CS via l1-l1and l1-l2minimization. We then establish bounds on the number of measurements required by these problems to successfully reconstruct the original signal. Our bounds and geometrical interpretations reveal that if the prior information has good enough quality, l1-l1minimization improves the performance of CS dramatically. In contrast, l1-l2minimization has a performance very similar to classical CS, and brings no significant benefits. In addition, we use the insight provided by our bounds to design practical schemes to improve prior information. All our findings are illustrated with experimental results.

The classical compressed sensing (CS) paradigm can be modified so as to leverage a signal correlated to the signal of interest, called side information, which is assumed to be provided a priori at the decoder in order to aid reconstruction. In this work, we propose a novel CS reconstruction method based on belief propagation principles, which manages to exploit side information generated from a diverse (or heterogeneous) data source by using the statistical model of copula functions. Through simulations, we demonstrate that the proposed method yields significant reduction in the mean-squared error of the reconstructed signal as compared to state-of-the-art methods in classical compressed sensing and compressed sensing with side information.

This paper investigates the impact of projection design on the reconstruction of high-dimensional signals from low-dimensional measurements in the presence of side information. In particular, we assume that both the signal of interest and the side information are described by a joint Gaussian mixture model (GMM) distribution. Sharp necessary and sufficient conditions on the number of measurements needed to guarantee that the average reconstruction error approaches zero in the low-noise regime are derived, for both cases when the side information is available at the decoder or at the decoder and encoder. Numerical results are also presented to showcase the impact of projection design on applications with real imaging data in the presence of side information.

We address the problem of reference-based compressed sensing: reconstruct a sparse signal from few linear measurements using as prior information a reference signal, a signal similar to the signal we want to reconstruct. Access to reference signals arises in applications such as medical imaging, e.g., through prior images of the same patient, and compressive video, where previously reconstructed frames can be used as reference. Our goal is to use the reference signal to reduce the number of required measurements for reconstruction. We achieve this via a reweighted ℓ1-ℓ1minimization scheme that updates its weights based on a sample complexity bound. The scheme is simple, intuitive and, as our experiments show, outperforms prior algorithms, including reweighted ℓ1minimization, ℓ1-ℓ1minimization, and modified CS.

We develop a general framework for compressive linear-projection measurements with side information. Side information is an additional signal correlated with the signal of interest. We investigate the impact of side information on classification and signal recovery from low-dimensional measurements. Motivated by real applications, two special cases of the general model are studied. In the first, a joint Gaussian mixture model is manifested on the signal and side information. The second example again employs a Gaussian mixture model for the signal, with side information drawn from a mixture in the exponential family. Theoretical results on recovery and classification accuracy are derived. The presence of side information is shown to yield improved performance, both theoretically and experimentally.

In support of art investigation, we propose a new source separation method that unmixes a single X-ray scan acquired from double-sided paintings. Unlike prior source separation methods, which are based on statistical or structural incoherence of the sources, we use visual images taken from the front- and back-side of the panel to drive the separation process. The coupling of the two imaging modalities is achieved via a new multi-scale dictionary learning method. Experimental results demonstrate that our method succeeds in the discrimination of the sources, while state-of-the-art methods fail to do so.

In this paper, we study the problem of projection kernel design for the reconstruction of high-dimensional signals from low-dimensional measurements in the presence of side information, assuming that the signal of interest and the side information signal are described by a joint Gaussian mixture model (GMM). In particular, we consider the case where the projection kernel for the signal of interest is random, whereas the projection kernel associated to the side information is designed. We then derive sufficient conditions on the number of measurements needed to guarantee that the minimum mean-squared error (MMSE) tends to zero in the low-noise regime. Our results demonstrate that the use of a designed kernel to capture side information can lead to substantial gains in relation to a random one, in terms of the number of linear projections required for reliable reconstruction.

This paper offers a characterization of fundamental limits on the classification and reconstruction of high-dimensional signals from low-dimensional features, in the presence of side information. We consider a scenario where a decoder has access both to linear features of the signal of interest and to linear features of the side information signal; while the side information may be in a compressed form, the objective is recovery or classification of the primary signal, not the side information. The signal of interest and the side information are each assumed to have (distinct) latent discrete labels; conditioned on these two labels, the signal of interest and side information are drawn from a multivariate Gaussian distribution that correlates the two. With joint probabilities on the latent labels, the overall signal-(side information) representation is defined by a Gaussian mixture model. By considering bounds to the misclassification probability associated with the recovery of the underlying signal label, and bounds to the reconstruction error associated with the recovery of the signal of interest itself, we then provide sharp sufficient and/or necessary conditions for these quantities to approach zero when the covariance matrices of the Gaussians are nearly low rank. These conditions, which are reminiscent of the well-known Slepian-Wolf and Wyner-Ziv conditions, are the function of the number of linear features extracted from signal of interest, the number of linear features extracted from the side information signal, and the geometry of these signals and their interplay. Moreover, on assuming that the signal of interest and the side information obey such an approximately low-rank model, we derive the expansions of the reconstruction error as a function of the deviation from an exactly low-rank model; such expansions also allow the identification of operational regimes, where the impact of side information on signal reconstruction is most relevant. Our framework, which offers a principled mechanism to integrate side information in high-dimensional data problems, is also tested in the context of imaging applications. In particular, we report state-of-theart results in compressive hyperspectral imaging applications, where the accompanying side information is a conventional digital photograph.

High dimensional data is modeled using low-rank subspaces, and the probability of misclassification is expressed in terms of the principal angles between subspaces. The form taken by this expression motivates the design of a new feature extraction method that enlarges inter-class separation, while preserving intra-class structure. The method can be tuned to emphasize different features shared by members within the same class. Classification performance is compared to that of state-of-the-art methods on synthetic data and on the real face database. The probability of misclassification is decreased when intra-class structure is taken into account.

We propose a recursive algorithm for estimating time-varying signals from a few linear measurements. The signals are assumed sparse, with unknown support, and are described by a dynamical model. In each iteration, the algorithm solves an ℓ1-ℓ1minimization problem and estimates the number of measurements that it has to take at the next iteration. These estimates are computed based on recent theoretical results for ℓ1-ℓ1minimization. We also provide sufficient conditions for perfect signal reconstruction at each time instant as a function of an algorithm parameter. The algorithm exhibits high performance in compressive tracking on a real video sequence, as shown in our experimental results.

This paper offers a characterization of performance limits for classification and reconstruction of high-dimensional signals from noisy compressive measurements, in the presence of side information. We assume the signal of interest and the side information signal are drawn from a correlated mixture of distributions/components, where each component associated with a specific class label follows a Gaussian mixture model (GMM). We provide sharp sufficient and/or necessary conditions for the phase transition of the misclassification probability and the reconstruction error in the low-noise regime. These conditions, which are reminiscent of the well-known Slepian-Wolf and Wyner-Ziv conditions, are a function of the number of measurements taken from the signal of interest, the number of measurements taken from the side information signal, and the geometry of these signals and their interplay.

This paper studies the performance associated with the classification of linear subspaces corrupted by noise with a mismatched classifier. In particular, we consider a problem where the classifier observes a noisy signal, the signal distribution conditioned on the signal class is zero-mean Gaussian with low-rank covariance matrix, and the classifier knows only the mismatched parameters in lieu of the true parameters. We derive an upper bound to the misclassification probability of the mismatched classifier and characterize its behaviour. Specifically, our characterization leads to sharp sufficient conditions that describe the absence of an error floor in the low-noise regime, and that can be expressed in terms of the principal angles and the overlap between the true and the mismatched signal subspaces.

Classical compressive sensing typically assumes a single measurement, and theoretical analysis often relies on corresponding concentration-of-measure results. There are many real-world applications involving multiple compressive measurements, from which the underlying signals may be estimated. In this paper, we establish a new concentration-of-measure inequality for a block-diagonal structured random compressive sensing matrix with Rademacher-ensembles. We discuss applications of this newly-derived inequality to two appealing compressive multiple-measurement models: for Gaussian and Poisson systems. In particular, Johnson-Lindenstrauss-type results and a compressed-domain classification result are derived for a Gaussian multiple-measurement model. We also propose, as another contribution, theoretical performance guarantees for signal recovery for multi-measurement Poisson systems, via the inequality.

The measurement matrix employed in compressive sensing typically cannot be known precisely a priori and must be estimated via calibration. One may take multiple compressive measurements, from which the measurement matrix and underlying signals may be estimated jointly. This is of interest as well when the measurement matrix may change as a function of the details of what is measured. This problem has been considered recently for Gaussian measurement noise, and here we develop this idea with application to Poisson systems. A collaborative maximum likelihood algorithm and alternating proximal gradient algorithm are proposed, and associated theoretical performance guarantees are established based on newly derived concentration-of-measure results. A Bayesian model is then introduced, to improve flexibility and generality. Connections between the maximum likelihood methods and the Bayesian model are developed, and example results are presented for a real compressive X-ray imaging system.

Conventional dictionary learning frameworks attempt to find a set of atoms that promote both signal representation and signal sparsity fora class of signals. In distributed compressive sensing (DCS), in addition to intra-signal correlation, inter-signal correlation is also exploited in the joint signal reconstruction, which goes beyond the aim of the conventional dictionary learning framework. In this letter, we propose a new dictionary learning framework in order to improve signal reconstruction performance in DCS applications. By capitalizing on the sparse common component and innovations (SCCI) model, which captures both intra- and inter-signal correlation, the proposed method iteratively finds a dictionary design that promotes various goals: i) signal representation; ii) intra-signal correlation; and iii) inter-signal correlation. Simulation results showthat our dictionary design leads to an improved DCS reconstruction performance in comparison to other designs.

We derive fundamental limits on the reliable classification of linear and affine subspaces from noisy, linear features. Drawing an analogy between discrimination among subspaces and communication over vector wireless channels, we define two Shannon-inspired characterizations of asymptotic classifier performance. First, we define the classification capacity, which characterizes the necessary and sufficient conditions for vanishing misclassification probability as the signal dimension, the number of features, and the number of subspaces to be discriminated all approach infinity. Second, we define the diversity-discrimination tradeoff, which, by analogy with the diversity-multiplexing tradeoff of fading vector channels, characterizes relationships between the number of discernible subspaces and the misclassification probability as the feature noise power approaches zero. We derive upper and lower bounds on these quantities which are tight in many regimes. Numerical results, including a face recognition application, validate the results in practice.

Using Shannon theory, we derive fundamental, asymptotic limits on the classification of low-dimensional subspaces from compressive measurements. We identify a syntactic equivalence between the classification of subspaces and the communication of codewords over non-coherent, multiple-antenna channels, from which we derive sharp bounds on the number of classes that can be discriminated with low misclassification probability as a function of the signal dimensionality and the signal-to-noise ratio. While the bounds are asymptotic in the limit of high dimension, they provide intuition for classifier design at finite dimension. We validate this intuition via an application to face recognition.

In this paper, we consider the problem of latent sentiment detection in Online Social Networks such as Twitter. Modeling the underlying social network as an Ising prior, we demonstrate the effect that the underlying social network structure has on the performance of a trivial sentiment detector. In doing so, we introduce a novel communications-oriented framework for characterizing the probability of error and the associated error exponent, based on information theoretic analysis. We study the variation of the calculated error exponent for several stylized network topologies such as the complete network, the star network and the closed-chain network, and show the importance of the network structure in determining detection performance.

We investigate design of general nonlinear functions for mapping high-dimensional data into a lower-dimensional (compressive) space. The nonlinear measurements are assumed contaminated by additive Gaussian noise. Depending on the application, we are either interested in recovering the high-dimensional data from the nonlinear compressive measurements, or performing classification directly based on these measurements. The latter case corresponds to classification based on nonlinearly constituted and noisy features. The nonlinear measurement functions are designed based on constrained mutual-information optimization. New analytic results are developed for the gradient of mutual information in this setting, for arbitrary input-signal statistics. We make connections to kernel-based methods, such as the support vector machine. Encouraging results are presented on multiple datasets, for both signal recovery and classification. The nonlinear approach is shown to be particularly valuable in high-noise scenarios.

Repurposing tools and intuitions from Shannon theory, we derive fundamental limits on the reliable classification of high-dimensional signals from low-dimensional features. We focus on the classification of linear and affine subspaces and suppose the features to be noisy linear projections. Leveraging a syntactic equivalence of discrimination between subspaces and communications over vector wireless channels, we derive asymptotic bounds on classifier performance. First, we define the classification capacity, which characterizes necessary and sufficient relationships between the signal dimension, the number of features, and the number of classes to be discriminated, as all three quantities approach infinity. Second, we define the diversitydiscrimination tradeoff, which characterizes relationships between the number of classes and the misclassification probability as the signal-to-noise ratio approaches infinity. We derive inner and outer bounds on these measures, revealing precise relationships between signal dimension and classifier performance.

Traditionally, codes are designed for an error correcting system to combat noisy transmission channels and achieve reliable communication. These codes can be used in syndrome coding, but it is shown in this study that the best performance is achieved with codes specifically designed for syndrome coding. In the view of the security of the communication, the best codes are the codes, which have the highest value of an information secrecy metric, the equivocation rate, for a given code length and code rate and are well packed codes. A code design technique is described, which produces the best binary linear codes for the syndrome coding scheme. An efficient recursive method to determine the equivocation rate for the binary symmetric channel and any linear binary code is also presented. A large online database of best equivocation codes for the syndrome coding scheme has been produced using the code design technique with some examples presented in the study. The presented results show that the best equivocation codes produce a higher level of secrecy for the syndrome coding scheme than almost all best known error correcting codes. Interestingly, it is unveiled that some outstanding best known error correcting codes are also best equivocation codes.

We consider the characterization of the asymptotic behavior of the average minimum mean-squared error (MMSE) in scalar and vector fading coherent channels, where the receiver knows the exact fading channel state but the transmitter knows only the fading channel distribution, driven by a range of inputs. In particular, we construct expansions of the quantities that are asymptotic in the signal-to-noise ratio (snr) for coherent channels subject to Rayleigh, Ricean or Nakagami fading and driven by discrete inputs and continuous inputs. The construction of these expansions leverages the fact that the average MMSE can be seen as an ψ-transform with a kernel of monotonic argument: this offers the means to use a powerful asymptotic expansion of integrals technique-the Mellin transform method-that leads immediately to the expansions of the average MMSE and- via the I-MMSE relationship-to expansions of the average mutual information, in terms of the so called canonical MMSE of a standard additive white Gaussian noise (AWGN) channel. We conclude with applications of the results to the optimization of the constrained capacity of a bank of parallel independent coherent fading channels driven by arbitrary discrete inputs.

We investigate the high-SNR capacity of multiple-antenna fading coherent channels driven by inputs that are constrained to obey a certain equiprobable discrete distribution rather than the optimal capacity-achieving Gaussian one. In particular, we capitalize on the machinery of asymptotic analysis, most notably the asymptotic expansion of integrals, in order to construct high-SNR expansions to the average minimum-mean squared error (MMSE) and—via the relation between MMSE and mutual information—high-SNR expansions to the average mutual information, thereby providing immediately a high-SNR expansion to the so-called constrained capacity. We use the expansions to characterize the constrained capacity of various multiple-antenna fading coherent channels, including Rayleigh fading models, Ricean fading models, and antenna-correlated models. The analysis unveils in detail the impact of the number of transmit and receive antennas, transmit and receive antenna correlation, line-of-sight components, and the geometry of the signaling scheme on the reliable information transmission rate. We also use the expansions to optimize the constrained capacity of multiple-antenna fading coherent channels, as an example, we focus on the derivation of design criteria for space-time signaling schemes.

A generalization of Bregman divergence is developed and utilized to unify vector Poisson and Gaussian channel models, from the perspective of the gradient of mutual information. The gradient is with respect to the measurement matrix in a compressive-sensing setting, and mutual information is considered for signal recovery and classification. Existing gradient-of-mutual-information results for scalar Poisson models are recovered as special cases, as are known results for the vector Gaussian model. The Bregman-divergence generalization yields a Bregman matrix, and this matrix induces numerous matrix-valued metrics. The metrics associated with the Bregman matrix are detailed, as are its other properties. The Bregman matrix is also utilized to connect the relative entropy and mismatched minimum mean squared error. Two applications are considered: 1) compressive sensing with a Poisson measurement model and 2) compressive topic modeling for analysis of a document corpora (word-count data). In both of these settings, we use the developed theory to optimize the compressive measurement matrix, for signal recovery and classification.

This paper advocates the use of the emerging distributed compressive sensing (DCS) paradigm in order to deploy energy harvesting (EH) wireless sensor networks (WSN) with practical network lifetime and data gathering rates that are substantially higher than the state-of-the-art. In particular, we argue that there are two fundamental mechanisms in an EH WSN: i) the energy diversity associated with the EH process that entails that the harvested energy can vary from sensor node to sensor node, and ii) the sensing diversity associated with the DCS process that entails that the energy consumption can also vary across the sensor nodes without compromising data recovery. We also argue that such mechanisms offer the means to match closely the energy demand to the energy supply in order to unlock the possibility for energy-neutral WSNs that leverage EH capability. A number of analytic and simulation results are presented in order to illustrate the potential of the approach.

We consider the problem of linear projection design for incoherent optical imaging systems. We propose a computationally efficient method to obtain effective measurement kernels that satisfy the physical constraints imposed by an optical system, starting first from arbitrary kernels, including those that satisfy a less demanding power constraint. Performance is measured in terms of mutual information between the source input and the projection measurement, as well as reconstruction error for real world images. A clear improvement in the quality of image reconstructions is shown with respect to both random and adaptive projection designs in the literature.

This paper investigates power allocation strategies over a bank of independent parallel Gaussian wiretap channels where a legitimate transmitter and a legitimate receiver communicate in the presence of an eavesdropper and a friendly jammer. We give algorithms to compute the optimal power allocation strategy of the jammer in the degraded scenario. We also give an algorithm to compute power allocations strategies of the jammer in a general scenario, leading to significant performance gains in relation to isotropic jamming. Additionally, we provide a set of results that cast further insight into the problem. In our scenario, which is applicable to current OFDM communications systems, we demonstrate that the proposed jammer power allocation strategy can lead to considerable secrecy gains.

The adequate operation of IPTV distribution networks heavily relies on the effective maintenance and management of their underlay DSL infrastructure. New hardware and software is required in order to improve monitoring capabilities and to directly diagnose anomalies that other segments of the DSL network cannot identify. In this work we initially compare the accuracy performance of SVM-specific formulations for constructing a robust ground truth within our classification procedure regarding abnormalities issued at anomaly-aware Digital Subscriber Line Access Multiplexers (DSLAMs) of the DSL infrastructure. Moreover, we consider the pragmatic cost of repairing anomalies that were misclassified and characterize each classifier according to the overall cost that is possible to incur to the network operator. In parallel, this work attempts to practically improve the network-wide anomaly classification performance by proposing a semi-supervised classification scheme that updates the initial supervised scheme by testing unlabelled anomalies occurring at anomaly-unaware DSLAMs.

We unveil asymptotic characterizations of the average minimum mean-squared error (MMSE) and the average mutual information in scalar fading coherent channels, where the receiver knows the exact fading channel state but the transmitter knows only the fading channel distribution, driven by a range of inputs both in the regimes of low-SNR - and at the heart of the novelty of the contribution - high-SNR. We also unveil connections to and generalizations of the MMSE dimension. By capitalizing on the characterizations, we conclude with applications of the results to the optimization of the constrained capacity of a bank of parallel independent coherent fading channels.

This paper presents fundamental limits associated with compressive classification of Gaussian mixture source models. In particular, we offer an asymptotic characterization of the behavior of the (upper bound to the) misclassification probability associated with the optimal Maximum-A-Posteriori (MAP) classifier that depends on quantities that are dual to the concepts of diversity gain and coding gain in multi-antenna communications. The diversity, which is shown to determine the rate at which the probability of misclassification decays in the low noise regime, is shown to depend on the geometry of the source, the geometry of the measurement system and their interplay. The measurement gain, which represents the counterpart of the coding gain, is also shown to depend on geometrical quantities. It is argued that the diversity order and the measurement gain also offer an optimization criterion to perform dictionary learning for compressive classification applications.

We investigate connections between information-theoretic and estimation-theoretic quantities in vector Poisson channel models. In particular, we generalize the gradient of mutual information with respect to key system parameters from the scalar to the vector Poisson channel model. We also propose, as another contribution, a generalization of the classical Bregman divergence that offers a means to encapsulate under a unifying framework the gradient of mutual information results for scalar and vector Poisson and Gaussian channel models. The so-called generalized Bregman divergence is also shown to exhibit various properties akin to the properties of the classical version. The vector Poisson channel model is drawing considerable attention in view of its application in various domains: as an example, the availability of the gradient of mutual information can be used in conjunction with gradient descent methods to effect compressive-sensing projection designs in emerging X-ray and document classification applications.

Motivated by applications in high-dimensional signal processing, we derive fundamental limits on the performance of compressive linear classifiers. By analogy with Shannon theory, we define the classification capacity, which quantifies the maximum number of classes that can be discriminated with low probability of error, and the diversity-discrimination tradeoff, which quantifies the tradeoff between the number of classes and the probability of classification error. For classification of Gaussian mixture models, we identify a duality between classification and communications over non-coherent multiple-antenna channels. This duality allows us to characterize the classification capacity and diversity-discrimination tradeoff using existing results from multiple-antenna communication. We also identify the easiest possible classification problems, which correspond to low-dimensional subspaces drawn from an appropriate Grassmann manifold.

We consider design of linear projection measurements for a vector Poisson signal model. The projections are performed on the vector Poisson rate, $X\in\mathbb{R}_+^n$, and the observed data are a vector of counts, $Y\in\mathbb{Z}_+^m$. The projection matrix is designed by maximizing mutual information between $Y$ and $X$, $I(Y;X)$. When there is a latent class label $C\in\{1,\dots,L\}$ associated with $X$, we consider the mutual information with respect to $Y$ and $C$, $I(Y;C)$. New analytic expressions for the gradient of $I(Y;X)$ and $I(Y;C)$ are presented, with gradient performed with respect to the measurement matrix. Connections are made to the more widely studied Gaussian measurement model. Example results are presented for compressive topic modeling of a document corpora (word counting), and hyperspectral compressive sensing for chemical classification (photon counting).

In this letter, we propose a new method for the joint design of both the projections and the sparsifying dictionary in order to improve signal reconstruction performance in compressive sensing (CS) applications. By capitalizing on the optimized projection matrix design in , which admits a closed-form expression as a function of any overcomplete dictionary, the proposed method does not need to involve directly the projection matrix. The projection matrix of our joint design can be directly derived based on the learned dictionary. Simulation results show that our joint design framework, which is constituted based on a set of training image patches, leads to an improved reconstruction performance in comparison to other recent approaches.

IPTV networks blindly rely on the adequate operation and management of the underlying infrastructure that in numerous cases is threaten by unexpected anomalous events which consequently cause QoS degradation to the end-user. Thus, it is of great importance to deploy techniques embodied with diagnostic and self-protection metrics for determining and predicting the arrival of such events in order to proactively charge defense mechanisms without the need of an exhaustive manual inspection by the network operator. In this paper we propose and demonstrate the applicability of the Rényi entropy as a useful diagnosis feature for explicitly characterizing DSL-level anomalies issued in an IPTV network of a large European ISP. It is revealed that different orders of the Rényi entropy can formulate meaningful detection and categorization of phenomena occurring on specific Digital Subscriber Line Access Multiplexers (DSLAMs) within the DSL infrastructure. Via the synergistic exploitation of the local maxima peaks generated by each Rényi-based distribution we exhibit the feasibility to extract and identify lightweight anomalies that under simple metrics cannot be detected.

Sparse signals can be sensed with a reduced number of random projections and then reconstructed if compressive sensing (CS) is employed. Traditionally, the projection matrix has been chosen as a random Gaussian matrix, but improved reconstruction performance can be obtained by optimizing the projection matrix. In this paper, we are interested in projection matrix designs for sensing sparse signals in overcomplete dictionaries. In particular, we put forth a closed form design that stems from the formulation of an optimization problem, which bypasses the complexity of iterative design approaches.

This paper is motivated by the challenge of high fidelity processing of images using a relatively small set of projection measurements. This is a problem of great interest in many sensing applications, for example where high photodetector counts are precluded by a combination of available power, form factor and expense. The emerging methods of dictionary learning and compressive sensing offer great potential for addressing this challenge. Combining these methods requires that the signals of interest be representable as a sparse combination of elements of some dictionary. This paper develops a method that aligns the discriminative power of such a dictionary with the physical limitations of the imaging system. Alignment is accomplished by designing a projection matrix that exposes and then aligns the modes of the noise with those of the dictionary. The design algorithm is obtained by modifying an algorithm for designing the pre-filter to maximize the rate and reliability of a Multiple Input Multiple Output (MIMO) communications channel. The difference is that in the communications problem a source is being matched to a channel, whereas in the imaging problem a channel, or equivalently the noise covariance, is being matched to a source. Our results shown that using the proposed communications design framework we can reduce reconstruction error between 20%, after only 20 projections of a 28 × 28 image, and 10% after 100 projections. Furthermore, we noticeably see the superior quality of the reconstructed images.

Compressive sensing (CS), a new sampling paradigm, has recently found several applications in wireless sensor networks (WSNs). In this paper, we investigate the design of novel sensing matrices which lead to good expected-case performance - a typical performance indicator in practice - rather than the conventional worst-case performance that is usually employed when assessing CS applications. In particular, we show that tight frames perform much better than the common CS Gaussian matrices in terms of the reconstruction average mean squared error (MSE). We also showcase the benefits of tight frames in two WSN applications, which involve: i) robustness to data sample losses; and ii) reduction of the communication cost.

This paper characterizes the constrained capacity of multiple-antenna fading coherent channels driven by arbitrary discrete inputs in the regime of high signal-to-noise ratio (snr). In particular, we capitalize on key asymptotic expansions to unveil the effect on the constrained capacity of various channel and system parameters, such as the fading model, the number of transmit and receive antennas, antenna correlation and the characteristics of the signalling scheme. It is also shown that the effect of certain parameters on the constrained capacity is radically different from their effect on the channel capacity achieved by Gaussian inputs.

We consider communication between a legitimate transmitter and a legitimate receiver in the presence of an eavesdropper in the conventional Gaussian wiretap channel setting. Based on Wyner's coding scheme, we show that a code construction using orthogonal codewords can achieve the secrecy capacity of the Gaussian wiretap channel in the wideband regime. This is illustrated through analysis of the error probability between the legitimate parties and the eavesdropper equivocation rate, as well as various simulation results.

We consider the recovery of an underlying signal $\mathbf{x}\in\mathbb{C}^m$ based on projection measurements of the form $\mathbf{y}=\mathbf{M}\mathbf{x}+\mathbf{w}$, where $\mathbf{y}\in\mathbb{C}^\ell$ and $\mathbf{w}$ is measurement noise; we are interested in the case $\ell\ll m$. It is assumed that the signal model $p(\mathbf{x})$ is known and that $\mathbf{w}\sim\mathcal{CN}(\mathbf{w};\boldsymbol{0},\bf \Sigma_w)$ for known $\bf \Sigma_w$. The objective is to design a projection matrix $\mathbf{M}\in\mathbb{C}^{\ell\times m}$ to maximize key information-theoretic quantities with operational significance, including the mutual information between the signal and the projections $\mathcal{I}(\mathbf{x};\mathbf{y})$ or the Rényi entropy of the projections $\mbox{h}_\alpha \left( \mathbf{y} \right)$ (Shannon entropy is a special case). By capitalizing on explicit characterizations of the gradients of the information measures with respect to the projection matrix, where we also partially extend the well-known results of Palomar and Verdú from the mutual information to the Rényi entropy domain, we reveal the key operations carried out by the optimal projection designs: mode exposure and mode alignment. Experiments are considered for the case of compressive sensing (CS) applied to imagery. In this context, we provide a demonstration of the performance improvement possible through the application of the novel projection designs in relation to conventional ones, as well as justification for a fast online projection design method with which state-of-the-art adaptive CS signal recovery is achieved.

This letter considers the design of sensing matrices with good expected-case performance for compressive sensing applications. By capitalizing on the mean squared error (MSE) of the oracle estimator, whose performance has been shown to act as a benchmark to the performance of standard sparse recovery algorithms, we demonstrate that a unit-norm tight frame is the closest design-in the Frobenius norm sense-to the solution of a convex relaxation of the optimization problem that relates to the minimization of the MSE of the oracle estimator with respect to the sensing matrix. Simulation results reveal that the MSE performance of a unit-norm tight frame based sensing matrix surpasses that of other standard sensing matrix designs in various scenarios, which include sparse recovery with basis pursuit denoise (BPDN), the Dantzig selector and orthogonal matching pursuit (OMP). This also has important practical implications because a unit-norm tight frame based sensing matrix can be designed very efficiently.

Compressive sensing leverages the compressibility of natural signals to trade off the convenience of data acquisition against computational complexity of data reconstruction. Thus, CS appears to be an excellent technique for data acquisition and reconstruction in a wireless sensor network (WSN) which typically employs a smart fusion center (FC) with a high computational capability and several dumb front-end sensors having limited energy storage. This paper presents a novel signal reconstruction method based on CS principles for applications in WSNs. The proposed method exploits both the intra-sensor and inter-sensor correlation to reduce the number of samples required for reconstruction of the original signals. The novelty of the method relates to the use of the Frechet mean of the signals as an estimate of their sparse representations in some basis. This crude estimate of the sparse representation is then utilized in an enhanced data recovering convex algorithm, i.e., the penalized ℓ1minimization, and an enhanced data recovering greedy algorithm, i.e., the precognition matching pursuit (PMP). The superior reconstruction quality of the proposed method is demonstrated by using data gathered by a WSN located in the Intel Berkeley Research lab.

In this paper, we propose a penalized ℓ1minimization algorithm for reconstructing a time-varying signal based on compressive sensing (CS) principles. The time-varying signal can be seen as a sequence of slow-changing frames. In the proposed algorithm, all frames of the sequence are sampled at an equal rate, which makes the encoder simpler than frame-categorized methods. We introduce a specialized Fréchet mean of the target frame and several adjacent frames as the penalty vector to make the algorithm close to ℓ0minimization. We prove that the specialized Fréchet mean is a good approximation of the target frame for a sequence of slow time-varying signals. Experimental results demonstrates the superior reconstruction quality of the proposed algorithm.

This paper introduces a divide and conquer approach to the design of transmit and receive filters for communication over a Multiple Input Multiple Output (MIMO) Gaussian channel subject to an average power constraint. It involves conversion to a set of parallel scalar channels, possibly with very different gains, followed by coding per sub-channel (i.e. over time) rather than coding across sub-channels (i.e. over time and space). The loss in performance is negligible at high signal-to-noise ratio (SNR) and not significant at medium SNR. The advantages are reduction in signal processing complexity and greater insight into the SNR thresholds at which a channel is first allocated power. This insight is a consequence of formulating the optimal power allocation in terms of an upper bound on error rate that is determined by parameters of the input lattice such as the minimum distance and kissing number. The resulting thresholds are given explicitly in terms of these lattice parameters. By contrast, when the optimization problem is phrased in terms of maximizing mutual information, the solution is mercury waterfilling, and the thresholds are implicit.

This paper considers the problem of filter design with secrecy constraints, where two legitimate parties (Alice and Bob) communicate in the presence of an eavesdropper (Eve), over a Gaussian multiple-input multiple-output (MIMO) wiretap channel. This problem involves the design of transmit and receive filters which minimize the mean-square error (MSE) between the legitimate parties, whilst assuring that the eavesdropper MSE remains above a certain level. We characterize the form of the optimal transmit filter when both the legitimate receiver and the eavesdropper employ Zero-Forcing (ZF) filters. By capitalizing on the dual problem, we also show that the original matrix optimization problem can be reduced to a simple scalar optimization problem, whose solution can be readily computed by employing a simple bisection method. Numerical results illustrate the main conclusions.

This paper presents a novel signal reconstruction method based on the distributed compressive sensing (DCS) framework for application to wireless sensor networks (WSN). The proposed method exploits both the intra-sensor correlation and the inter-sensor correlation to reduce the number of samples required for recovering the original signals. An innovative feature of our method is using the Fr' echet mean of the signals to discover the common support of their sparse representations in some basis. Then a new greedy algorithm, called precognition matching pursuit (PMP), is proposed to further reduce the number of required samples with the knowledge of the common support. The superior reconstruction quality of the proposed method is demonstrated by both computer-generated signals and real data gathered by a WSN located in the Intel Berkeley Research lab.

This paper investigates the constrained capacity of the canonical i.i.d. multi-antenna Rayleigh fading coherent channel with arbitrary equiprobable discrete inputs in a regime of high signal-to-noise ratio (SNR), in the scenario where the receiver knows the channel state but the transmitter knows only the channel distribution. By capitalizing on the relation between mutual information and minimum mean-squared error (MMSE), we provide high-SNR asymptotic expansions to the average mutual information and the average MMSE. The expansions unveil the impact of the number of transmit and receive antennas and the geometry of the multi-dimensional input constellations on the constrained capacity of the communications channel.

This paper considers the problem of Alter design with secrecy constraints, where two legitimate parties, Alice and Bob, communicate in the presence of an eavesdropper, Eve, over multiple-input multiple-output (MIMO) Gaussian channels. In particular, we consider the design of transmit and receive filters that minimize the mean-squared error (MSE) between the legitimate parties subject to a certain eavesdropper MSE level, in the situation where the eavesdropper MIMO channel is a degraded version of the main MIMO channel. We characterize the form of the optimal receive filters as well as the form of optimal transmit filter in different scenarios. We also put forth an iterative algorithm to obtain the optimal values of the transmit and receive filters. Finally, we present a set of numerical results that illustrate the conclusions.

This paper introduces a less is more approach to the modelling of iterative decoding for bit-interleaved coded modulation (BICM). It considers modelling the iterative exchange of soft a priori information between the demapper and decoder with a binary erasure channel model. This simplification, unlike the usual additive white Gaussian noise (AWGN) channel model, allows us to express the extrinsic information transfer (EXIT) chart curves for the demapper with polynomials whose order makes explicit the degrees of freedom available for design. The accuracy of our expressions simplifies the design process enabling us to compare in a more manageable manner the key performance parameters: the BER performance and the convergence threshold. This enabled us to find the pairing of memory-two recursive systematic convolutional (RSC) codes and mappers that achieve the earliest convergence threshold for a target BER floor at convergence: for 8-PSK semi-set partitioning (SSP) was optimal and converged at a signal-to-noise ratio (SNR)= 3.0 dB for a BER target of at least 10-5; two new mappings were the optimal pairings for 16-QAM, where the L4b2 mapping converged at 2.9 dB for a BER floor ≤ 3.46 × 10-4and the H2b2 mapping converged at 3.1 dB for a BER floor ≤ 2.44 × 10-5.

In this correspondence, we characterize the probability of secrecy-outage and the asymptotic high-signal-to-noise ratio (SNR) ε-outage secrecy-capacity for a single-input-single-output-multi-eavesdropper (SISOME) wireless system with eavesdroppers performing maximum ratio combining (MRC) or selection diversity combining (SDC) reception. We also consider the SISO2E case with eavesdropper antenna-correlation and finally, analyze the scenario where the eavesdropper has Rician fading links with the transmitter.

In this paper, we consider secure communications between a single antenna transmitter and a single antenna receiver in the presence of a multiple antenna eavesdropper employing MRC or SDC reception. We show that an M-antenna eavesdropper performing SDC reception has precisely the same degrading effect on secrecy as M single antenna eavesdroppers under independent fading conditions. We also show that an M-antenna eavesdropper performing MRC reception causes greater secrecy degradation than an M-antenna eavesdropper performing SDC reception or M single antenna eavesdroppers. We also derive closed-form expressions for the asymptotic high SNR outage secrecy capacity for both the MRC and the SDC eavesdropper cases. Various numerical results are presented that corroborate the analysis.

In this paper, we put forth a model to generate closed-form expressions for EXIT chart curves for BICM-ID systems. This model is based on the representation of the a-priori AWGN channels by BEC channels. We present various applications of the model in the estimation of key performance metrics in BICM-ID systems. This also includes the selection of suitable mapper-encoder pairings for quasi-static fading channels that exhibit the best known performance.

This paper investigates the transmission in time dispersive channels of Spectrally Efficient Frequency Division Multiplexed (SEFDM) signals, where carrier orthogonality is intentionally violated in order to increase bandwidth efficiency. Sufficient statistics of the transmitted SEFDM signal can be obtained by projecting the received signal onto an orthonormal base generated at the receiver using an Iterative Modified Gram Schmidt (IMGS) procedure. In order to reduce the computational complexity resulting from Inter-Carrier Interference (ICI), detection has been implemented based on a Regularized Sphere Decoding (RSD) algorithm. The proposed scheme was previously tested in Additive White Gaussian Noise (AWGN) for various SEFDM signal parameters. In the present work, these results are extended to account for the effect of time dispersive channels. Randomly generated SEFDM symbols are used as pilots to provide estimates of the channel impulse response in systems with or without cyclic prefixes. A joint equalization-detection is subsequently performed in a RSD stage. We show that it is possible to detect optimally SEFDM signals of small dimensionality (e.g. N = 32), with up to 20% bandwidth gain with respect to OFDM systems of the same symbol-rate. This indicates that the wireless transmission of non orthogonal SEFDM signals is tangible.

In this study, the authors investigate the performance-complexity tradeoff of convolutional codes for broadband fixed wireless access systems by considering the effects of quantisation and path metric memory in practical Viterbi decoding implementations. They show that in systems with limited antenna diversity, low-memory codes achieve a better error-rate performance compared to that of high-memory codes. Only in systems with considerable antenna diversity, can the performance of a convolutional code be improved by increasing its memory size. Nevertheless, the authors demonstrate that the coding advantage offered by the high-memory codes is not large enough to justify the significant increase in implementation complexity. In particular, memory-2 convolutional codes achieve a coding gain of up to 1.2 dB over their memory-8 counterparts in single-input single-output fixed wireless access systems. The situation is reversed when multiple antennas are used, but the decoder of memory-8 codes occupies at least 130 times more silicon area than that of memory-2 codes.

In this paper, we investigate the linear precoding and power allocation policies that maximize the mutual information for general multiple-input-multiple-output (MIMO) Gaussian channels with arbitrary input distributions, by capitalizing on the relationship between mutual information and minimum mean-square error (MMSE). The optimal linear precoder satisfies a fixed-point equation as a function of the channel and the input constellation. For non-Gaussian inputs, a nondiagonal precoding matrix in general increases the information transmission rate, even for parallel noninteracting channels. Whenever precoding is precluded, the optimal power allocation policy also satisfies a fixed-point equation; we put forth a generalization of the mercury/waterfilling algorithm, previously proposed for parallel noninterfering channels, in which the mercury level accounts not only for the non-Gaussian input distributions, but also for the interference among inputs.

This paper investigates the transmission of frequency division multiplexed (FDM) signals, where carrier orthogonality is intentionally violated in order to increase bandwidth efficiency. In analogy to conventional OFDM, signal generation relies on an inverse fractional Fourier transform (IFRFT) that can be implemented with O(N log2N) algorithmic complexity. Optimal maximum likelihood (ML) detection is overly complex due to the presence of substantial intercarrier interference (ICI). Consequently, we investigate an alternative detection mechanism based on the generalized sphere decoding (GSD) algorithm. We examine the bandwidth efficiency and the error performance in additive white gaussian noise (AWGN), for various FDM signal parameters. In particular, we show that it is possible to detect optimally and efficiently FDM signals, with 25% bandwidth gain with respect to analogous OFDM signals. This indicates that the transmission of spectrally efficient non orthogonal FDM signals is tangible.

This paper considers communication over channels subject to jamming. By capitalizing on the relationship between the mutual information and the minimum mean-squared error (MMSE), we investigate the interference covariance that minimizes the mutual information of a deterministic multiple-input multiple-output (MIMO) channel subject to Gaussian noise and Gaussian interference with arbitrary (not necessarily Gaussian) input distributions. We show that the worst interference covariance satisfies a fixed-point equation involving key system quantities, including the MMSE matrix. We also specialize the form of the worst interference covariance to the asymptotic regimes of low and high snr. We demonstrate that in the low-snr regime the worst interference covariance injects an appropriate amount of power directly into the channel eigenmodes. In contrast, in the high-snr regime the worst interference covariance minimizes the minimum distance between a modified version of the constellation vectors. Numerical results illustrate that optimization of the interference covariance has the potential to substantially decrease the reliable information transmission rate between a transmitter-receiver pair. The results are also applicable to scenarios where a jammer aims to impair the secrecy rate of wiretap channels.

Recent years have witnessed some interest in Spectrally Efficient Frequency Division Multiplexing (SEFDM) communications systems, where subcarrier orthogonality is intentionally violated to improve the spectral efficiency at the expense of system complexity. This paper investigates reliable polynomial-time hard detection techniques for SEFDM systems, by relaxing the optimal combinatorial Maximum Likelihood (ML) detection to a Semidefinite Program (SDP). SDP can be solved in almost cubic complexity over the number of the SEFDM subcarriers, N. However, the relaxation results into a degradation of the system error performance. In particular, we study the effect of the number of SEFDM subcarriers, N, and the subcarrier separation, ¿f, on the SDP relaxation gap in the presence of Additive White Gaussian Noise (AWGN). We find that as N increases and/or ¿f decreases, the SDP estimate gradually diverges from the optimal solution. To overcome this problem, we propose the use of a boxed ML procedure around the SDP estimate. We show by simulation that the SDP-ML combination approximates the optimum detection for N ¿ 32 subcarriers and up to 20% of bandwidth reduction with respect to an equivalent Orthogonal FDM (OFDM). Our SDP results show a small error penalty when compared to optimal Sphere Decoders (SD), whose computational effort is random and noise dependant, and thereby indicate that our proposed technique is useable in practical SEFDM systems with a moderate number of subcarriers.

The objective of this paper is two-fold. Initially, we present an analytic technique to rapidly evaluate an approximation to the union bound on the bit error probability of turbo codes. This technique exploits the most significant terms of the union bound, which can be calculated straightforwardly by considering the properties of the constituent convolutional encoders. Subsequently, we use the bound approximation to demonstrate that specific punctured rate-1/2 turbo codes can achieve a lower error floor than that of their rate-1/3 parent codes. In particular, we propose pseudo-random puncturing as a means of improving the bandwidth efficiency of a turbo code and simultaneously lowering its error floor.

Convolutional block codes, which are commonly used as constituent codes in turbo code configurations, accept a block of information bits as input rather than a continuous stream of bits. In this paper, we propose a technique for the calculation of the transfer function of convolutional block codes, both punctured and nonpunctured. The novelty of our approach lies in the augmentation of the conventional state diagram, which allows the enumeration of all codeword sequences of a convolutional block code. In the case of a turbo code, we can readily calculate an upper bound to its bit error rate performance if the transfer function of each constituent convolutional block code has been obtained. The bound gives an accurate estimate of the error floor of the turbo code and, consequently, our method provides a useful analytical tool for determining constituent codes or identifying puncturing patterns that improve the bit error rate performance of a turbo code, at high signal-to-noise ratios.

In this paper, we investigate the possibility of reliable and computationally efficient detection for spectrally efficient non-orthogonal Multiplexing (FDM) system, exhibiting varying levels of intercarrier interference. Optimum detection is based on the Maximum Likelihood (ML) principle. However, ML is impractical due to its computational complexity. On the other hand, linear detection techniques such as Zero Forcing (ZF) and Minimum Mean Square Error (MMSE) exhibit poor performance. Consequently, we explore the combination of MMSE estimation with ML estimation around a neighborhood of the MMSE estimate. We evaluate the performance of the different schemes in Additive White Gaussian Noise (AWGN), with reference to the number of FDM carriers and their frequency separation. The combined MMSE-ML scheme achieves a near optimum error performance with polynomial complexity for a small number of BPSK FDM carriers. For QPSK modulation the performance of the proposed system improves for a large number of ML comparisons. In all cases, the detectability of the FDM signal is bounded by the signal dimension and the carriers frequency distance.

We introduce the problem of filter design with secrecy constraints in the classical wiretap scenario, where two legitimate parties, Alice and Bob, communicate in the presence of an eavesdropper, Eve. In particular, we consider the design of transmit and receive filters that minimize the mean-squared error (MSE) between the legitimate parties whilst guaranteeing a certain eavesdropper MSE level subject to a total average power constraint, in the situation where the main channel and the wiretap channel consist of a bank of parallel independent degraded Gaussian channels (a scenario representative of orthogonal frequency division multiplexing (OFDM) communications systems). We derive the form of the optimal receive filter and the optimal transmit filter, which are diagonal. In the regime of low available power, we demonstrate that the diagonal elements of the optimal transmit filter satisfy a waterfilling type of expression. In contrast, in the regime of high available power the diagonal elements of the optimal transmit filter satisfy a simple expression involving the linear minimum mean-squared error (LMMSE) that obeys a simple operational interpretation. A range of numerical results corroborate the conclusions.

We determine the linear precoding policy that maximizes the mutual information for general multiple-input multiple-output (MIMO) Gaussian channels with arbitrary input distributions, by capitalizing on the relationship between mutual information and minimum mean squared error (MMSE). The optimal linear precoder can be computed by means of a fixed- point equation as a function of the channel and the input constellation. We show that diagonalizing the channel matrix does not maximize the information transmission rate for nonGaussian inputs. A full precoding matrix may significantly increase the information transmission rate, even for parallel non-interacting channels. We illustrate the application of our results to typical Gigabit DSL systems.

We investigate the input covariance that maximizes the mutual information of deterministic multiple-input multipleo-utput (MIMO) Gaussian channels with arbitrary (not necessarily Gaussian) input distributions, by capitalizing on the relationship between the gradient of the mutual information and the minimum mean-squared error (MMSE) matrix. We show that the optimal input covariance satisfies a simple fixed-point equation involving key system quantities, including the MMSE matrix. We also specialize the form of the optimal input covariance to the asymptotic regimes of low and high snr. We demonstrate that in the low-snr regime the optimal covariance fully correlates the inputs to better combat noise. In contrast, in the high-snr regime the optimal covariance is diagonal with diagonal elements obeying the generalized mercury/waterfilling power allocation policy. Numerical results illustrate that covariance optimization may lead to significant gains with respect to conventional strategies based on channel diagonalization followed by mercury/waterfilling or waterfilling power allocation, particularly in the regimes of medium and high snr.

The performance of turbo codes in quasi-static fading channels both with and without antenna diversity is investigated. In particular, simple analytic techniques that relate the frame error rate of a turbo code to both its average distance spectrum as well as the iterative decoder convergence characteristics are developed. Both by analysis and simulation, the impact of the constituent recursive systematic convolutional (RSC) codes, the interleaver size and the number of decoding iterations on the performance of turbo codes are also investigated. In particular, it is shown that in systems with limited antenna diversity different constituent RSC codes or interleaver sizes do not affect the performance of turbo codes. In contrast, in systems with significant antenna diversity, particular constituent RSC codes and interleaver sizes have the potential to significantly enhance the performance of turbo codes. These results are attributed to the fact that in single transmit–single receive antenna systems, the performance primarily depends on the decoder convergence characteristics for Eb/N0 values of practical interest. However, in multiple transmit–multiple receive antenna systems, the performance depends on the code characteristics.

This paper considers the transmission of confidential data over wireless channels. Based on an information-theoretic formulation of the problem, in which two legitimates partners communicate over a quasi-static fading channel and an eavesdropper observes their transmissions through a second independent quasi-static fading channel, the important role of fading is characterized in terms of average secure communication rates and outage probability. Based on the insights from this analysis, a practical secure communication protocol is developed, which uses a four-step procedure to ensure wireless information-theoretic security: (i) common randomness via opportunistic transmission, (ii) message reconciliation, (iii) common key generation via privacy amplification, and (iv) message protection with a secret key. A reconciliation procedure based on multilevel coding and optimized low-density parity-check (LDPC) codes is introduced, which allows to achieve communication rates close to the fundamental security limits in several relevant instances. Finally, a set of metrics for assessing average secure key generation rates is established, and it is shown that the protocol is effective in secure key renewal—even in the presence of imperfect channel state information.

It has been observed that particular rate-1/2 partially systematic parallel concatenated convolutional codes (PCCCs) can achieve a lower error floor than that of their rate-1/3 parent codes. Nevertheless, good puncturing patterns can only be identified by means of an exhaustive search, whilst convergence towards low bit error probabilities can be problematic when the systematic output of a rate-1/2 partially systematic PCCC is heavily punctured. In this paper, we present and study a family of rate-1/2 partially systematic PCCCs, which we call pseudo-randomly punctured codes. We evaluate their bit error rate performance and we show that they always yield a lower error floor than that of their rate-1/3 parent codes. Furthermore, we compare analytic results to simulations and we demonstrate that their performance converges towards the error floor region, owning to the moderate puncturing of their systematic output. Consequently, we propose pseudo-random puncturing as a means of improving the bandwidth efficiency of a PCCC and simultaneously lowering its error floor.

In this paper, we investigate in detail the performance of iterative demapping and decoding techniques over quasi-static fading channels both with and without antenna diversity. In particular, we consider the effect on the system performance of various mapping schemes, different coding schemes, the inter- leaver size as well as space diversity. Results demonstrate that over quasi-static fading channels characterized by significant antenna diversity, mappings traditionally optimized for iterative receivers (e.g., Boronka mapping) outperform mappings more appropriate for non-iterative receivers (e.g., Gray mapping). In contrast, over quasi-static fading channels characterized by limited antenna diversity, Gray mapping always outperform Boronka mapping for all Eb/N0and all iterations. Strikingly, this situation is in sharp contrast to that in the AWGN case. We note that we can further improve the performance of non-iterative systems (e.g., Gray mapping) having limited antenna diversity by increasing the memory size and removing the interleaving.

Multiple input multiple output-orthogonal frequency division multiplexing-code division multiplexing (MIMO-OFDM-CDM) techniques are considered to improve the link reliability/spectral efficiency of very high data rate communication systems. In particular, lattice-reduction-aided receivers are proposed for MIMO-OFDM-CDM systems. Simulation results show that the proposed receivers significantly outperform the conventional zero-forcing, minimum mean-squared error, or vertical Bell Labs layered space–time receivers without severely compromising system complexity.

A novel approach for the calculation of the transfer function of a terminated punctured convolutional code, which can be seen as a convolutional block code, is presented in this paper. The transfer function of the convolutional block code can be then used to evaluate the transfer function of a punctured turbo code and derive a tight upper bound on the bit error probability. Furthermore, the approach is used to find puncturing patterns that generate punctured turbo codes with optimal performance.

We consider the transmission of confidential data over wireless channels with multiple communicating parties. Based on an information-theoretic problem formulation in which two legitimate partners communicate over a quasi-static fading channel and an eavesdropper observes their transmissions through another independent quasi-static fading channel, we define the secrecy capacity in terms of outage probability and provide a complete characterization of the maximum transmission rate at which the eavesdropper is unable to decode any information. In sharp contrast with known results for Gaussian wiretap channels (without feedback), our contribution shows that in the presence of fading information-theoretic security is achievable even when the eavesdropper has a better average signal-to-noise ratio (SNR) than the legitimate receiver - fading thus turns out to be a friend and not a foe.

In this paper, we investigate in detail the performance of turbo codes in quasi-static fading channels both with and without antenna diversity. First, we develop a simple and accurate analytic technique to evaluate the performance of turbo codes in quasi-static fading channels. The proposed analytic technique relates the frame error rate of a turbo code to the iterative decoder convergence threshold, rather than to the turbo code distance spectrum. Subsequently, we compare the performance of various turbo codes in quasi-static fading channels. We show that, in contrast to the situation in the AWGN channel, turbo codes with different interleaver sizes or turbo codes based on RSC codes with different constraint lengths and generator polynomials exhibit identical performance. Moreover, we also compare the performance of turbo codes and convolutional codes in quasi-static fading channels under the condition of identical decoding complexity. In particular, we show that turbo codes do not outperform convolutional codes in quasi-static fading channels with no antenna diversity; and that turbo codes only outperform convolutional codes in quasi-static fading channels with antenna diversity

In this paper, we propose using selective mapping (SLM) and partial transmit sequences (PTS) based on an intermodulation distortion (IMD) reduction strategy to improve the error probability performance of non-linearly distorted orthogonal frequency division multiplex (OFDM) signals. Simulation results demonstrate that in the presence of nonlinearities OFDM systems using SLM or PTS with IMD reduction perform better than those with peak-to-average power ratio (PAPR) reduction. Additionally, simulation results demonstrate that the average out-of-band power generated by IMD reduction strategies is lower than that generated by PAPR reduction strategies. Finally, we propose a sub-optimum solution for the practical realization of SLM OFDM and PTS OFDM systems based on IMD reduction which does not result in a significant loss in performance.

We investigate the influence of different Walsh-Hadamard (WH) code allocation techniques on the performance of multi-carrier code division multiple access (CDMA) systems in the presence of high power amplifier (HPA) non-linearities. We consider two different multi-carrier CDMA schemes: multi-carrier CDMA (MC-CDMA) and multi-carrier direct sequence CDMA (MC-DS-CDMA) and analyse their performance in terms of total degradation and spectral spreading for different numbers of active users.

The combination of code division multiple access (CDMA) and multi-carrier modulation (MCM) (referred to as multi-carrier CDMA) has been proposed as a possible candidate for future generations of wireless/mobile systems. Much work has been done on the performance of such schemes in various environments, with different channel models, predominantly assuming linear channels. Accordingly, in this paper we explore the influence of the effects of non-linearities (introduced by the transmitter high power amplifier (HPA)) on multi-carrier CDMA systems using higher order modulation schemes. We consider two techniques for combining CDMA with MCM: multi-carrier CDMA (MC-CDMA) and multi-carrier direct sequence CDMA (MC-DS-CDMA) and analyse their performance in the presence of an HPA and additive white Gaussian noise (AWGN), paying particular attention to the total degradation and the spectral spreading.