Change detection (CD) in remote sensing images is an important problem requiring the analysis of high-dimensional data such as radar, multispectral, and hyperspectral images. Solutions for radar and multispectral data are available in the literature. However, the use of hyperspectral images is still an area of improvement, not only because the existing methods were designed to handle a reduced number of spectral bands but also because they do not fully exploit the rich spectral information available for CD. Additionally to the use of hyperspectral data, the interest in multimodal CD is that different sensors, such as optical and synthetic aperture radar (SAR), can contribute with essential features improving the generation of high-quality change maps. This paper addresses the multimodal CD problem using hyperspectral and optical/SAR images or pairs of hyperspectral images using a multivariate manifold estimation model based on spectral unmixing. The estimated abundance maps associated with hyperspectral images are used as inputs to the CD strategy, instead of the original hyperspectral data enabling significant dimensionality reduction. Several simulations demonstrate the effectiveness of the proposed approach, achieving, for instance, overall accuracy values above 91%, AUC values up to 0.95, and consistently higher recall compared to competing methods. These results highlight the strong performance of the proposed method while preserving its flexibility to handle any combination of radar, multispectral, and hyperspectral images. • Spectral information can play a fundamental role for change detection. • Proposes a multivariate manifold estimation model based on spectral unmixing. • A model that adapts and accommodates various multimodal sensor combinations.

Compressive imaging (CI) consists of reconstructing images from incomplete observed data. The reconstruction process involves solving an ill-posed inverse problem which is highly dependent on the number of real measurements, with a greater number of measurements typically leading to more accurate reconstructions. Due to their ability to learn data distributions, diffusion models (DM) have emerged as promising techniques for various inverse problems. Mainly, DMs solve inverse problems by conditioning the generation process to the acquired measurements. In this work, we introduce a new approach to improve this conditioning by exploiting synthetic measurements, which come from a synthetic sensing matrix. Synthetic measurements are estimated from real data via a neural network. The combined real and synthetic measurements form an augmented set, which is input into the conditional DM to enhance reconstruction capacity. Computational experiments demonstrate that augmenting measurements with the conditional DM improves performance compared to using only real measurements.

In remote sensing and environmental monitoring, the authenticity of the collected hyperspectral image (HSI) is critical since unauthorized changes could cause inaccurate evaluation. Traditional approaches ensure HSI integrity after the acquisition, leaving the data vulnerable to unauthorized access or modification before applying a protective strategy. In response, this work presents a pushbroom HSI system that integrates an authentication key on the acquisition process, ensuring the key imperceptibility without affecting performance in spectral classification tasks. In addition, this work develops an algorithm designed to simulate attacks through class modifications in the HSI. Additionally, a detection algorithm is proposed to identify modifications by comparing the signed HSI with the altered version, effectively recognizing the embedded key. The proposed method was tested using simulated data and acquired data in the optical laboratory. Results demonstrate that the system maintains classification accuracy, further enhancing imperceptibility. Additionally, modifications are detectable by the key with 99% accuracy.

In compressive imaging systems, the scene is acquired via linear coded noisy projections, known as measurements, requiring a recovery process to estimate the underlying signal. This recovery is inherently ill-posed, posing a challenge for accurate signal recovery. Existing methods that employ prior information about the signal often fail in practical scenarios. In this work, instead of developing a new prior over the signal, we exploit the structure of the low-dimensional measurements to synthesize an augmented measurement set that can be used in various recovery methods to improve its performance. We used a deep neural network to generate the synthetic measurements from the acquired data. We show the benefits of this approach in two schemes, deep learning-based recovery and the plug-and-play (PnP) algorithm. Particularly, our method is interpreted as a non-linear preconditioning technique for the PnP algorithm. We show improved performance for different sensing matrices.

In computational optical imaging and wireless communications, signals are acquired through linear coded and noisy projections, which are recovered through computational algorithms. Deep model-based approaches, i.e., neural networks incorporating the sensing operators, are the state-of-the-art for signal recovery. However, these methods require exact knowledge of the sensing operator, which is often unavailable in practice, leading to performance degradation. Consequently, we propose a new recovery paradigm based on knowledge distillation. A teacher model, trained with full or almost exact knowledge of a synthetic sensing operator, guides a student model with an inexact real sensing operator. The teacher is interpreted as a relaxation of the student since it solves a problem with fewer constraints, which can guide the student to achieve higher performance. We demonstrate the improvement of signal reconstruction in computational optical imaging for single-pixel imaging with miscalibrated coded apertures systems and multiple-input multiple-output symbols detection with inexact channel matrix.

Phase retrieval (PR) reconstructs phase information from magnitude measurements, known as coded diffraction patterns (CDPs), whose quality depends on the number of snap-shots captured using coded phase masks. High-quality phase estimation requires multiple snapshots, which is not desired for efficient PR systems. End-to-end frameworks enable joint optimization of the optical system and the recovery neural network. However, their application is constrained by physical implementation limitations. Additionally, the framework is prone to gradient vanishing issues related to its global optimization process. This paper introduces a Knowledge Distillation (KD) optimization approach to address these limitations. KD transfers knowledge from a larger, lower-constrained network (teacher) to a smaller, more efficient, and implementable network (student). In this method, the teacher, a PR system trained with multiple snapshots, distills its knowledge into a single-snapshot PR system, the student. The loss functions compare the CPMs and the feature space of the recovery network. Simulations demonstrate that this approach improves reconstruction performance compared to a PR system trained without the teacher’s guidance.

Imaging inverse problems aim to recover high-dimensional signals from undersampled, noisy measurements, a fundamentally ill-posed task with infinite solutions in the null-space of the sensing operator. To resolve this ambiguity, prior information is typically incorporated through handcrafted regularizers or learned models that constrain the solution space. However, these priors typically ignore the task-specific structure of that null-space. In this work, we propose Non-Linear Projections of the Null-Space (NPN), a novel class of regularization that, instead of enforcing structural constraints in the image domain, promotes solutions that lie in a low-dimensional projection of the sensing matrix's null-space with a neural network. Our approach has two key advantages: (1) Interpretability: by focusing on the structure of the null-space, we design sensing-matrix-specific priors that capture information orthogonal to the signal components that are fundamentally blind to the sensing process. (2) Flexibility: NPN is adaptable to various inverse problems, compatible with existing reconstruction frameworks, and complementary to conventional image-domain priors. We provide theoretical guarantees on convergence and reconstruction accuracy when used within plug-and-play methods. Empirical results across diverse sensing matrices demonstrate that NPN priors consistently enhance reconstruction fidelity in various imaging inverse problems, such as compressive sensing, deblurring, super-resolution, computed tomography, and magnetic resonance imaging, with plug-and-play methods, unrolling networks, deep image prior, and diffusion models.

Seismic data frequently exhibit missing traces, substantially affecting subsequent seismic processing and interpretation. Deep learning-based approaches have demonstrated significant advancements in reconstructing irregularly missing seismic data through supervised and unsupervised methods. Nonetheless, substantial challenges remain, such as generalization capacity and computation time cost during the inference. This work introduces a reconstruction method that uses a pretrained generative diffusion model for image synthesis and incorporates deep image prior (DIP) to enforce data consistency when reconstructing missing traces in seismic data. The proposed method has demonstrated strong robustness and high reconstruction capability of poststack and prestack data with different levels of structural complexity, even in field and synthetic scenarios where test data were outside the training domain. This indicates that our method can handle the high geological variability of different exploration targets. Additionally, compared to other state-of-the-art seismic reconstruction methods using diffusion models, during inference, our approach reduces the number of sampling timesteps by up to$4\times $. Our implementation is available athttps://github.com/PAULGOYES/CDDIP.git.

The global demand for unconventional energy sources such as geothermal energy and white hydrogen requires new exploration techniques for precise subsurface structure characterization and potential reservoir identification. The magnetotelluric (MT) method is crucial for these tasks, providing critical information on the distribution of subsurface electrical resistivity at depths ranging from hundreds to thousands of meters. However, traditional iterative algorithm-based inversion methods require the adjustment of multiple parameters, demanding time-consuming and exhaustive tuning processes to achieve proper cost function minimization. Recent advances have incorporated deep learning algorithms for MT inversion, primarily based on supervised learning, and large labeled datasets are needed for training. This work utilizes TensorFlow operations to create a differentiable forward MT operator, leveraging its automatic differentiation capability. Moreover, instead of solving for the subsurface model directly, as classical algorithms perform, this letter presents a new deep unsupervised inversion algorithm guided by physics to estimate 1-D MT models. Instead of using datasets with the observed data and their respective model as labels during training, our method employs a differentiable modeling operator that physically guides the cost function minimization, making the proposed method solely dependent on observed data. Therefore, the optimization algorithm updates the network weights to minimize the data misfit. We test the proposed method with field and synthetic data at different acquisition frequencies, demonstrating that the resistivity models obtained are more accurate than those calculated using other techniques. Our implementation is available athttps://github.com/PAULGOYES/MT_guided1DInversion.git.

The modern surge in camera usage alongside widespread computer vision technology applications poses significant privacy and security concerns. Current artificial intelligence (AI) technologies aid in recognizing relevant events and assisting in daily tasks in homes, offices, hospitals, etc. The need to access or process personal information for these purposes raises privacy concerns. While software-level solutions like face de-identification provide a good privacy/utility tradeo-ff, they present vulnerabilities to sniffing attacks. In this paper, we propose a hardware-level face de-identification method to solve this vulnerability. Specifically, our approach first learns an optical encoder along with a regression model to obtain a face heatmap while hiding the face identity from the source image. We also propose an anonymization framework that generates a new face using the privacy-preserving image, face heatmap, and a reference face image from a public dataset as input. We validate our approach with extensive simulations and hardware experiments.

Quantized neural networks employ reduced precision representations for both weights and activations. This quantization process significantly reduces the memory requirements and computational complexity of the network. Binary Neural Networks (BNNs) are the extreme quantization case, representing values with just one bit. Since the sign function is typically used to map real values to binary values, smooth approximations are introduced to mimic the gradients during error backpropagation. Thus, the mismatch between the forward and backward models corrupts the direction of the gradient causing training inconsistency problems and performance degradation. In contrast to current BNN approaches, we propose to employ a binary periodic (BiPer) function during binarization. Specifically, we use a square wave for the forward pass to obtain the binary values and employ the trigonometric sine function with the same period of the square wave as a differentiable surrogate during the backward pass. We demonstrate that this approach can control the quantization error by using the frequency of the periodic function and improves network performance. Extensive experiments validate the effectiveness of BiPer in benchmark datasets and network architectures, with improvements of up to 1% and 0.69% with respect to state-of-the-art methods in the classification task over CIFAR-10 and ImageNet, respectively. Our code is publicly available at https://github.com/edmav4/BiPer.

This paper introduces a plug-and-play algorithm for enhancing compressive spectral imaging (CSI) through the integration of both a quadratic envelope (QE) regularizer and a deep prior. Our method employs the QE-based regularizer to foster a low-rank structure in conjunction with deep priors, synergistically integrated within a Plug-and-Play (PnP) framework. The distinct advantage of our chosen QE-regularizer is its propensity for uncovering low-rank solutions devoid of bias, distinguishing it from the nuclear norm. Through this fusion of QE and deep priors, we harness the complementary strengths of both techniques, resulting in a mutually reinforcing effect for CSI.

Commercial sensors often suffer from overexposure in bright regions, leading to signal clipping and information loss because of saturation. Existing solutions involve either employing logarithmic irradiance response sensors or capturing multiple shots from different saturation levels. However, these approaches can be complex or rely on static scenes, limiting their effectiveness in fully addressing the saturation problem. A promising solution is the use of unsaturated sensors, also known as modulo cameras, which employ an array of self-reset pixels to wrap the signal when it reaches the saturation level. The resulting image exhibits a noisy and discontinuous shape, requiring an unwrapping algorithm to obtain a smooth and continuous representation of the scene. We propose a deep plug-and-play algorithm that combines model-based optimization with a deep denoiser. By leveraging the spatial correlation of the scene within the close solution of an unwrapping step, our approach successfully unwraps the continuous values while simultaneously reducing noise. Extensive evaluations show the superiority of our method compared to state-of-the-art unwrapping and unmodulo algorithms in terms of reconstruction quality.

Recent developments in spectrum-sharing technologies include integrated sensing and communications (ISAC) systems to save resources, cost, and power. In this paper, we consider a co-existence topology with n-tuple radar and communications transmitters, wherein neither the transmitted signal nor the channels are known. Estimating these unknown quantities is modeled as a n-tuple blind deconvolution problem (NTBD). The receiver is considered to be a uniform linear antenna array. Thus, the channels are modeled as continuous-valued time delay, Doppler modulation, and direction of arrival (DoA). Also, harnessing the sparse nature of the channels and their continuousvalued parametrization, we propose a 3D n-tuple atomic norm minimization (NANM). Casting the NANM problem to its corresponding dual optimization problem, and employing the theory of positive trigonometric polynomial, we formulate a semidefinite program for the estimation of the unknown channel parameters. Performance guarantees of the proposed algorithm are provided in terms of the minimum number of samples required for exact recovery. Finally, numerical simulations validate our theoretical insights.

In the era of cloud computing and data-driven applications, it is crucial to protect sensitive information to maintain data privacy, ensuring truly reliable systems. As a result, preserving privacy in deep learning systems has become a critical concern. Existing methods for privacy preservation rely on image encryption or perceptual transformation approaches. However, they often suffer from reduced task performance and high computational costs. To address these challenges, we propose a novel Privacy-Preserving framework that uses a set of deformable operators for secure task learning. Our method involves shuffling pixels during the analog-to-digital conversion process to generate visually protected data. Those are then fed into a well-known network enhanced with deformable operators. Using our approach, users can achieve equivalent performance to original images without additional training using a secret key. Moreover, our method enables access control against unauthorized users. Experimental results demonstrate the efficacy of our approach, showcasing its potential in cloud-based scenarios and privacy-sensitive applications.

Chronic wounds pose an ongoing health concern globally, largely due to the prevalence of conditions such as diabetes and leprosy’s disease. The standard method of monitoring these wounds involves visual inspection by healthcare professionals, a practice that could present challenges for patients in remote areas with inadequate transportation and healthcare infrastructure. This has led to the development of algorithms designed for the analysis and follow-up of wound images, which perform image-processing tasks such as classification, detection, and segmentation. However, the effectiveness of these algorithms heavily depends on the availability of comprehensive and varied wound image data, which is usually scarce. This paper introduces the CO2Wounds-V2 dataset, an extended collection of RGB wound images from leprosy patients with their corresponding semantic segmentation annotations, aiming to enhance the development and testing of image-processing algorithms in the medical field.

Computational optical imaging (COI) systems have enabled the acquisition of high-dimensional signals through optical coding elements (OCEs). OCEs encode the high-dimensional signal in one or more snapshots, which are subsequently decoded using computational algorithms. Currently, COI systems are optimized through an end-to-end (E2E) approach, where the OCEs are modeled as a layer of a neural network and the remaining layers perform a specific imaging task. However, the performance of COI systems optimized through E2E is limited by the physical constraints imposed by these systems. This paper proposes a knowledge distillation (KD) framework for the design of highly physically constrained COI systems. This approach employs the KD methodology, which consists of a teacher-student relationship, where a high-performance, unconstrained COI system (the teacher), guides the optimization of a physically constrained system (the student) characterized by a limited number of snapshots. We validate the proposed approach, using a binary coded apertures single pixel camera for monochromatic and multispectral image reconstruction. Simulation results demonstrate the superiority of the KD scheme over traditional E2E optimization for the designing of highly physically constrained COI systems.

Signal processing over hypercomplex numbers arises in many optical imaging applications. In particular, spectral image or color stereo data are often processed using octonion algebra. Recently, the eight-band multispectral image phase recovery has gained salience, wherein it is desired to recover the eight bands from the phaseless measurements. In this letter, we tackle this hitherto unaddressed hypercomplex variant of the popular phase retrieval (PR) problem. We propose octonion Wirtinger flow (OWF) to recover an octonion signal from its intensity-only observation. However, contrary to the complex-valued Wirtinger flow, the non-associative nature of octonion algebra and the consequent lack of octonion derivatives make the extension to OWF non-trivial. We resolve this using the pseudo-real-matrix representation of octonion to perform the derivatives in each OWF update. We demonstrate that our approach recovers the octonion signal up to a right-octonion phase factor. Numerical experiments validate OWF-based PR with high accuracy under both noiseless and noisy measurements.

Seismic data interpolation of irregularly missing traces plays a crucial role in subsurface imaging, enabling accurate analysis and interpretation throughout the seismic processing workflow. Despite the widespread exploration of deep supervised learning methods for seismic data reconstruction, several challenges remain. Particularly, the requirement for extensive training data and poor domain generalization due to the seismic survey’s variability pose significant issues. To overcome these limitations, this article introduces a deep-learning-based seismic data reconstruction approach that leverages data redundancy. This method involves a two-stage training process. First, an adversarial generative network is trained using synthetic seismic data, enabling the extraction and learning of their primary and local seismic characteristics. Second, a reconstruction network is trained with synthetic data generated by the generative adversarial network (GAN), which dynamically adjusts the distortion level at each epoch to promote feature diversity. This approach enhances the generalization capabilities of the reconstruction network by allowing control over the generation of seismic patterns from the latent space of the GAN, thereby reducing the dependency on large seismic databases. Experimental results on field and synthetic seismic datasets, both pre-stack and post-stack, show that the proposed method outperforms the baseline supervised learning and unsupervised approaches, such as deep seismic prior (DSP) and internal learning (IL), by up to 8 dB of PSNR.

Seismic data acquisition is essential for discovering new hydrocarbon targets, where a high-resolution regular-spaced acquisition is critical to obtain high-quality seismic images. However, the high acquisition costs and environmental impacts have motivated designing seismic surveys with fewer sources and receivers than regular-spaced sensing approaches. After the undersampled measurements are acquired, an algorithm reconstructs the missing information necessary for the subsequent processing and interpretation analysis. The removed data is currently selected using random, jittered, and uniform sensing schemes leading to suboptimal seismic image recovery. Therefore, a guided design of undersampled seismic surveys is important as it determines the quality of the reconstructed information. This work proposes an end-to-end (E2E) optimization to design an undersampled seismic acquisition pattern that preserves the high quality of the reconstructed data. The sensing pattern is modeled as a deep binary layer to learn the location of receivers and sources for a particular seismic survey. Simultaneously, a deep neural network recovers the underlying removed data. Once the sensing pattern is designed, it can be used as a seismic acquisition geometry in an area that exhibits a similar geological setting to the training dataset of the E2E model. Extensive experiments were conducted on synthetic and real seismic data from different geological settings. The proposed design was compared with the traditional random, jittered, and uniform sensing schemes. The results validate that a guided design improves the quality of the reconstructed data by up to 4 and 2 dB in peak-signal-to-noise ratio for trace and shot gather reconstruction, respectively.

Hadamard single-pixel imaging (HSI) is a promising sensing approach for acquiring spectral images in the near-infrared spectrum with high spatial resolution and fast recovery times due to the efficient invertible properties of the Hadamard matrix. The potential of the HSI system is diminished because of the large number of required measurements which implies long acquisition times. Recent advances proposed optimizing the HSI sensing matrix structure based on a superpixels map estimated from a side-information acquisition of the scene, reducing the number of required measurements. However, these matrix designs are detached from the recovery task, which falls on a sub-optimal strategy. In this work, we proposed an adaptive end-to-end sensing methodology for the HSI sensing matrix design based on deep superpixels estimation by coupling the sensing and recovery of the near-infrared spectral images. Experimental results show the superiority of the proposed sensing methodology compared with state-of-art sensing design schemes.

Due to environmental and economic constraints inherent to seismic exploration, there are often missing shotpoints and receivers that degrade the resolution of the final seismic image. Hence sophisticated interpolation techniques are required for the recovery of dense and uniform spatial sampling. Recent approaches improve the interpolation by adopting robust models through denoisers. We introduce a 3D shot gather interpolation method that jointly considers a sparse prior and a regularization induced by a multichannel volumetric denoiser. The proposed volumetric regularization uses collaborative filters that perform denoising through transform-domain shrinkage of a group of similar seismic cubes extracted from a land seismic acquisition. This grouping and collaborative filtering paradigm exploit the local correlation present in each cube and the non-local correlation between different cubes. Experiments on theStratton 3D surveyshow that the proposed method can interpolate 3D shot gathers in an orthogonal seismic recording from a swath geometry, outperforming methods based on 2D denoisers and 5D seismic data reconstruction in terms of root mean square error and in the recovery of seismic reflections.

Physical and budget constraints often result in irregular sampling, which complicates accurate subsurface imaging. Pre-processing approaches, such as missing trace or shot interpolation, are typically employed to enhance seismic data in such cases. Recently, deep learning has been used to address the trace interpolation problem at the expense of large amounts of training data to adequately represent typical seismic events. Nonetheless, most research in this area has focused on trace reconstruction, with little attention having been devoted to shot interpolation. Furthermore, existing methods assume regularly spaced receivers/sources failing in approximating seismic data from real (irregular) surveys. This work presents a novel shot gather interpolation approach which uses a continuous coordinate-based representation of the acquired seismic wavefield parameterized by a neural network. The proposed unsupervised approach, which we call coordinate-based seismic interpolation (CoBSI), enables the prediction of specific seismic characteristics in irregular land surveys without using external data during neural network training. Experimental results on real and synthetic 3D data validate the ability of the proposed method to estimate continuous smooth seismic events in the time-space and frequency-wavenumber domains, improving sparsity or low-rank-based interpolation methods.

We consider a general spectral coexistence scenario, wherein the channels and transmit signals of both radar and communications systems are unknown at the receiver. In this dual-blind deconvolution (DBD) problem, a common receiver admits the multi-carrier wireless communications signal that is overlaid with the radar signal reflected-off multiple targets. When the radar receiver is not collocated with the transmitter, such as in passive or multistatic radars, the transmitted signal is also unknown apart from the target parameters. Similarly, apart from the transmitted messages, the communications channel may also be unknown in dynamic environments such as vehicular networks. As a result, the estimation of unknown target and communications parameters in a DBD scenario is highly challenging. In this work, we exploit the sparsity of the channel to solve DBD by casting it as an atomic norm minimization problem. Our theoretical analyses and numerical experiments demonstrate perfect recovery of continuous-valued range-time and Doppler velocities of multiple targets as well as delay-Doppler communications channel parameters using uniformly-spaced time samples in the dual-blind receiver.

Image captioning is a challenging task that connects two major artificial intelligence fields: computer vision and natural language processing. Image captioning models use traditional images to generate a natural language description of the scene. However, the scene could contain private information that we want to hide but still generate the captions. Inspired by the trend of jointly designing optics and algorithms, this paper addresses the problem of privacy-preserving scene captioning. Our approach promotes privacy preservation, by hiding the faces in the images, during the acquisition process with a designed refractive camera lens while extracting useful features to perform image captioning. The refractive lens and an image captioning deep network architecture are optimized end-to-end to generate descriptions directly from the blurred images. Simulations show that our privacy-preserving approach degrades private visual attributes (e.g., face detection fails with our distorted images) while achieving comparable captioning performance with traditional non-private methods on the COCO dataset.

Seismic acquisition is important in the exploration of the subsurface to find new petroleum fields, where a regular-spaced dense acquisition is critical to obtain high-quality seismic images. However, the acquisition costs and environmental impacts have motivated undersampled acquisition schemes, where several sensing points are removed to decrease the total seismic sources. After the undersampled measurements are acquired, a recovery algorithm reconstructs the missing seismic images (shots). The removed sources are currently selected using random sensing schemes, leading to suboptimal quality in the recovered seismic images. Thus, an optimal design of the removed sources is crucial as it determines the quality of the recovered shots. This work proposes an end-to-end optimization to jointly design undersampled seismic acquisition geometries while preserving the high-quality of the reconstructed data. The seismic acquisition geometry is modeled as a deep binary layer to learn the optimal sensing pattern, while a deep neural network is used to recover the underlying removed shots. Extensive simulations were carried out on a realistic-synthetic Foothills model. The results obtained on the reconstructed data validate that the proposed acquisition design outperforms the state-of-the-art random, uniform, and jitter sensing schemes in 4 dB.

Physical and budget constraints often result in inadequate sampling for accurate subsurface imaging. Preprocessing approaches, such as missing trace interpolation, are typically employed to enhance seismic data in such cases. The compressed sensing (CS) framework has been applied for modeling missing seismic data, which is estimated by sparsity-based computational algorithms. While existing work mainly focuses on recovering missing traces resulting from receiver subsampling, source subsampling has greater economical advantages, as sources are more expensive than receivers. Moreover, stronger image models different from sparsity have not been explored for source recovery. This work presents a consensus equilibrium (CE) approach to recover missing seismic shots, which enables to incorporate various regularization operators modeling different data priors. Simulation results from a real 3-D land seismic dataset demonstrate that the CE approach provides more accurate estimations of the linear and hyperbolic events in the recovered shots, compared with pure sparsity-based reconstructions.

The fusion of a low-spatial-and-high-spectral resolution hyperspectral image (HSI) with a high-spatial-and-low-spectral resolution multispectral image (MSI) allows synthesizing a high-resolution image (HRI), supporting remote sensing applications such as disaster management, material identification, and precision agriculture. Unlike existing variational methods using low-rank regularizations separately, we present an HSI-MSI fusion method promoting various low-rank regularizations jointly. Our method refines the HRI spatial and spectral correlations from the individual HSI and MSI data through the proper plug-and-play (PnP) of a nonlocal patch-based denoiser in the alternating direction method of multipliers (ADMM). Notably, we consider the nonlocal self-similarity, the spectral low-rank, and introduce a rank-one similarity prior. Furthermore, we demonstrate via an extensive empirical study that the rank-one similarity prior is an inherent characteristic of the HRI. Simulations over standard benchmark datasets show the effectiveness of the proposed HSI-MSI fusion outperforming state-of-the-art methods, particularly in recovering low-contrast areas.

Monitoring agriculture from satellite remote sensing data, such as multispectral images, has become a powerful tool since it has demonstrated a great potential for providing timely and accurate knowledge of crops. Detecting anomalies in time series of multispectral remote sensing images for crop monitoring is generally performed using a large sample of historical data at a pixel level. Conversely, this article presents a framework for anomaly detection (AD), localization, and classification that exploits the temporal information contained in a given season at a parcel level to detect and localize outliers using hidden Markov models (HMMs). Specifically, the AD part is based on the learning of HMM parameters associated with unlabeled normal data that are used in a second step to detect abnormal crop parcels referred to as anomalies. The learned HMM can also be used in time segments to temporally localize the anomalies affecting the crop parcels. The detected and localized anomalies are finally classified using a supervised classifier, e.g., based on support vector machines. The proposed framework is applicable to images partially covered by clouds and can handle a set of crop parcels acquired in the same season bypassing problems due to crop rotations. Numerical experiments are conducted on synthetic and real data, where the real data correspond to vegetation indices extracted from several multitemporal Sentinel-2 images of rapeseed crops. The proposed approach is compared to standard AD methods yielding better detection rates with the advantage of allowing anomalies to be localized and characterized.

Compressive covariance estimation has arisen as a class of techniques whose aim is to obtain second-order statistics of stochastic processes from compressive measurements. Recently, these methods have been used in various image processing and communications applications, including denoising, spectrum sensing, and compression. Notice that estimating the covariance matrix from compressive samples leads to ill-posed minimizations with severe performance loss at high compression rates. In this regard, a regularization term is typically aggregated to the cost function to consider prior information about a particular property of the covariance matrix. Hence, this paper proposes an algorithm based on the projected gradient method to recover low-rank or Toeplitz approximations of the covariance matrix from compressive measurements. The proposed algorithm divides the compressive measurements into data subsets projected onto different subspaces and accurately estimates the covariance matrix by solving a single optimization problem assuming that each data subset contains an approximation of the signal statistics. Furthermore, gradient filtering is included at every iteration of the proposed algorithm to minimize the estimation error. The error induced by the proposed splitting approach is analytically derived along with the convergence guarantees of the proposed method. The proposed algorithm estimates the covariance matrix of hyperspectral images from synthetic and real compressive samples. Extensive simulations show that the proposed algorithm can effectively recover the covariance matrix of hyperspectral images from compressive measurements with high compression ratios ( 8-15% approx) in noisy scenarios. Moreover, simulations and theoretical results show that the filtering step reduces the recovery error up to twice the number of eigenvectors. Finally, an optical implementation is proposed, and real measurements are used to validate the theoretical findings.

Recent advances in deep learning led to several algorithms for the accurate diagnosis of pneumonia from chest X-rays. However, these models require large training medical datasets, which are sparse, isolated, and generally private. Furthermore, these models in medical imaging are known to over-fit to a particular data domain source, i.e., these algorithms do not conserve the same accuracy when tested on a dataset from another medical center, mainly due to image distribution discrepancies. In this work, a domain adaptation and classification technique is proposed to overcome the over-fit challenges on a small dataset. This method uses a private-small dataset (target domain), a public-large labeled dataset from another medical center (source domain), and consists of three steps. First, it performs a data selection of the source domain's most representative images based on similarity constraints through principal component analysis subspaces. Second, the selected samples from the source domain are fit to the target distribution through an image to image translation based on a cycle-generative adversarial network. Finally, the target train dataset and the adapted images from the source dataset are used within a convolutional neural network to explore different settings to adjust the layers and perform the classification of the target test dataset. It is shown that fine-tuning a few specific layers together with the selected-adapted images increases the sorting accuracy while reducing the trainable parameters. The proposed approach achieved a notable increase in the target dataset's overall classification accuracy, reaching up to 97.78 % compared to 90.03 % by standard transfer learning.

Deep learning End-to-End (E2E) approaches have emerged as alternative optical design models, which jointly train the optical parameters of the sensing protocol, and the parameters of the deep neural network to achieve a specific task. This E2E model is particularly useful in the design of coding optical systems to address relevant constraints of the coded aperture (CA) design. To name, recent works address the binary constraint by incorporating regularization functions in the E2E optimization problem to promote binary value entries. How-ever, they do not consider other important CA assembling properties as the transmittance level, which plays a crucial role in implementable setups. Therefore, this work proposes two transmittance regularizers that jointly induce binary en-tries and adjust the transmittance level to be incorporated in an E2E approach. In particular, one of the regularizers allows achieving an exact value of the transmittance level when required for specific applications.

This paper presents a uniqueness result which states that a band- limited signal can be recovered from at least 3B measurements where B is the bandwidth from the radar ambiguity function (AF). This function is a two-dimensional mapping of the propagation delay and Doppler frequency. This formal model represents the distortion of a returned pulse due to the receiver matched filter. To estimate a time/band-limited signal from its radar AF, a trust region algorithm that minimizes a smoothed non-convex least-squares objective function is proposed. The method consists of two steps. First, we approximate the signal by an iterative spectral algorithm. Then, the attained initialization is refined based upon a sequence of gradient iterations. To the best of our knowledge this work is seminal in the sense of solving the radar phase retrieval problem for band-limited signals. Simulations results suggest that the proposed algorithm is able to estimate band-limited signals from its radar AF for both complete and incomplete radar cases. The AF is incomplete when only few shifts are considered. Numerical results show that the proposed algorithm estimates the signal with mean-square-error of 5 × 10-2for both complete and incomplete noisy cases.

The widespread use of always-connected digital cameras in our everyday life has led to increasing concerns about the users’ privacy and security. How to develop privacy- preserving computer vision systems? In particular, we want to prevent the camera from obtaining detailed visual data that may contain private information. However, we also want the camera to capture useful information to perform computer vision tasks. Inspired by the trend of jointly designing optics and algorithms, we tackle the problem of privacy-preserving human pose estimation by optimizing an optical encoder (hardware-level protection) with a software decoder (convolutional neural network) in an end-to- end framework. We introduce a visual privacy protection layer in our optical encoder that, parametrized appropriately, enables the optimization of the camera lens’s point spread function (PSF). We validate our approach with extensive simulations and a prototype camera. We show that our privacy-preserving deep optics approach successfully degrades or inhibits private attributes while maintaining important features to perform human pose estimation.

Compressive imaging using coded apertures (CA) is a powerful technique that can be used to recover depth, light fields, hyperspectral images and other quantities from a single snapshot. The performance of compressive imaging systems based on CAs mostly depends on two factors: the properties of the mask's attenuation pattern, that we refer to as "codification", and the computational techniques used to recover the quantity of interest from the coded snapshot. In this work, we introduce the idea of using time-varying CAs synchronized with spatially varying pixel shutters. We divide the exposure of a sensor into sub-exposures at the beginning of which the CA mask changes and at which the sensor's pixels are simultaneously and individually switched "on" or "off". This is a practically appealing codification as it does not introduce additional optical components other than the already present CA but uses a change in the pixel shutter that can be easily realized electronically. We show that our proposed time-multiplexed coded aperture (TMCA) can be optimized end to end and induces better coded snapshots enabling superior reconstructions in two different applications: compressive light field imaging and hyperspectral imaging. We demonstrate both in simulation and with real captures (taken with prototypes we built) that this codification outperforms the state-of-the-art compressive imaging systems by a large margin in those applications.

This paper presents a recovery method for compressive spectral imaging (CSI) based on the training-data independent deep image prior approach, where the prior information of the image is learned through the weights and the structure of the neural network. Specifically, we propose an interpretable architecture inspired in the linear mixture model for spectral images, where the image is decomposed as the product between a basis matrix, known as endmembers, and a coefficient matrix, known as abundances. These matrices are learned as the weights and the features of the proposed network, respectively. Simulations and experiments show that the proposed recovery method outperforms the state-of-the-art CSI recovery methods, even against training-data dependent methods. Furthermore, the architecture structure inspired by the linear mixture model gives interpretability of some outputs that can be useful for subsequent high-level image processing.

This paper presents an end-to-end (E2E) deep learning approach for the fusion of the data from two compressive spectral imaging systems, where a single neural network is developed to simultaneously optimize the sensing matrices and the decoder operator. The proposed E2E method models the sensing operator of the systems to fuse as optical layers, where the learnable parameters are the coded apertures of these CSI systems. These optical layers are then concatenated to an inspired unrolled deep neural network, where after training, these sensing matrices remain non-trainable along the optimization stages. Finally, a loss function is proposed. Simulation results show an improvement of the proposed coupled method compared with previous work and an enhancement due to the training of the sensing matrices and the proposed loss function.

Compressive covariance sampling (CCS) theory aims to recover the covariance matrix (CM) of a signal, instead of the signal itself, from a reduced set of random linear projections. Although several theoretical works demonstrate the CCS theory’s advantages in compressive spectral imaging tasks, a real optical implementation has no been proposed. Therefore, this paper proposes a compressive spectral sensing protocol for the dual-dispersive coded aperture spectral snapshot imager (DD-CASSI) to directly estimate the covariance matrix of the signal. Specifically, we propose a coded aperture design that allows recasting the vector sensing problem into matrix form, which enables to exploit the covariance matrix structure such as positive-semidefiniteness, low-rank, or Toeplitz. Additionally, a low-rank approximation of the image is reconstructed using a Principal Components Analysis (PCA) based method. In order to test the precision of the reconstruction, some spectral signatures of the image are captured with a spectrometer and compared with those obtained in the reconstruction using the covariance matrix. Results show the reconstructed spectrum is accurate with a spectral angle mapper (SAM) of less than 14°. RGB image composites of the spectral image also provide evidence of a correct color reconstruction.

In remote sensing, hyperspectral (HS) and multispectral (MS) image fusion have emerged as a synthesis tool to improve the data set resolution. However, conventional image fusion methods typically degrade the performance of the land cover classification. In this paper, a feature fusion method from HS and MS images for pixel-based classification is proposed. More precisely, the proposed method first extracts spatial features from the MS image using morphological profiles. Then, the feature fusion model assumes that both the extracted morphological profiles and the HS image can be described as a feature matrix lying in different subspaces. An algorithm based on combining alternating optimization (AO) and the alternating direction method of multipliers (ADMM) is developed to solve efficiently the feature fusion problem. Finally, extensive simulations were run to evaluate the performance of the proposed feature fusion approach for two data sets. In general, the proposed approach exhibits a competitive performance compared to other feature extraction methods.

Image fusion aims at estimating a high-resolution spectral image from a low-spatial-resolution hyperspectral image and a low-spectral-resolution multispectral image. In this regard, compressive spectral imaging (CSI) has emerged as an acquisition framework that captures the relevant information of spectral images using a reduced number of measurements. Recently, various image fusion methods from CSI measurements have been proposed. However, these methods exhibit high running times and face the challenging task of choosing sparsity-inducing bases. In this paper, a deep network under the algorithm unrolling approach is proposed for fusing spectral images from compressive measurements. This architecture, dubbed LADMM-Net, casts each iteration of a linearized version of the alternating direction method of multipliers into a processing layer whose concatenation deploys a deep network. The linearized approach enables obtaining fusion estimates without resorting to costly matrix inversions. Furthermore, this approach exploits the benefits of learnable transforms to estimate the image details included in both the auxiliary variable and the Lagrange multiplier. Finally, the performance of the proposed technique is evaluated on two spectral image databases and one dataset captured at the laboratory. Extensive simulations show that the proposed method outperforms the state-of-the-art approaches that fuse spectral images from compressive measurements.

Compressive spectral imaging (SI) (CSI) acquires few random projections of an SI reducing acquisition, storage, and, in some cases, processing costs. Then, this acquisition framework has been widely used in various tasks, such as target detection, video processing, and fusion. Particularly, compressive spectral image fusion (CSIF) aims at obtaining a high spatial-spectral resolution SI from two sets of compressed measurements: one from a hyperspectral image with a high-spectral low-spatial resolution, and one from a multispectral image with high-spatial low-spectral resolution. Most of the literature approaches include prior information, such as global low rank, smoothness, and sparsity, to solve the resulting ill-posed CSIF inverse problem. More recently, the high self-similarities exhibited by SIs have been successfully used to improve the performance of CSI inverse problems, including a nonlocal low-rank (NLLR) prior. However, to the best of our knowledge, this NLLR prior has not been implemented in the solution of the CSIF inverse problem. Therefore, this article formulates an approach that jointly includes the global low rank, the smoothness, and the NLLR priors to solve the CSIF inverse problem. The global low-rank prior is introduced with the linear mixture model that describes the SI as a linear combination of a set of few end-members to specific abundances. In this article, either the end-members are accurately estimated from the compressed measurements or initialized from a fast reconstruction of the hyperspectral image. Also, it assumes that the abundances preserve the smoothness and NLLR priors of the SI so that the fused image is obtained from the end-members and abundances that result when minimizing a cost function including the sum of two data fidelity terms and two regularizations: the smoothness and the NLLR. Simulations over three data sets show that the proposed approach increases the CSIF performance compared with literature approaches.

Single-pixel camera (SPC) captures encoded projections of the scene in a unique detector such that the number of compressive projections is lower than the size of the image. Traditionally, classification is not performed in the compressive domain because it is necessary to recover the underlying image before to classification. Based on the success of Deep learning (DL) in classification approaches, this paper proposes to classify images using compressive measurements of SPC. Furthermore, the proposed DL approach designs the binary sensing matrix in the SPC to improve the classification accuracy. In particular, a whole neural network is trained to learn the SPC sensing matrix, in the first layer, and extracts features from the single-pixel compressive measurements. The proposed approach overcomes two approaches of the state-of-the-art in terms of classification accuracy.

Spectral Videos (SV) are datasets containing spatial-spectral-and-temporal information of a moving scene and this kind of information have been successfully used in medicine, remote sensing, and military application. However, expensive acquisition processes and difficulties in equipment manufacture lead to low-resolution datasets. Therefore, super-resolution (SR) techniques have emerged as a processing tool that recovers a high-resolution dataset by expressing the measurements as compressed versions of the desired data. Furthermore, the Convolutional Sparse Coding (CSC) has been developed as a signal model that learns a dictionary directly from the target signal, improving the reconstruction quality. This work proposes to extend the CSC formulation to consider temporal correlations in SVs, exploiting the shifting invariance property of the CSC model. The simulation results show a PSNR improvement in up to 2.5dB with respect to the state-of-the-art methods, preserving the edges and textures of the spectral video frames.

In most imaging applications the spatial resolution is a concern of the systems, but increasing the resolution of the sensor increases substantially the implementation cost. One option with lower cost is the use of spatial light modulators, which allows improving the reconstructed image resolution by including a high-resolution codification. In this paper, we propose a reconstruction methodology that exploits the intrinsic information contained in low-resolution measurements generated by the use of high-resolution spatial light modulators and high-resolution approximations obtained via a CNN. Specifically, based on a high-resolution CNN approximation, an l2fidelity regularization term is introduced into a traditional l2-l1optimization problem. Finally, the simulations of the proposed l2-l1-l2reconstruction approach show a quality improvement in up to 3.7dB in averaged PSNR against the use of the traditional l2-l1approach.

Spectral image classification is an active research topic in remote sensing. In this sense, various multi-sensor spectral image fusion algorithms have been recently evaluated via pixel-based classification. In general, the sizes of multi-sensor images challenge the storing and processing capabilities of sensing systems. Therefore, different image fusion algorithms from measurements captured by multi-resolution compressive spectral imaging (CSI) sensors have been proposed. However, the computational costs for reconstructing and fusing spectral images from compressive measurements are high, and these approaches do not consider the huge amount of information embedded in acquired data. In this article, a spectral image classification scheme from multi-sensor CSI projections is developed. Specifically, this scheme includes a feature extraction procedure that exploits the fact that CSI data contain relevant information of the spectral image, and therefore, low-dimensional features can be obtained from measurements. Furthermore, a fusion model is presented to combine the information of the extracted features with the aim of estimating high-resolution classification attributes. Then, a pixel-based classifier is applied to the fused features with the goal of labeling the corresponding high-resolution spectral image. The performance of the proposed classification scheme is compared to other methods on the Salinas Valley data set for different supervised classifiers and various downsampling settings. Extensive simulations on the Pavia University data set are also shown, where the proposed method outperforms other classification approaches that reconstruct and fuse from compressive measurements. Finally, the effectiveness of the proposed classification approach is validated in real multi-sensor data.

Super-resolution phase retrieval is an inverse problem that appears in diffractive optical imaging (DOI) and consists in estimating a high-resolution image from low-resolution phaseless measurements. DOI has three diffraction zones where the data can be acquired, known as near, middle, and far fields. Recent works have studied super-resolution phase retrieval under a setup that records coded diffraction patterns at the near and far fields. However, the attainable resolution of the image is mainly governed by the sensor characteristics, whose cost increases in proportion to the resolution. Also, these methodologies lack theoretical analysis. Hence, this work derives super-resolution models from low-resolution coded phaseless measurements at any diffraction zone that in contrast to prior contributions, the attainable resolution of the image is determined by the resolution of the coded aperture. For the proposed models, the existence of a unique solution (up to a global unimodular constant) is guaranteed with high probability, which can be increased by designing the coded aperture. Therefore, a strategy that designs the spatial distribution of the coded aperture is developed. Additionally, a super-resolution phase retrieval algorithm that minimizes a smoothed nonconvex least-squares objective function is proposed. The method first approximates the image by a spectral algorithm, which is then refined based upon a sequence of alternate steps. Simulation results show that the proposed algorithm overcomes state-of-the-art methods in reconstructing the high-resolution image. In addition, the reconstruction quality using designed coded apertures is higher than that of the non-designed ensembles.

Compressive spectral imaging (CSI) sensors allow the acquisition of spatial and spectral data using a set of coded projections. The single pixel camera (SPC) is a low-cost CSI architecture capable of sensing high-resolution spectral images, whose potential is diminished by its slow acquisition time due to the large number of required projections. To partially alleviate this issue, dual arm optical systems have been designed such that side information of the scene is captured to guide the reconstruction process. To fully exploit the capabilities of the dual system, the SPC sensing matrix, or equivalently the coding patterns; should be properly designed. Therefore, this work proposes an optimized sensing matrix design for the SPC based on a super-pixel map of the scene, obtained from the side information, such that the number of projections is drastically reduced while the reconstruction quality is improved. Indeed, theoretical analysis based on the restricted isometry property indicates that the error of the reconstruction vanishes when the SPC uses the designed sensing matrix. Simulation and experimental results show that the proposed sensing matrix design improves the reconstruction quality. Specifically, the proposed approach improves image quality in up to 15dB compared with the state of the art sensing matrix designs. Moreover, a fast multi-resolution reconstruction approach is proposed based on the designed matrix, which reduces computation time by two orders of magnitude and does not require an iterative process.

Phase retrieval is an inverse problem that consists in estimating a scene from diffraction intensities. This problem appears in optical imaging, which has three main diffraction zones where the measurements can be acquired, i.e., near, middle and far. Recent works have theoretically solved this inverse problem for the far zone, creating redundancy in the measurement process by including a coded aperture, which allows to modulate the scene and acquire coded diffraction patterns (CDP). However, in the state-of-the-art, the PR problem has not been theoretically studied for CDP at the near and middle zones. Moreover, the structure of the coded aperture is selected at random, leading to suboptimal estimations. Indeed, some of the coding elements employed in the literature are unfeasible because they increase the power of the scene in the modulation process. This paper provides theoretical guarantees for the recovery of a scene from CDP acquired at the three diffraction zones using admissible modulations. Based on the theoretical results, it is established that the image reconstruction quality directly depends on the coded aperture structure; therefore, a design strategy is proposed. In fact, when the scene can be sparsely represented in some basis, its support can be better estimated for a suitable choice of the coding elements in the modulation process. Experimental results show that the scene is successfully recovered by using designed coded apertures with up to 40% less measurements compared to non-designed ensembles.

Spectral videos contain highly redundant information across spatial, spectral and temporal axes which can be exploited through a temporal-data-learned sparsifying basis. However, in compressive spectral video acquisition, tackling dictionary learning is time-consuming since it increases the computational complexity and presents drawbacks for realtime processing, where offline learning is required. This paper introduces a tensor-decomposition learning (TenDL) framework for simultaneous online sparsifying and recovering the spatialspectral- temporal information of a spectral video performed on several temporal superpixels (TSP-TenDL) for time processing reduction. The framework is composed of two main stages: preprocessing and joint estimation. The preprocessing stage includes a strategy for a grayscale approximation of the video to provide a suitable initialization of the sparsifying basis to be learned. To fully exploit the high signal correlation, a set of temporal superpixels is estimated from the grayscale approximation, reducing the reconstruction time of the large-scale data. Then, the outcome of the first stage is used to estimate the basis and the signal coefficients, where an optimization problem is solved to learn and reconstruct the basis and the signal, respectively, following a block-descent coordinate strategy. The proposed approach is compared from simulations with an offline-learned based method, traditional matrix-based recovery algorithms and the tensor-based recovery, the two latter using a fixed basis, where TSP-TenDL exhibits higher image quality results and lower computation time. Specifically, our methodology gains up to 7dB in terms of PSNR and a speedup of up to 6.6× compared with state-of-the-art counterparts.

In recent years there has been an increasing interest in sensing devices that capture multidimensional information such as the spectral light field (SLF) images, which are 5-dimensional (5D) representations of a scene including 2D spatial, 2D angular and 1D spectral information. Spatio-spectral and angular information plays an important role in modern applications spanning from microscopy to computer vision. However, SLF sensors use expensive beam-splitters or cameras arrays placed in tandem, which split the sensing problem in two time consuming and independent tasks: spectral and light field imaging tasks. This work proposes a compressive spectral light field imaging architecture that builds on the principles of the compressive imaging framework, to capture multiplexed representations of the multidimensional information, so that, less measurements are required to capture the SLF data cube. Alongside, we propose a computational algorithm to recover the 5D information from the compressed measurements, exploiting the inherent high correlations within the SLF by treating them as 3D tensors. Furthermore, exploiting the geometry properties of the proposed optical architecture, the Tucker decomposition is applied to the set of compressed measurements, so that, an ad-hoc dictionary-like image representation basis is calculated online. This in turn, entails a more accurate reconstruction of the SLF since the dictionary fits the specific characteristics of the image itself. We demonstrate through simulations over three SLF datasets captured in our laboratory, and an experimental proof-of-concept implementation, that the proposed compressive imaging device together with the proposed computational algorithm represent an efficient alternative to capture SLF, compared to conventional methods that employ either side-information or multiple sensors. Also, we show that the tensor-based proposed algorithm exhibits a lower computational complexity than the matrix-based state of the art counterparts, thus enabling fast processing of multidimensional images.

Compressive spectral imaging (CSI) is a framework that captures coded-and-multiplexed low-dimensional projections of spectral data-cubes. In general, the sensing process in many CSI architectures is described using binary matrices, so-called sensing/projection matrices, whose elements can be either random or designed. However, some characteristics of the spectral data, such as the ℓ2-norm or the second moment statistics, can be lost when this dimensionality reduction is performed. Similarly, principal component analysis (PCA) is a data dimensionality reduction technique that minimizes the least-squared error between the spectral data and its low-dimensional projection, but preserving its structure or variance. Thus, PCA can be used to guide the CSI acquisition process by designing the binary sensing matrix. Nonetheless, PCA requires to know the spectral image a-priori, and also, its associated projection matrix is not binary, as required by CSI optical architectures. Therefore, in this paper, an algorithm to design CSI sensing matrices by exploiting the structure-preserving property of the PCA projection is proposed. First, a set of compressive measurements obtained with random sensing matrices is used to rapidly estimate the covariance matrix associated with the spectral data. Then, a new sensing matrix is designed by solving a non-convex optimization problem that finds a set of binary vectors that approximate the principal components of the covariance matrix, thus maximizing the explanation of the data variance. Experimental results show an improvement of up to 3 dB in image reconstruction quality, in terms of the peak signal to noise ratio (PSNR), when the binary PCA-based sensing matrices are used and compared with conventional random sensing matrices and state-of-art designed matrices based on PCA.

Coded aperture compressive spectral imagers allow sensing a three-dimensional (3D) data cube by using two-dimensional (2D) projections of the coded and spectrally dispersed source. The traditional block-unblock coded apertures have been recently replaced by patterned optical filter arrays, allowing to modulate the spatial and spectral information. The real implementation of these patterned or "colored" coded apertures in terms of cost and complexity, directly depends on the number of filters to be used as well as the number of snapshots to be captured. This paper introduces a coded aperture optimization having in consideration these restrictions, the final design obtained is a moving colored coded aperture, which improves the reconstruction quality of the data cube and is physically implementable. Simulations show the accuracy and performance achieved with the proposed approach yielding up to 3 dB gain in PSNR over the current literature designs.

Compressive spectral imaging (CSI) acquires coded projections of a spectral image by performing a modulation of the data cube followed by a spectral-wise integration. To avoid the spectral image reconstruction procedure, this paper proposes a classification approach that extracts features directly from multi-sensor CSI measurements. Particularly, the proposed method obtains the features by considering the spectral information extracted from Hyperspectral CSI measurements, and the local spatial information extracted by clustering the Multispectral CSI measurements using a superpixel algorithm. This approach is evaluated on Pavia University and Salinas Valley datasets. Extensive simulations show that considering the local spatial information boosts the overall accuracy up to 3% in comparison with traditional approaches that only uses the spectral information. Furthermore, the computation time of the approach that reconstructs, fuses and classifies takes approximately 87.43 [s], while classifying directly from multi-sensor compressive measurements takes only 0.74 [s], achieving similar classification results.

Compressive spectral imaging (CSI) has emerged as an alternative acquisition framework that simultaneously senses and compresses spectral images. In this context, the spectral image classification from CSI compressive measurements has become a challenging task since the feature extraction stage usually requires reconstructing the spectral image. Moreover, most approaches do not consider multi-sensor compressive measurements. In this paper, an approach for fusing features obtained from multi-sensor compressive measurements is proposed for spectral image classification. To this end, linear models describing low-resolution features as degraded versions of the high-resolution features are developed. Furthermore, an inverse problem is formulated aiming at estimating high-resolution features including both a sparsity-inducing term and a total variation (TV) regularization term to exploit the correlation between neighboring pixels of the spectral image, and therefore, to improve the performance of pixel-based classifiers. An algorithm based on the alternating direction method of multipliers (ADMM) is described for solving the fusion problem. The proposed feature fusion approach is tested for two CSI architectures: three-dimensional coded aperture snapshot spectral imaging (3D-CASSI) and colored CASSI (C-CASSI). Extensive simulations on various spectral image data sets show that the proposed approach outperforms other classification approaches under different performance criteria.

Compressive sensing enables efficient acquisition of hyperspectral images (HSIs) by assuming high redundancy on natural scenes. Several reconstruction algorithms have been proposed to retrieve the underlying image, and most of them take advantage of the spatial and spectral correlations. However, reconstruction may not be necessary in certain applications such as land cover classification. Instead of knowing the full image, researchers are interested in features that could be extracted directly from the compressed measurements, which provide high inference capabilities. Low-rank (LR) matrix approximation has been widely used in feature extraction (FE), because it reduces the data dimension and computational cost. Therefore, in this paper, compressive hyperspectral imaging and FE are combined in a framework for HSI classification using an LR matrix approximation model. In the proposed framework, the compressed measurements are acquired from a single-pixel spectrometer. Instead of using the traditional high-complexity reconstruction model, an LR matrix factorization problem is formulated. The LR problem maximizes the posterior distribution with respect to the feature space and coefficients, and it is numerically solved based on an alternating optimization strategy. By incorporating spatial information, the numerical procedure minimizes the total variational of the feature coefficients subject to an orthogonality constraint for the feature space. Experiments on real HSIs show that the proposed approach can provide equally competitive classification results when compared to the traditional approach that performs FE and classification on the recovered images.

In the past years, one common way of enhancing the spatial resolution of a hyperspectral (HS) image has been to fuse it with complementary information coming from multispectral (MS) or panchromatic images. This paper proposes a new method for reconstructing a high-spatial, high-spectral image from measurements acquired after compressed sensing by multiple sensors of different spectral ranges and spatial resolutions, with specific attention to HS and MS compressed images. To solve this problem, we introduce a fusion model based on the linear spectral unmixing model classically used for HS images and investigate an optimization algorithm based on a block coordinate descent strategy. The nonnegative and sum-to-one constraints resulting from the intrinsic physical properties of abundances as well as a total variation penalization are used to regularize this ill-posed inverse problem. Simulation results conducted on realistic compressed HS and MS images show that the proposed algorithm can provide fusion results that are very close to those obtained with uncompressed images, with the advantage of using a significantly reduced number of measurements.

Compressive spectral video sensing (CSVS) systems obtain spatial, spectral, and temporal information of a dynamic scene through the encoding of the incoming light rays by using a temporal-static coded aperture (CA). CSVS systems use CAs with binary entries spatially distributed at random. The random spatial encoding of the binary CAs entails a poor quality in the reconstructed images even though the CSVS sensing matrix is incoherent with the sparse representation basis. In addition, since some pixels are totally blocked, information such as object motion is missed over time. This paper substitutes the temporal-static binary coded apertures by a richer spatio-spectro-temporal encoding based on selectable color filters, named temporal colored coded apertures (T-CCA). The spatial, spectral, and time distributions of the T-CCAs are optimized by better satisfying the restricted isometry property (RIP) of the CSVS system. The RIP-optimized T-CCAs lead to spatio-spectral-time structures that tend to sense more uniformly the spatial, spectral, and temporal dimensions. An algorithm for optimally designing the T-CCAs is developed. In addition, a regularization term based on the scene motion is included in the inverse problem leading to a better quality of the reconstructed images. Computational experiments using four different spectral videos show an improvement of up to 6 dB in terms of peak signal-to-noise ratio of the reconstructed images by using the proposed inverse problem and the T-CCA patterns compared with the binary CAs and random and image-optimized CCA patterns.

Compressive spectral imagers reduce the number of sampled pixels by coding and combining the spectral information. However, sampling compressed information with simultaneous high spatial and high spectral resolution demands expensive high-resolution sensors. This work introduces a model allowing data from high spatial/low spectral and low spatial/high spectral resolution compressive sensors to be fused. Based on this model, the compressive fusion process is formulated as an inverse problem that minimizes an objective function defined as the sum of a quadratic data fidelity term and smoothness and sparsity regularization penalties. The parameters of the different sensors are optimized and the choice of an appropriate regularization is studied in order to improve the quality of the high resolution reconstructed images. Simulation results conducted on synthetic and real data, with different CS imagers, allow the quality of the proposed fusion method to be appreciated.

Phase retrieval is a kind of ill-posed inverse problem, which is present in various applications, such as optics, astronomical imaging, and X-ray crystallography. Mathematically this inverse problem consists on recovering an unknown signal x ∈ Rn/Cnfrom a set of absolute square projections yk= |(ak, x)|2, k = 1,··· , m, where ak are the sampling vectors. However, the square absolute function is in general nonconvex and non-differentiable, which are desired properties in order to solve the problem, when traditional convex optimization algorithms are used. Therefore, this paper introduces a special differentiable function, known as smoothing function, in order to solve the phase retrieval problem by using the smoothing projected gradient (SPG) method. Moreover, to accelerate the convergence of this algorithm, this paper uses a nonlinear conjugate gradient method applied to the smoothing function as the search direction. Simulation results are provided to validate its efficiency on existing algorithms for phase retrieval. It is shown that compared with recently developed algorithms, the proposed method is able to accelerate the convergence.

Massive amounts of data in spectral imagery increase acquisition, storing and processing costs. Compressive spectral imaging (CSI) methods allow the reconstruction of spatial and spectral information from a small set of random projections. The single pixel camera is a low cost optical architecture which enables the compressive acquisition of spectral images. Traditional CSI reconstruction methods obtain a sparse approximation of the underlying spatial and spectral information, however the complexity of these algorithms increases in proportion to the dimensionality of the data. This work proposes a multiresolution (MR) CSI reconstruction approach from single pixel camera measurements that exploits spectral similarities between pixels to group them in super-pixels such that the total number of unknowns in the inverse problem is reduced. Specifically, two different types of super-pixels are considered: rectangular and irregular structures. Simulation and experimental results show that the proposed MR scheme improves reconstruction quality in up to 6dB of PSNR and reconstruction time in up to 90% with respect to the traditional full resolution reconstructions.

Recently, an important set of high dimensional signals (HDS) applications has successfully implemented compressive sensing (CS) sensors in which their efficiency depends on physical elements that perform a binary codification over the HDS. The structure of the binary codification is crucial as it determines the HDS sensing matrices. For a correct reconstruction, this class of matrices drastically differs from the dense or i.i.d. assumptions usually made in CS. Therefore, current CS matrix design algorithms are impractical. This paper proposes a novel strategy to design structured, sparse, and binary HDS measurement matrices based on promoting linear independence between rows by minimizing the number of its zero singular values. The design constraints lead to keep uniform both, the number of non-zero elements per row and also the number of non-zero elements per column. An algorithm based on an optimal selection of non-zero entries positions is developed to implement this strategy. Simulations show that the proposed optimization improves the quality of the reconstructed HDS in up to 8 dB of PSNR compared with non-optimized matrices.

This paper introduces a hierarchical Bayesian model for the reconstruction of hyperspectral images using compressed sensing measurements. This model exploits known properties of natural images, promoting the recovered image to be sparse on a selected basis and smooth in the image domain. The posterior distribution of this model is too complex to derive closed form expressions for the estimators of its parameters. Therefore, an MCMC method is investigated to sample this posterior distribution. The resulting samples are used to estimate the unknown model parameters and hyperparameters in an unsupervised framework. The results obtained on real data illustrate the improvement in reconstruction quality when compared to some existing techniques.

X-ray crystallography is an experimental technique to estimate the 3D atomic positions of the elements present in a crystal. This technique constructs the 3D structure from the phase of diffracted and patterned X-rays (DPX). Multiple intensity DPX measurements are acquired to solve the phase retrieval problem. The feasibility of implementing this technique depends on solving the phase retrieval problem using expensive multiple valued patterns and the Truncated Wirtinger Flow Algorithm. This paper presents a Stochastic Truncated Wirtinger Flow Algorithm (STWF) which solves the phase retrieval problem based on DPX measurements low-cost boolean block-unblock coded apertures. Several simulations are realized to demonstrate the convergence of the STWF algorithm and the optimal parameters of the boolean coded apertures. The results indicate that given the DPX measurements, the quality of reconstructed phase images using STWF attained up 24:63dB of PSNR.

Compressive spectral imagers drastically reduce the number of sampled pixels by performing linear combinations of coded spectral information. However, compressing information with simultaneously high spatial and high spectral resolutions demands expensive high-resolution sensors. This work introduces a model allowing compressive data from high spatial/low spectral and low spatial/high spectral resolution sensors to be fused. The sensing matrix of this model is designed carefully to be incoherent with the dictionary associated with the unknown image. Based on this model, the compressive fusion process is formulated as an inverse problem that minimizes an objective function defined as the sum of a quadratic data fidelity term and smoothness and sparsity regularization penalties.

In this work, a compression strategy for Focal Plane Array Measurements (FPA_M) in Compressive Spectral Imaging (CSI) is presented. The strategy is applied to four CSI architectures. The implementation consists of two stages: transformation and coding. The transformation stage is performed according to the FPA_M structure. Finally, an arithmetic coding is applied.

Compressive spectral imaging (CSI) has demonstrated to be a feasible technique for capturing the 3D spatio spectral information of a scene through less measurements than the Nyquist rate. The coded aperture snapshot spectral imaging (CAS-SI) is an example of a CSI optical architecture, which has been proposed to work on the visible electromagnetic spectrum. Due to the rich information contained in the infrared spectrum, in this paper, we mathematically model and demonstrate the implementation of a broadband CASSI system covering the visible and the near infrared spectra between 448 nm to 1436 nm. Particularly, we developed a Digital Micromirror Device-based CSI system, which implements dual-band CS measurement processes of 3D spatio-spectral scenes.

Summary form only given. The multi-shot Coded Aperture Snapshot Spectral Imaging system (CASSI) is an imaging architecture that senses the spectral imaging information of a 3D cube using a 2D focal plane array snapshot. Modeled as the summation of coded and shifted versions of different spectral voxels, the compressive CASSI measurements are difficult to be further compressed due to little correlation and redundancy. This paper is dedicated to retaining the correlations in the original data cube by an embedded transform coding on the compressive measurements. Through statistical modeling, the transformed measurements prove to be more approximate to the distribution of original spectral images than the compressive measurements. Meanwhile, bit-plane coding for the transformed measurements is applied by taking advantage of the known code aperture. The significant performance of this embedded coder on random compressive measurements is evaluated for different number of snapshots.

Compressive spectral imagers have gained popularity recently due to their ability to sense a three-dimensional (3D) data cube with just a few two dimensional (2D) coded aperture projection snapshots. The coded apertures are realized by digital micromirror devices (DMD) which often do not match the pitch resolution of the focal plane array (FPA). This paper introduces the forward model and associated reconstruction algorithm for such mismatched spectral imagers, without the loss of spectral and spatial resolution. Simulations show the improvements in the reconstructions achieved with the proposed approach yielding up to 12 dB gain in PSNR with respect to traditional.

Compressive spectral imaging captures the spatial and spectral information of a scene using a set of two-dimensional random projections. Compressed sensing reconstruction algorithms are then used to recover the underlying three-dimensional source. This work presents a new generation of devices that attain compressive spectral image measurements by means of a colored-patterned detector and a dispersive element. Simulations show that these new generation devices can recover spectral scenes with up to 5 dB gain in PSNR with respect to traditional Coded Aperture Snapshot Spectral Imaging (CASSI) systems.

Compressive spectral imaging (CSI) systems capture the 3D spatio-spectral information of a scene by measuring 2D focal plane array (FPA) coded projections. A reconstruction algorithm exploiting the sparsity of the signal is then used to recover the underlying hyperspectral scene. CSI systems use a set of binary coded apertures, commonly realized through photomasks, to modulate the spatial characteristics of the scene. The reconstruction image quality in CSI is determined by the design of a 2D coded aperture binary set which block or unblock light from the scene onto the detector. This work extends the framework of CSI by replacing the traditional block-unblock photomasks by colored coded apertures which modulate the source not only spatially but spectrally as well. Simulations show a significant improvement in the quality of spectral image reconstructions.

Coded Aperture Snapshot Spectral Imaging (CASSI) systems capture the spatial and spectral information of a scene by measuring 2D coded projections on a focal plane array (FPA). Compressed sensing reconstruction algorithms are then used to recover the underlying spectral data cube. The quality of the reconstructions in CASSI is determined by the design of a set of block-unblock coded apertures. In this work, the block-unblock coded apertures in CASSI are replaced by colored coded apertures. The Restricted Isometry Property (RIP) of the colored CASSI is developed and the structure of the colored coded apertures is designed such that the RIP is better satisfied. Simulations show significant gain in the quality of reconstructions for the optimized colored coded apertures over that attained by traditional block-unblock coded apertures.

Traditional hyperspectral imaging sensors acquire high-dimensional data that are used for the discrimination of objects and features in a scene. Recently, a novel architecture known as the coded-aperture snapshot spectral imaging (CASSI) system has been developed for the acquisition of compressive spectral image data with just a few coded focal plane array measurements. This paper focuses on developing a classification approach with hyperspectral images directly from CASSI compressive measurements, without first reconstructing the full data cube. The proposed classification method uses the compressive measurements to find the sparse vector representation of the test pixel in a given training dictionary. The estimated sparse vector is obtained by solving a sparsity-constrained optimization problem and is then used to directly determine the class of the unknown pixel. The performance of the proposed classifier is improved by taking optimal CASSI compressive measurements obtained when optimal coded apertures are used in the optical system. The set of optimal coded apertures is designed such that the CASSI sensing matrix satisfies a restricted isometry property with high probability. Several simulations illustrate the performance of the proposed classifier using optimal coded apertures and the gain in the classification accuracy obtained over using traditional aperture codes in CASSI.

Compressive spectral imaging (CSI) senses the spatio-spectral information of a scene by measuring 2D coded projections on a focal plane array. A ℓ1-norm-based optimization algorithm is then used to recover the underlying discretized spectral image. The coded aperture snapshot spectral imager (CASSI) is an architecture realizing CSI where the reconstruction image quality relies on the design of a 2D set of binary coded apertures which block-unblock the light from the scene. This paper extends the compressive capabilities of CASSI by replacing the traditional blocking-unblocking coded apertures by a set of colored coded apertures. The colored coded apertures are optimized such that the number of projections is minimized while the quality of reconstruction is maximized. The optimal design of the colored coded apertures aims to better satisfy the restricted isometry property in CASSI. The optimal designs are compared with random colored coded aperture patterns and with the traditional blocking-unblocking coded apertures. Extensive simulations show the improvement in reconstruction PSNR attained by the optimal colored coded apertures designs.

Coded aperture snapshot spectral imaging systems (CASSI) measure the 3D spatio-spectral information of a scene using several compressive 2D focal plane array (FPA) snapshots. The image reconstruction algorithms utilized in CASSI use a first-order approximation of the underlying analog sensing phenomena. A calibration method is then used to compensate for the coarse approximation - an approach not well suited for multishot CASSI systems. This paper develops a more accurate computational model for CASSI which provides a higher quality of image reconstruction. Several simulations are shown to illustrate the performance improvement attained by the new model.

A new code aperture design framework for multiframe code aperture snapshot spectral imaging (CASSI) system is presented. It aims at the optimization of code aperture sets such that a group of compressive spectral measurements is constructed, each with information from a specific subset of bands. A matrix representation of CASSI is introduced that permits the optimization of spectrally selective code aperture sets. Furthermore, each code aperture set forms a matrix such that rank minimization is used to reduce the number of CASSI shots needed. Conditions for the code apertures are identified such that a restricted isometry property in the CASSI compressive measurements is satisfied with higher probability. Simulations show higher quality of spectral image reconstruction than that attained by systems using Hadamard or random code aperture sets.

This paper addresses the problem of video anomaly recovery from a sequence of spectrally compressed video frames. Analysis of anomalies occurring in both time and spectrum is important in video surveillance applications. We present a methodology for the recovery of anomalies such as moving objects and their spectral signatures from spectrally compressed video. The spectrally compressed video frames are obtained by using a Coded Aperture Snapshot Spectral Imaging (CASSI) system. The CASSI system encodes a 3-D data cube containing both 2-D spatial information and spectral information in a single 2-D measurement. In the proposed methodology, we use the spectrally compressed video as columns of a large data matrix Q. Principal Component Pursuit (PCP) is then used to decompose Q into the stationary background and a sparse matrix capturing the anomalies in the foreground. The sparse matrix is then used jointly with Q to recover the spectral information of the objects of interest. An example for the recovery of video anomalies in a 3-channel spectral video system (RGB) is presented.