Neural networks are frequently used in medical diagnosis. However, due to their black-box nature, model explainers are used to help clinicians understand better and trust model outputs. This paper introduces an explainer method for classifying Retinopathy of Prematurity (ROP) from fundus images. Previous methods fail to generate explanations that preserve input image structures such as smoothness and sparsity. We introduce Sparse and Smooth Explainer (SSplain), a method that generates pixel-wise explanations while preserving image structures by enforcing smoothness and sparsity. This results in realistic explanations to enhance the understanding of the given black-box model. To achieve this goal, we define an optimization problem with combinatorial constraints and solve it using the Alternating Direction Method of Multipliers (ADMM). Experimental results show that SSplain outperforms commonly used explainers in terms of both post-hoc accuracy and smoothness analyses. Additionally, SSplain identifies features that are consistent with domain-understandable features that clinicians consider as discriminative factors for ROP. We also show SSplain’s generalization by applying it to additional publicly available datasets. Code is available at https://github.com/neu-spiral/SSplain.

One of the main challenges of federated learning (FL) is handling non-independent and identically distributed (non-IID) client data, which may occur in practice due to unbalanced datasets and use of different data sources across clients. Knowledge sharing and model personalization are key strategies for addressing this issue. Clustered federated learning is a class of FL methods that groups clients that observe similarly distributed data into clusters, such that every client is typically associated with one data distribution and participates in training a model for that distribution along their cluster peers. In this paper, we present a unified Bayesian framework for clustered FL which associates clients to clusters. Then we propose several practical algorithms to handle the, otherwise growing, data associations in a way that trades off performance and computational complexity. This work provides insights on client-cluster associations and enables client knowledge sharing in new ways. The proposed framework circumvents the need for unique client-cluster associations, which is seen to increase the performance of the resulting models in a variety of experiments.

Learning dynamics governing physical and spatiotemporal processes is a challenging problem, especially in scenarios where states are partially measured. In this work, we tackle the problem of learning dynamics governing these systems when parts of the system's states are not measured, specifically when the dynamics generating the non-measured states are unknown. Inspired by state estimation theory and Physics Informed Neural ODEs, we present a sequential optimization framework in which dynamics governing unmeasured processes can be learned. We demonstrate the performance of the proposed approach leveraging numerical simulations and a real dataset extracted from an electro-mechanical positioning system. We show how the underlying equations fit into our formalism and demonstrate the improved performance of the proposed method when compared with baselines.

Brain-Computer Interfaces (BCIs) help people with severe speech and motor disabilities communicate and interact with their environment using neural activity. This work focuses on the Rapid Serial Visual Presentation (RSVP) paradigm of BCIs using noninvasive electroencephalography (EEG). The RSVP typing task is a recursive task with multiple sequences, where users see only a subset of symbols in each sequence. Extensive research has been conducted to improve classification in the RSVP typing task, achieving fast classification. However, these methods struggle to achieve high accuracy and do not consider the typing mechanism in the learning procedure. They apply binary target and non-target classification without including recursive training. To improve performance in the classification of symbols while controlling the classification speed, we incorporate the typing setup into training by proposing a Partially Observable Markov Decision Process (POMDP) approach. To the best of our knowledge, this is the first work to formulate the RSVP typing task as a POMDP for recursive classification. Experiments show that the proposed approach, MarkovType, results in a more accurate typing system compared to competitors. Additionally, our experiments demonstrate that while there is a trade-off between accuracy and speed, MarkovType achieves the optimal balance between these factors compared to other methods.

State-space estimation and tracking rely on accurate dynamical models to perform well. However, obtaining an accurate dynamical model for complex scenarios or adapting to changes in the system poses challenges to the estimation process. Recently, augmented physics-based models (APBMs) appear as an appealing strategy to cope with these challenges where the composition of a small and adaptive neural network with known physics-based models (PBM) is learned on the fly following an augmented state-space estimation approach. A major issue when introducing data-driven components in such a scenario is the danger of compromising the meaning (or interpretability) of estimated states. In this work, we propose a novel constrained estimation strategy that constrains the APBM dynamics close to the PBM. The novel state-space constrained approach leads to more flexible ways to impose constraints than the traditional APBM approach. Our experiments with a radar-tracking scenario demonstrate different aspects of the proposed approach and the trade-offs inherent in the imposed constraints.

Recent advances in the theory of Neural Operators (NOs) have enabled fast and accurate computation of the solutions to complex systems described by partial differential equations (PDEs). Despite their great success, current NO-based solutions face important challenges when dealing with spatio-temporal PDEs over long time scales. Specifically, the current theory of NOs does not present a systematic framework to perform data assimilation and efficiently correct the evolution of PDE solutions over time based on sparsely sampled noisy measurements. In this paper, we propose a learning-based state-space approach to compute the solution operators to infinite-dimensional semilinear PDEs. Exploiting the structure of semilinear PDEs and the theory of nonlinear observers in function spaces, we develop a flexible recursive method that allows for both prediction and data assimilation by combining prediction and correction operations. The proposed framework is capable of producing fast and accurate predictions over long time horizons, dealing with irregularly sampled noisy measurements to correct the solution, and benefits from the decoupling between the spatial and temporal dynamics of this class of PDEs. We show through experiments on the Kuramoto-Sivashinsky, Navier-Stokes and Korteweg-de Vries equations that the proposed model is robust to noise and can leverage arbitrary amounts of measurements to correct its prediction over a long time horizon with little computational overhead.

Deep learning-based frameworks have been recently applied to hyperspectral umixing due to their flexibility and powerful representation capabilities. However, such techniques either use black-box models which are not physically interpretable, or fail to address the non-idealities of the unmixing problem. In this paper, we propose a physically interpretable deep learning method for hyperspectral unmixing accounting for nonlinearity and the variability of the endmembers. The proposed method is based on a probabilistic variational deep learning framework which employs semi-supervised disentanglement learning to properly separate the abundances and endmembers. A self-supervised strategy is used to generate labeled training data, and the model is learned end-to-end using stochastic backpropagation. Experimental results on both synthetic and real datasets illustrate the performance of the proposed method compared to state-of-the-art algorithms.

Monitoring interferences to satellite-based navigation systems is of paramount importance in order to reliably operate critical infrastructures, navigation systems, and a variety of applications relying on satellite-based positioning. This paper investigates the use of crowd-sourced data to achieve such detection and monitoring at a central node that receives the data from an arbitrary number of agents in an area of interest. Under ideal conditions, the pathloss model is used to compute the Cramer-Rao Bound of accuracy as well as the´ corresponding maximum likelihood estimator. However, in real scenarios where obstructions and reflections are common, the signal propagation is far more complex than the pathloss model can explain. We propose to augment the pathloss model with a data-driven component, able to explain the complexities of the propagation channel. The paper shows a general methodology to jointly estimate the interference location and the parameters of the augmented model, showing superior performances in complex scenarios such as those encountered in urban environments.

Subspace clustering algorithms are used for understanding the cluster structure that explains the patterns prevalent in the dataset well. These methods are extensively used for data-exploration tasks in various areas of Natural Sciences. However, most of these methods fail to handle confounding attributes in the dataset. For datasets where a data sample represent multiple attributes, naively applying any clustering approach can result in undesired output. To this end, we propose a novel framework for jointly removing confounding attributes while learning to cluster data points in individual subspaces. Assuming we have label information about these confounding attributes, we regularize the clustering method by adversarially learning to minimize the mutual information between the data representation and the confounding attribute labels. Our experimental result on synthetic and real-world datasets demonstrate the effectiveness of our approach.

We study the problem of inferring user intent from noninvasive electroencephalography (EEG) to restore communication for people with severe speech and physical impairments (SSPI). The focus of this work is improving the estimation of posterior symbol probabilities in a typing task. At each iteration of the typing procedure, a subset of symbols is chosen for the next query based on the current probability estimate. Evidence about the user’s response is collected from event-related potentials (ERP) in order to update symbol probabilities, until one symbol exceeds a predefined confidence threshold. We provide a graphical model describing this task, and derive a recursive Bayesian update rule based on a discriminative probability over label vectors for each query, which we approximate using a neural network classifier. We evaluate the proposed method in a simulated typing task and show that it outperforms previous approaches based on generative modeling.

A lower bound is an important tool for predicting the performance that an estimator can achieve under a particular statistical model. Bayesian bounds are a kind of such bounds which not only utilizes the observation statistics but also includes the prior model information. In reality, however, the true model generating the data is either unknown or simplified when deriving estimators, which motivates the works to derive estimation bounds under modeling mismatch situations. This paper provides a derivation of a Bayesian Cramér-Rao bound under model misspecification, by introducing important concepts such as the pseudotrue parameter in a Bayesian context which was not identified in previous works. The general result is particularized in linear and Gaussian problems, where closed-forms are available and results are used to validate the results.

Humans can perform seamless object handovers with little to no effort. These handovers are characterized by an early movement onset that anticipates the handover location and a smooth velocity profile with minimal trajectory corrections. Replicating these characteristics in an object handover task between humans and robots presents a significant modeling challenge. In this paper we implement a Gaussian Process prediction model to serve as a robotic surrogate of human inference, and investigate how this model affects the kinematics of a human giver handing an object to the robot. Additionally, we analyze how the resulting robot kinematics compare to those of a human, and gauge human comfort through subjective reporting. Human giver kinematics during human-robot handover compared closely to human-human giver kinematics with respect to movement speed, movement timing, movement smoothness, and handover distance. Notable differences were observed in reach time and receiver peak transport velocity. When asked how well four attributes of their human-robot handovers (receiver competence, handover comfort, handover naturalness, handover safety) compared to those attributes in human-human handovers, subjects gave mean scores ranging from 4.43 (naturalness) to 5.13 (safety) on a 7 point Likert scale.

Multitemporal hyperspectral unmixing (MTHU) is a fundamental tool in the analysis of hyperspectral image sequences. It reveals the dynamical evolution of the materials (endmembers) and of their proportions (abundances) in a given scene. However, adequately accounting for the spatial and temporal variability of the endmembers in MTHU is challenging, and has not been fully addressed so far in unsupervised frameworks. In this work, we propose an unsupervised MTHU algorithm based on variational recurrent neural networks. First, a stochastic model is proposed to represent both the dynamical evolution of the endmembers and their abundances, as well as the mixing process. Moreover, a new model based on a low-dimensional parametrization is used to represent spatial and temporal endmember variability, significantly reducing the amount of variables to be estimated. We propose to formulate MTHU as a Bayesian inference problem. However, the solution to this problem does not have an analytical solution due to the nonlinearity and non-Gaussianity of the model. Thus, we propose a solution based on deep variational inference, in which the posterior distribution of the estimated abundances and endmembers is represented by using a combination of recurrent neural networks and a physically motivated model. The parameters of the model are learned using stochastic backpropagation. Experimental results show that the proposed method outperforms state of the art MTHU algorithms.

The recent evolution of hyperspectral imaging technology and the proliferation of new emerging applications press for the processing of multiple temporal hyperspectral images. In this work, we propose a novel spectral unmixing (SU) strategy using physically motivated parametric endmember (EME) representations to account for temporal spectral variability. By representing the multitemporal mixing process using a state-space formulation, we are able to exploit the Bayesian filtering machinery to estimate the EME variability coefficients. Moreover, by assuming that the temporal variability of the abundances is small over short intervals, an efficient implementation of the expectation–maximization (EM) algorithm is employed to estimate the abundances and the other model parameters. Simulation results indicate that the proposed strategy outperforms state-of-the-art multi-temporal SU (MTSU) algorithms.

Autoencoder (AEC) networks have recently emerged as a promising approach to perform unsupervised hyperspectral unmixing (HU) by associating the latent representations with the abundances, the decoder with the mixing model, and the encoder with its inverse. AECs are especially appealing for nonlinear HU since they lead to unsupervised and model-free algorithms. However, existing approaches fail to explore the fact that the encoder should invert the mixing process, which might reduce their robustness. In this letter, we propose a model-based AEC for nonlinear HU by considering the mixing model a nonlinear fluctuation over a linear mixture. Different from previous works, we show that this restriction naturally imposes a particular structure to both the encoder and decoder networks. This introduces prior information in the AEC without reducing the flexibility of the mixing model. Simulations with synthetic and real data indicate that the proposed strategy improves nonlinear HU.

Multiple endmember spectral mixture analysis (MESMA) is one of the leading approaches to perform spectral unmixing (SU) considering the variability of the endmembers (EMs). It represents each EM in the image using libraries of spectral signatures acquireda priori. However, existing spectral libraries are often small and unable to properly capture the variability of each EM in practical scenes, which compromises the performance of MESMA. In this letter, we propose a library augmentation strategy to increase the diversity of existing spectral libraries, thus improving their ability to represent the materials in real images. First, we leverage the power of deep generative models to learn the statistical distribution of the EMs based on the spectral signatures available in the existing libraries. Afterward, new samples can be drawn from the learned EM distributions and used to augment the spectral libraries, improving the overall quality of the SU process. Experimental results using synthetic and real data attest to the superior performance of the proposed method even under library mismatch conditions.

This contribution presents a novel resampling scheme that leverages Gaussian Processes (GPs) to more accurately approximate the posterior distribution from a set of random measures and, ultimately, enhance resampling by sampling from such approximation. Resampling is a critical step in particle filtering, impacting its estimation performance and parallelization capabilities. The approach can be seen as a kernel-based density approximation. As a byproduct, we are able to i) derive an explicit formula for minimum mean squared error (MMSE) state estimation, and ii) provide a well defined optimization problem for determining the maximum a posteriori (MAP) state estimation. The results on a target tracking problem show the performance improvements of the so-called Gaussian Process Particle Filter (GPPF) when compared to standard particle filtering.

Traditional hyperspectral unmixing methods neglect the underlying variability of spectral signatures often observed in typical hyperspectral images (HI), propagating these mismodeling errors throughout the whole unmixing process. Attempts to model material spectra as members of sets or as random variables tend to lead to severely ill-posed unmixing problems. Although parametric models have been proposed to overcome this drawback by handling endmember (EM) variability through generalizations of the mixing model, the success of these techniques depends on employing appropriate regularization strategies. Moreover, the existing approaches fail to adequately explore the natural multidimensinal representation of HIs. Recently, tensor-based strategies considered low-rank decompositions of HIs as an alternative to impose low-dimensional structures on the solutions of standard and multitemporal unmixing problems. These strategies, however, present two main drawbacks: 1) they confine the solutions to low-rank tensors, which often cannot represent the complexity of real-world scenarios and 2) they lack guarantees that EMs and abundances will be correctly factorized in their respective tensors. In this article, we propose a more flexible approach, called unmixing with low-rank tensor regularization algorithm accounting for EM variability (ULTRA-V), that imposes low-rank structures through regularizations whose strictness is controlled by scalar parameters. Simulations attest the superior accuracy of the method when compared with state-of-the-art unmixing algorithms that account for spectral variability.

Image fusion combines data from different heterogeneous sources to obtain more precise information about an underlying scene. Hyperspectral-multispectral (HS-MS) image fusion is currently attracting great interest in remote sensing since it allows the generation of high spatial resolution HS images and circumventing the main limitation of this imaging modality. Existing HS-MS fusion algorithms, however, neglect the spectral variability often existing between images acquired at different time instants. This time difference causes variations in spectral signatures of the underlying constituent materials due to the different acquisition and seasonal conditions. This paper introduces a novel HS-MS image fusion strategy that combines an unmixing-based formulation with an explicit parametric model for typical spectral variability between the two images. Simulations with synthetic and real data show that the proposed strategy leads to a significant performance improvement under spectral variability and state-of-the-art performance otherwise.

Spectral variability in hyperspectral images can result from factors including environmental, illumination, atmospheric and temporal changes. Its occurrence may lead to the propagation of significant estimation errors in the unmixing process. To address this issue, extended linear mixing models have been proposed which lead to large scale nonsmooth ill-posed inverse problems. Furthermore, the regularization strategies used to obtain meaningful results have introduced interdependencies among abundance solutions that further increase the complexity of the resulting optimization problem. In this paper we present a novel data dependent multiscale model for hyperspectral unmixing accounting for spectral variability. The new method incorporates spatial contextual information to the abundances in extended linear mixing models by using a multiscale transform based on superpixels. The proposed method results in a fast algorithm that solves the abundance estimation problem only once in each scale during each iteration. Simulation results using synthetic and real images compare the performances, both in accuracy and execution time, of the proposed algorithm and other state-of-the-art solutions.

Introducing spatial prior information in hyperspectral imaging (HSI) analysis has led to an overall improvement of the performance of many HSI methods applied for denoising, classification, and unmixing. Extending such methodologies to nonlinear settings is not always straightforward, specially for unmixing problems where the consideration of spatial relationships between neighboring pixels might comprise intricate interactions between their fractional abundances and nonlinear contributions. In this paper, we consider a multiscale regularization strategy for nonlinear spectral unmixing with kernels. The proposed methodology splits the unmixing problem into two sub-problems at two different spatial scales: a coarse scale containing low-dimensional structures, and the original fine scale. The coarse spatial domain is defined using superpixels that result from a multiscale transformation. Spectral unmixing is then formulated as the solution of quadratically constrained optimization problems, which are solved efficiently by exploring their strong duality and a reformulation of their dual cost functions in the form of root-finding problems. Furthermore, we employ a theory-based statistical framework to devise a consistent strategy to estimate all required parameters, including both the regularization parameters of the algorithm and the number of superpixels of the transformation, resulting in a truly blind (from the parameters setting perspective) unmixing method. Experimental results attest the superior performance of the proposed method when comparing with other, state-of-the-art, related strategies.

Endmember (EM) variability has an important impact on the performance of hyperspectral image (HI) analysis algorithms. Recently, extended linear mixing models have been proposed to account for EM variability in the spectral unmixing (SU) problem. The direct use of these models has led to severely ill-posed optimization problems. Different regularization strategies have been considered to deal with this issue, but none so far has consistently exploited the information provided by the existence of multiple pure pixels often present in HIs. In this work, we propose to break the SU problem into a sequence of two problems. First, we use pure pixel information to estimate an interpolated tensor of scaling factors representing spectral variability. This is done by considering the spectral variability to be a smooth function over the HI and confining the energy of the scaling tensor to a low-rank structure. Afterwards, we solve a matrix-factorization problem to estimate the fractional abundances using the variability scaling factors estimated in the previous step, what leads to a significantly more well-posed problem. Simulations with synthetic and real data attest the effectiveness of the proposed strategy.

Sparse hyperspectral unmixing from large spectral libraries has been considered to circumvent the limitations of endmember extraction algorithms in many applications. This strategy often leads to ill-posed inverse problems, which can greatly benefit from spatial regularization strategies. However, existing spatial regularization strategies lead to large-scale nonsmooth optimization problems. Thus, efficiently introducing spatial context in the unmixing problem remains a challenge and a necessity for many real world applications. In this letter, a novel multiscale spatial regularization approach for sparse unmixing is proposed. The method uses a signal-adaptive spatial multiscale decomposition based on segmentation and oversegmentation algorithms to decompose the unmixing problem into two simpler problems: one in an approximation image domain and another in the original domain. Simulation results using both synthetic and real data indicate that the proposed method outperforms the state-of-the-art total variation-based algorithms with a computation time comparable to that of their unregularized counterparts.

Endmember variability is an important factor for accurately unveiling vital information relating the pure materials and their distribution in hyperspectral images. Recently, the extended linear mixing model (ELMM) has been proposed as a modification of the linear mixing model (LMM) to consider endmember variability effects resulting mainly from illumination changes. In this paper, we further generalize the ELMM leading to a new model (GLMM) to account for more complex spectral distortions where different wavelength intervals can be affected unevenly. We also extend the existing methodology to jointly estimate the variability and the abundances for the GLMM. Simulations with real and synthetic data show that the unmixing process can benefit from the extra flexibility introduced by the GLMM.

Kernel-based nonlinear mixing models have been applied to unmix spectral information of hyperspectral images when the type of mixing occurring in the scene is too complex or unknown. Such methods, however, usually require the inversion of matrices of sizes equal to the number of spectral bands. Reducing the computational load of these methods remains a challenge in large-scale applications. This paper proposes a centralized band selection (BS) method for supervised unmixing in the reproducing kernel Hilbert space. It is based upon the coherence criterion, which sets the largest value allowed for correlations between the basis kernel functions characterizing the selected bands in the unmixing model. We show that the proposed BS approach is equivalent to solving a maximum clique problem, i.e., searching for the biggest complete subgraph in a graph. Furthermore, we devise a strategy for selecting the coherence threshold and the Gaussian kernel bandwidth using coherence bounds for linearly independent bases. Simulation results illustrate the efficiency of the proposed method.

Mixing phenomena in hyperspectral images depend on a variety of factors, such as the resolution of observation devices, the properties of materials, and how these materials interact with incident light in the scene. Different parametric and nonparametric models have been considered to address hyperspectral unmixing problems. The simplest one is the linear mixing model. Nevertheless, it has been recognized that the mixing phenomena can also be nonlinear. The corresponding nonlinear analysis techniques are necessarily more challenging and complex than those employed for linear unmixing. Within this context, it makes sense to detect the nonlinearly mixed pixels in an image prior to its analysis, and then employ the simplest possible unmixing technique to analyze each pixel. In this paper, we propose a technique for detecting nonlinearly mixed pixels. The detection approach is based on the comparison of the reconstruction errors using both a Gaussian process regression model and a linear regression model. The two errors are combined into a detection statistics for which a probability density function can be reasonably approximated. We also propose an iterative endmember extraction algorithm to be employed in combination with the detection algorithm. The proposed detect-then-unmix strategy, which consists of extracting endmembers, detecting nonlinearly mixed pixels and unmixing, is tested with synthetic and real images.

This paper investigates the use of Gaussian processes to detect non-linearly mixed pixels in hyperspectral images. The proposed technique is independent of nonlinear mixing mechanism, and therefore is not restricted to any prescribed nonlinear mixing model. The observed reflectances are estimated using both the least squares method and a Gaussian process. The fitting errors of the two approaches are combined in a test statistics for which it is possible to estimate a detection threshold given a required probability of false alarm. The proposed detector is compared to a robust nonlinearity detector recently proposed using synthetic data and is shown to provide a better detection performance. The new detector is also tested on a real hyperspectral image.