Methane (CH4) is the second most powerful greenhouse gas after carbon dioxide and plays a crucial role in climate change due to its high global warming potential. Accurately modeling CH4 fluxes across the globe and at fine temporal scales is essential for understanding its spatial and temporal variability and developing effective mitigation strategies. In this work, we introduce the first-of-its-kind cross-scale global wetland methane benchmark dataset (X-MethaneWet), which synthesizes physics-based model simulation data from TEM-MDM and the real-world observation data from FLUXNET-CH4. This dataset can offer opportunities for improving global wetland CH4 modeling and science discovery with new AI algorithms. To set up AI model baselines for methane flux prediction, we evaluate the performance of various sequential deep learning models on X-MethaneWet. Furthermore, we explore four different transfer learning techniques to leverage simulated data from TEM-MDM to improve the generalization of deep learning models on real-world FLUXNET-CH4 observations. Our extensive experiments demonstrate the effectiveness of these approaches, highlighting their potential for advancing methane emission modeling and identifying new opportunities for developing more accurate and scalable AI-driven climate models.

Face Super-Resolution (FSR), aiming to improve the quality of Low-Resolution (LR) facial images, has been greatly propelled by the deep learning techniques. However, existing approaches, whether based on Convolutional Neural Networks (CNNs) or Transformers, are either inherently damaging facial structures limited by their architectures or failing to capture essential multi-scale textures due to the rigid receptive fields. To address these concerns, we propose a novel dual-domain feature interaction method called Spatial-frequency Multi-scale feature Learning Network (SMLNet) for FSR by employing a dual-branch architecture. Specifically, the frequency branch captures high-quality global structures and fine high-frequency details, while the spatial branch operates complementarily to preserve fine-grained local texture patterns. Moreover, we further introduce a Multi-scale Spatial-frequency feature Interaction Module (MSIM), which combines a Multi-scale Feature Extraction Block (MFEB) and a Spatial-Frequency feature Interaction Module (SFIM) to interact and aggregate multi-level complementary features from the dual branches. Extensive quantitative experiments and qualitative analyses across multiple datasets, together with evaluations on real-world images, demonstrate that the proposed SMLNet significantly outperforms other state-of-the-art methods.

Polarization and intensity images fusion (PIF) has extracted extensive attentions as it can generate images with clear scene information and salient texture details of the object surface that are important for downstream applications. However, existing deep learning-based PIF methods usually lack interpretability and ignore the interactions among multi-modal features. To this end, we propose a novel interpretable low-rank sparse representation guided fusion network for polarization and intensity images (termed LSRNet). Specifically, a low-rank sparse representation deep unfolding module is designed to acquire the base and detail features of the source images, with the ability of improving the interpretability of the network. In addition, a cross-modal connection complementary feature extraction module is proposed, which aims to establish dependency among features of multi-modalities to fully extract complementary features of the source images. In order to demonstrate the validity of our LSRNet and take into account shortcomings of existing datasets for PIF, a multi-scene polarization and intensity image dataset, named MSPI dataset, is constructed, which includes 1034 high-resolution aligned image pairs. According to the best of our knowledge, this is the most comprehensive dataset for PIF that with a large number of image pairs, high resolution and multiple scene types. Extensive experiments on our MSPI dataset and two publicly available datasets (i.e., 12CFC and HCP) demonstrate the superior fusion performance, generalization ability, and desirable running efficiency of our LSRNet. Our codes and dataset will be publicly available at https://github.com/thebinyang/LSRNet.

Cellular responses to genetic perturbations are prevalent in wide contexts from the fundamental understandings on pathology to the development of clinical therapies and the discovery of novel drug targets. Nonetheless, the substantial amount of possible perturbation combinations renders wet-lab experiments prohibitively expensive and time-consuming. To address it, the BRNET model is proposed for predicting non-linear transcriptional outcomes where multiple perturbations exist. BRNET integrates prior knowledge with advanced embeddings into a non-stacked neural structure to predict transcriptional responses to both individual and multiple genetic perturbations. For unseen scenarios, BRNET also generalizes well under the corresponding perturbations. Experimental results highlight the capabilities of BRNET, demonstrating promising performance as compared to established deep learning models.

Unsupervised domain adaptation (UDA) addresses the domain shift problem by transferring knowledge from labeled source domain data (e.g. CT) to unlabeled target domain data (e.g. MRI). While state-of-the-art methods reduce domain gaps via image- or feature-level alignment, their reliance on spurious correlations in the training data often limits generalization across domains. To overcome this limitation, we propose the Causal Intervention Segmentation Network (CiSeg), a novel framework that first integrates causal inference into UDA. A Structural Causal Model (SCM) is first constructed for the source domain to disentangle causal variables from bias variables, alleviating the impact of spurious correlations. Based on this SCM, we introduce a Counterfactual Disentanglement (CD) module to decompose the source domain's latent features into distinct causal and bias components, effectively eliminating their mutual dependencies. To enhance cross-domain consistency, two auxiliary components are introduced: Prototype-guided Contrastive Learning (PCL) and Causal-bias Residual Alignment (CBRA). PCL aligns pixel-level representations with their corresponding semantic prototypes, promoting stronger intra-class consistency and clearer inter-class separability. CBRA employs adversarial learning to align causal and bias residual features across domains, further enhancing feature-level invariance. Extensive experiments on cardiac, abdominal multi-organ, and BraTS18 segmentation tasks demonstrate that CiSeg outperforms state-of-the-art methods, achieving superior segmentation performance and robust cross-domain generalization. Code and models are available at https://github.com/lvpeiqing/CiSeg.

Selective prediction [Dru13, QV19] models the scenario where a forecaster freely decides on the prediction window that their forecast spans. Many data statistics can be predicted to a non-trivial error rate *without any* distributional assumptions or expert advice, yet these results rely on that the forecaster may predict at any time. We introduce a model of Prediction with Limited Selectivity (PLS) where the forecaster can start the prediction only on a subset of the time horizon. We study the optimal prediction error both on an instance-by-instance basis and via an average-case analysis. We introduce a complexity measure that gives instance-dependent bounds on the optimal error. For a randomly-generated PLS instance, these bounds match with high probability.

Broad learning system (BLS) is an effective neural network requiring no deep architecture, however it is somehow fragile to noisy data. The previous robust broad models directly map features from the raw data, which inevitably learn useless or even harmful features for data representation when the inputs are corrupted by noise and outliers. To address this concern, a discriminative and robust network named as dynamic graph regularized broad learning (DGBL) with marginal fisher representation is proposed for noisy data classification. Different from the previous works, DGBL eliminates the effect of noise before the random feature mapping by the proposed robust and dynamic marginal fisher analysis (RDMFA) algorithm. The RDMFA is able to extract more robust and informative representations for classification from the latent clean data space with dynamically generated graphs. Furthermore, the dynamic graphs learned from RDMFA are incorporated as regularization terms into the objective of DGBL to enhance the discrimination capacity of the proposed network. Extensive quantitative and qualitative experiments conducted on numerous benchmark datasets demonstrate the superiority of the proposed model compared to several state-of-the-art methods.

The interpretability of quantum machine learning (QML) refers to the capability to provide clear and understandable explanations for the predictions and decision-making processes of QML models. However, most quantum convolutional layers (QCLs) utilize closed-box structures that are inherently devoid of interpretability, leading to the opacity of principles and the suboptimal mapping of classical data. This significantly undermines the reliability of QML models. In addition, most of the current QML interpretability focuses on post hoc interpretability seriously neglecting the importance of exploring intrinsic causes. To tackle these challenges, we introduce the quantum adjoint convolution operation (QACO). It is an intrinsic interpretability scheme based on quantum evolution, as its quantum mapping precisely corresponds to the position and pixel values of the image and its principle is equivalent to the Frobenius inner product (FIP). Furthermore, we extend the QACO concept into the quantum adjoint convolutional layer (QACL) by integrating the quantum phase estimation (QPE) algorithm, enabling the parallel computation of all FIPs. Experimental results on PennyLane and TensorFlow platforms demonstrate that our method achieves a 6.3%, 3.4%, and 2.9% higher average test accuracy on Fashion MNIST, MNIST, and DermaMNIST datasets compared to classical and uninterpretable quantum counterparts, respectively, while maintaining 73.3% noise-robust accuracy under Gaussian noise, showcasing its superior generalizability and resilience in practical scenarios.

As typical structural noise, stripes commonly occur in remote sensing images captured by linear array sensors, which seriously lowers the image quality and hinders the downstream applications. Differing from the conventional methods, this article explores the ability of the ordering domain in separable stripe representation and provides a new perspective for destriping. To enhance the model flexibility to adapt to different types of stripes, the additive and multiplicative stripe components are fully considered and creatively incorporated into the observation model. Based on the additive-multiplicative observation model and the ordering domain transformation, we propose a novel destriping model, called ordering domain destriping (ODD), which constrains the additive and multiplicative stripe components in line with their statistical distribution characteristics. The results obtained on simulated and real striped images show that the proposed method can successfully estimate the latent clean images without losing stripe-like object details in challenging test scenarios, such as mixed additive-multiplicative stripes, wide stripes, and deadlines. The qualitative and quantitative comparisons with six other destriping methods verify the effectiveness and stability of the proposed model.

Deep learning (DL)-based methods have achieved tremendous success in remote sensing (RS) change detection (CD). However, most DL-based methods heavily rely on high-quality labeled samples, where noisy labels are inevitably introduced during the annotation process, especially at edge regions. Under the supervision of such noisy labels, the performance of RS CD models will deteriorate significantly. To break the limitation, a novel uncertainty-aware noisy label learning network is proposed for RS CD, termed UNLLNet. Specifically, a joint detection strategy based on uncertainty analysis is proposed, which leverages samples characterized by low uncertainty and high probability to detect and correct potential noisy labels, thereby mitigating the risk of erroneous correction. Given that noisy labels are more likely to be introduced at edge regions, an edge-guided inter-level difference refinement module (EIDRM) is designed to effectively calibrate the edge structure of change regions, thereby facilitating better identification of noisy labels at edge regions. Moreover, a noise-robust loss function with adaptive hard sample enhancement is introduced, assigning loss weights to each sample based on their uncertainty to further enhance the model robustness over noisy labels. Experimental results on LEVIR-CD, CDD, and CLCD datasets validate the effectiveness and advantages of UNLLNet compared to other state-of-the-art RS CD methods.

The synthetic aperture radar (SAR) systems can provide submeter terrain mapping and accurate point elevation information quickly and efficiently. The interferometric SAR (InSAR) technology has proven to be a powerful method for producing digital elevation models (DEMs). However, DEM generation using InSAR technology is limited by mountain shadow overlap, atmospheric noise, low backscatter coefficient, and spatiotemporal incoherence, leading to the problem of voids. This article proposes an intelligent learning reconfiguration model based on optimized transformer and multisource features (TMSFs). First, the intelligent learning reconfiguration model based on the transformer and convolutional neural network (CNN) was constructed, and the multisource feature connection module was used for feature supervision and loss function optimization. Then, the relationship of nonvoid areas between the low-resolution (LR) DEM and the high-resolution (HR) InSAR DEM was found, and the voids were intelligently filled. The experiments used 19 TerraSAR-X images in San Diego (SD), USA, and 18 PAZ images in Yan’an (YA), China, to generate high-precision InSAR void DEMs and intelligently fill the voids. Compared with traditional interpolation or deep learning models, modeling accuracy improved by 11.31%–45.74% and 2.32%–8.78% in the SD and YA areas, respectively. Using the photogrammetric DEM to evaluate the accuracy of the filled DEM, the new method showed improvements of 15.64%–25.91% and 5.60%–28.26%, respectively. In addition, 122 Ice, Cloud, and land Elevation Satellite (ICESat)/Geosciences Laser Altimeter System (GLAS) points collected in the YA area were further validated, with an improvement of 4.40%–22.28%. The generated DEM has considerable advantages for terrain feature preservation and river network extraction, and the new method can provide technical support for DEM void filling.

The convolutional neural network (CNN) and transformer have gained significant attention in the field of single image super-resolution (SISR), owing to their powerful capacity in nonlinear feature extraction. Nonetheless, these two types of approaches hold their own limitations. For instance, the interaction between convolutional kernels and image content is agnostic in CNN, while the computational complexity increases quadratically along with the spatial resolution in the transformer. To address these concerns, in this article, we propose a novel unified framework named transformer and convolution coupled contrastive network (TC3Net) for SISR, which holds a triple-branch structure to integrate the merits of both CNN and transformer. The proposed TC3Net is mainly composed of several stacked CNN feature extraction (CFE) blocks, transformer feature extraction (TFE) blocks, and coupled contrastive blocks (CCBs) for diverse feature extraction. Particularly, the CCB that consists of the coupled attention block (CAB) and the local-global feature extraction (LGFE) block is designed to fuse feature maps and extract coupled information for better image reconstruction. Moreover, a contrastive loss between the transformer and CNN feature maps is further introduced to enhance their discriminative characteristics and complement the fused features. Experimental results demonstrate that TC3Net outperforms several state-of-the-art (SOTA) methods in the aspect of achieving a better balance between model size and performance.

Training machine learning (ML) models for scientific problems is often challenging due to limited observation data. To overcome this challenge, prior works commonly pre-train ML models using simulated data before having them fine-tuned with small real data. Despite the promise shown in initial research across different domains, these methods cannot ensure improved performance after fine-tuning because (i) they are not designed for extracting generalizable physics-aware features during pre-training, (ii) the features learned from pre-training can be distorted by the fine-tuning process. In this paper, we propose a new learning method for extracting, preserving, and adapting physics-aware features. We build a knowledge-guided neural network (KGNN) model based on known dependencies amongst physical variables, which facilitate extracting physics-aware feature representation from simulated data. Then we fine-tune this model by alternately updating the encoder and decoder of the KGNN model to enhance the prediction while preserving the physics-aware features learned through pre-training. We further propose to adapt the model to new testing scenarios via a teacher-student learning framework based on the model uncertainty. The results demonstrate that the proposed method outperforms many baselines by a good margin, even using sparse training data or under out-of-sample testing scenarios.

With the rapid development of remote sensing imaging technology, change detection (CD) with heterogeneous images has become a hot topic in the community. Given the distinct physical properties of heterogeneous images, it is difficult for direct extraction of change information. Some models that transform heterogeneous images into a mutual feature domain can be beneficial. However, the transformation may be influenced by the changed areas that are not the discrepancy of the domains, which further decreases the accuracy of CD. To solve the problem, we propose a reciprocal transformation-based joint deep and broad learning (RTDBL) model for CD with heterogeneous images. In the RTDBL model, in order to rapidly extract features, a deep feature extraction (DFE) module is designed without the need for training. In addition, for directly highlighting change information and eliminating the influence of changed areas, a reciprocal heterogeneous nodes transformation (RHNT) module is designed to construct regression functions for achieving reciprocal transformation. Subsequently, to achieve cross-spatial information interaction, a structural nodes extraction (SNE) module is proposed for obtaining structural nodes. For effectively utilizing aforementioned information and exploring the connections of heterogeneous nodes, a heterogeneous dual broad learning (HDBL) is developed to predict the change map. According to the best of our knowledge, this is the first attempt that joints deep learning and broad learning for CD with heterogeneous images. The efficacy of the proposed RTDBL is demonstrated through experimental analysis on four widely used datasets, in comparison with ten state-of-the-art models.

Deep learning-based hyperspectral image (HSI) change detection (CD) approaches have a strong ability to leverage spectral-spatial-temporal information through automatic feature extraction, and currently dominate in the research field. However, their efficiency and universality are limited by the dependency on labeled data. Although the newly applied untrained networks can avoid the need for labeled data, their feature volatility from the simple difference space easily leads to inaccurate CD results. Inspired by the interesting finding that salient changes appear as bright "stripes" in a new feature space, we propose a novel unsupervised CD method that represents and models changes in stripes for HSIs (named as StripeCD), which integrates optimization modeling into an untrained network. The StripeCD method constructs a new feature space that represents change features in stripes and models them in a novel optimization manner. It consists of three main parts: 1) dual-branch untrained convolutional network, which is utilized to extract deep difference features from bitemporal HSIs and combined with a two-stage channel selection strategy to emphasize the important channels that contribute to CD. 2) multiscale forward-backward segmentation framework, which is proposed for salient change representation. It transforms deep difference features into a new feature space by exploiting the structure information of ground objects and associates salient changes with the stripe-shaped change component. 3) stripe-shaped change extraction model, which characterizes the global sparsity and local discontinuity of salient changes. It explores the intrinsic properties of deep difference features and constructs model-based constraints to better identify changed regions in a controllable manner. The proposed StripeCD method outperformed the state-of-the-art unsupervised CD approaches on three widely used datasets. In addition, the proposed StripeCD method indicates the potential for further investigation of untrained networks in facilitating reliable CD.

Traditional partition-based clustering is very sensitive to the initialized centroids, which are easily stuck in the local minimum due to their nonconvex objectives. To this end, convex clustering is proposed by relaxing$K$-means clustering or hierarchical clustering. As an emerging and excellent clustering technology, convex clustering can solve the instability problems of partition-based clustering methods. Generally, convex clustering objective consists of the fidelity and the shrinkage terms. The fidelity term encourages the cluster centroids to estimate the observations and the shrinkage term shrinks the cluster centroids matrix so that their observations share the same cluster centroid in the same category. Regularized by the$\ell_{p_n}$-norm ($p_n\in\{1,2,+\infty\}$), the convex objective guarantees the global optimal solution of the cluster centroids. This survey conducts a comprehensive review of convex clustering. It starts with the convex clustering as well as its nonconvex variants and then concentrates on the optimization algorithms and the hyperparameters setting. In particular, the statistical properties, the applications, and the connections of convex clustering with other methods are reviewed and discussed thoroughly for a better understanding the convex clustering. Finally, we briefly summarize the development of convex clustering and present some potential directions for future research.

Broad learning system (BLS) is a novel neural network with efficient learning and expansion capacity, but it is sensitive to noise. Accordingly, the existing robust broad models try to suppress noise by assigning each sample an appropriate scalar weight to tune down the contribution of noisy samples in network training. However, they disregard the useful information of the noncorrupted elements hidden in the noisy samples, leading to unsatisfactory performance. To this end, a novel BLS with adaptive reweighting (BLS-AR) strategy is proposed in this article for the classification of data with label noise. Different from the previous works, the BLS-AR learns for each sample a weight vector rather than a scalar weight to indicate the noise degree of each element in the sample, which extends the reweighting strategy from sample level to element level. This enables the proposed network to precisely identify noisy elements and thus highlight the contribution of informative ones to train a more accurate representation model. Thanks to the separability of the model, the proposed network can be divided into several subnetworks, each of which can be trained efficiently. In addition, three corresponding incremental learning algorithms of the BLS-AR are developed for adding new samples or expanding the network. Substantial experiments are conducted to explicate the effectiveness and robustness of the proposed BLS-AR model.

A novel neural network, namely, broad learning system (BLS), has shown impressive performance on various regression and classification tasks. Nevertheless, most BLS models may suffer serious performance degradation for contaminated data, since they are derived under the least-squares criterion which is sensitive to noise and outliers. To enhance the model robustness, in this article we proposed a modal-regression-based BLS (MRBLS) to tackle the regression and classification tasks of data corrupted by noise and outliers. Specifically, modal regression is adopted to train the output weights instead of the minimum mean square error (MMSE) criterion. Moreover, the$\ell_{2,1}$-norm-induced constraint is used to encourage row sparsity of the connection weight matrix and achieve feature selection. To effectively and efficiently train the network, the half-quadratic theory is used to optimize MRBLS. The validity and robustness of the proposed method are verified on various regression and classification datasets. The experimental results demonstrate that the proposed MRBLS achieves better performance than the existing state-of-the-art BLS methods in terms of both accuracy and robustness.

This paper proposes a physics-guided neural network model to predict crop yield and maintain the fairness over space. Failures to preserve the spatial fairness in predicted maps of crop yields can result in biased policies and intervention strategies in the distribution of assistance or subsidies in supporting individuals at risk. Existing methods for fairness enforcement are not designed for capturing the complex physical processes that underlie the crop growing process, and thus are unable to produce good predictions over large regions under different weather conditions and soil properties. More importantly, the fairness is often degraded when existing methods are applied to different years due to the change of weather conditions and farming practices. To address these issues, we propose a physics-guided neural network model, which leverages the physical knowledge from existing physics-based models to guide the extraction of representative physical information and discover the temporal data shift across years. In particular, we use a reweighting strategy to discover the relationship between training years and testing years using the physics-aware representation. Then the physics-guided neural network will be refined via a bi-level optimization process based on the reweighted fairness objective. The proposed method has been evaluated using real county-level crop yield data and simulated data produced by a physics-based model. The results demonstrate that this method can significantly improve the predictive performance and preserve the spatial fairness when generalized to different years.

Spatio-temporal machine learning is critically needed for a variety of societal applications, such as agricultural monitoring, hydrological forecast, and traffic management. These applications greatly rely on regional features that characterize spatial and temporal differences. However, spatio-temporal data often exhibit complex patterns and significant data variability across different locations. The labels in many real-world applications can also be limited, which makes it difficult to separately train independent models for different locations. Although meta learning has shown promise in model adaptation with small samples, existing meta learning methods remain limited in handling a large number of heterogeneous tasks, e.g., a large number of locations with varying data patterns. To bridge the gap, we propose task-adaptive formulations and a model-agnostic meta-learning framework that transforms regionally heterogeneous data into location-sensitive meta tasks. We conduct task adaptation following an easy-to-hard task hierarchy in which different meta models are adapted to tasks of different difficulty levels. One major advantage of our proposed method is that it improves the model adaptation to a large number of heterogeneous tasks. It also enhances the model generalization by automatically adapting the meta model of the corresponding difficulty level to any new tasks. We demonstrate the superiority of our proposed framework over a diverse set of baselines and state-of-the-art meta-learning frameworks. Our extensive experiments on real crop yield data show the effectiveness of the proposed method in handling spatial-related heterogeneous tasks in real societal applications.

In many environmental applications, recurrent neural networks (RNNs) are often used to model physical variables with long temporal dependencies. However, due to minibatch training, temporal relationships between training segments within the batch (intra-batch) as well as between batches (inter-batch) are not considered, which can lead to limited performance. Stateful RNNs aim to address this issue by passing hidden states between batches. Since Stateful RNNs ignore intra-batch temporal dependency, there exists a trade-off between training stability and capturing temporal dependency. In this paper, we provide a quantitative comparison of different Stateful RNN modeling strategies, and propose two strategies to enforce both intra- and inter-batch temporal dependency. First, we extend Stateful RNNs by defining a batch as a temporally ordered set of training segments, which enables intra-batch sharing of temporal information. While this approach significantly improves the performance, it leads to much larger training times due to highly sequential training. To address this issue, we further propose a new strategy which augments a training segment with an initial value of the target variable from the timestep right before the starting of the training segment. In other words, we provide an initial value of the target variable as additional input so that the network can focus on learning changes relative to that initial value. By using this strategy, samples can be passed in any order (mini-batch training) which significantly reduces the training time while maintaining the performance. In demonstrating the utility of our approach in hydrological modeling, we observe that the most significant gains in predictive accuracy occur when these methods are applied to state variables whose values change more slowly, such as soil water and snowpack, rather than continuously moving flux variables such as streamflow.

Broad learning system (BLS), an efficient neural network with a flat structure, has received a lot of attention due to its advantages in training speed and network extensibility. However, the conventional BLS adopts the least square loss, which treats each sample equally and thus is sensitivity to noise and outliers. To address this concern, in this article we propose a self-paced BLS (SPBLS) model by incorporating the novel self-paced learning (SPL) strategy into the network for noisy data regression. With the assistance of the SPL criterion, the model output is used as feedback to learn appropriate priority weight to readjust the importance of each sample. Such a reweighting strategy can help SPBLS to distinguish samples from "easy" to "difficult" in model training, equipping the model robust to noise and outliers while maintaining the characteristics of the original system. Moreover, two incremental learning algorithms associated to SPBLS have also been developed, with which the system can be updated quickly and flexibly without retraining the entire model when new training samples are added or the network needs to be expanded. Experiments conducted on various datasets demonstrate that the proposed SPBLS can achieve satisfying performance for noisy data regression.

Deep learning approaches have been extensively applied to change detection in hyperspectral images (HSIs). However, the majority of them encounter scarcity of training samples or rely on complex structures and learning strategies. Although untrained change detection models have been proved to be effective in relief above problems, they were constructed using regular convolutions and treated spatial locations and channels equally, which are insufficient to extract discriminative features and lead to limited accuracy. Given this, a novel untrained framework using randomly initialized models with spatial-channel augmentation (RICD) is proposed for HSI change detection in this paper. It consists of two major modules: 1) an enhanced feature extraction network using successive dilation-deformable feature extraction blocks, which can extract multiscale spatial-spectral features over unfixed sampling locations. It enlarges the field of view of convolutions and takes arbitrary neighborhood into consideration, which helps to increase the discriminativeness of the extracted features; 2) a change sensitive feature augmentation and comparison module integrating feature selection and spatial-channel augmentation strategies, which can exploit spatial context and channel importance. It magnifies difference between changed pixels and unchanged ones and emphasizes contribution of significant channels of the selected change sensitive features. Despite that convolution operations are included in RICD, all the weights are untrained and fixed once they are randomly initialized, indicating that the RICD can work in an unsupervised manner. Its performance is tested over three widely used hyperspectral datasets. Quantitative and qualitative comparisons with several state-of-the-art unsupervised methods reveal the effectiveness of the RICD method.

Multi-shot coded aperture snapshot spectral imaging (CASSI) uses multiple measurement snapshots to encode the three-dimensional hyperspectral image (HSI). Increasing the number of snapshots will multiply the number of measurements, making CASSI system more appropriate for detailed spatial or spectrally rich scenes. However, the reconstruction algorithms still face the challenge of being ineffective or inflexible. In this paper, we propose a plug-and-play (PnP) method that uses denoiser as priors for multi-shot CASSI. Specifically, the proposed PnP method is based on the primal-dual algorithm with linesearch (PDAL), which makes it flexible and can be used for any multi-shot CASSI mechanisms. Furthermore, a new subspaced-based nonlocal reweighted low-rank (SNRL) denoiser is presented to utilize the global spectral correlation and nonlocal self-similarity priors of HSI. By integrating the SNRL denoiser into PnP-PDAL, we show the balloons ( 512×512×31 ) in CAVE dataset recovered from two snapshots compressive measurements with MPSNR above 50 dB. Experimental results demonstrate that our proposed method leads to significant improvements compared to the current state-of-the-art methods.

Manifold learning-based face hallucination technologies have been widely developed during the past decades. However, the conventional learning methods always become ineffective in noise environment due to the least-square regression, which usually generates distorted representations for noisy inputs they employed for error modeling. To solve this problem, in this article, we propose a modal regression-based graph representation (MRGR) model for noisy face hallucination. In MRGR, the modal regression-based function is incorporated into graph learning framework to improve the resolution of noisy face images. Specifically, the modal regression-induced metric is used instead of the least-square metric to regularize the encoding errors, which admits the MRGR to robust against noise with uncertain distribution. Moreover, a graph representation is learned from feature space to exploit the inherent typological structure of patch manifold for data representation, resulting in more accurate reconstruction coefficients. Besides, for noisy color face hallucination, the MRGR is extended into quaternion (MRGR-Q) space, where the abundant correlations among different color channels can be well preserved. Experimental results on both the grayscale and color face images demonstrate the superiority of MRGR and MRGR-Q compared with several state-of-the-art methods.

Recently, learning-based representation techniques have been well exploited for grayscale face image hallucination. For color images, the previous methods only handle the luminance component or each color channel individually, without considering the abundant correlations among different channels as well as the inherent geometrical structure of data manifold. In this article, we propose a learning-based model in quaternion space with graph representation for color face hallucination. Instead of the spatial domain, the color image is represented in the quaternion domain to preserve correlations among different color channels. Moreover, a quaternion graph is learned to smooth the quaternion feature space, which helps to not only stabilize the linear system but also enclose the inherent topology structure of quaternion patch manifold. Besides, considering that single low-resolution (LR) image patch can just provide limited informative information in representation, we propose to simultaneously encode the query smaller LR patch as well as a larger patch containing the surrounding pixels seated at the same position in the objective. The larger patch with rich patterns is used to compensate the lost information in the query LR patch, which further enhances the manifold consistency assumption between the LR and HR patch spaces. The experimental results demonstrated the efficiency of the proposed method in hallucinating color face images.

Face hallucination technologies have been widely developed during the past decades, among which the sparse manifold learning (SML)-based approaches have become the popular ones and achieved promising performance. However, these SML methods always failed in handling noisy images due to the least-square regression (LSR) they used for error approximation. To this end, we propose, in this article, a smooth correntropy representation (SCR) model for noisy face hallucination. In SCR, the correntropy regularization and smooth constraint are combined into one unified framework to improve the resolution of noisy face images. Specifically, we introduce the correntropy induced metric (CIM) rather than the LSR to regularize the encoding errors, which admits the proposed method robust to noise with uncertain distributions. Besides, the fused LASSO penalty is added into the feature space to ensure similar training samples holding similar representation coefficients. This encourages the SCR not only robust to noise but also can well exploit the inherent typological structure of patch manifold, resulting in more accurate representations in noise environment. Comparison experiments against several state-of-the-art methods demonstrate the superiority of SCR in super-resolving noisy low-resolution (LR) face images.

Unsupervised hashing has attracted much attention for binary representation learning due to the requirement of economical storage and efficiency of binary codes. It aims to encode high-dimensional features in the Hamming space with similarity preservation between instances. However, most existing methods learn hash functions in manifold-based approaches. Those methods capture the local geometric structures (i.e., pairwise relationships) of data, and lack satisfactory performance in dealing with real-world scenarios that produce similar features (e.g., color and shape) with different semantic information. To address this challenge, in this work, we propose an effective unsupervised method, namely, Jointly Personalized Sparse Hashing (JPSH), for binary representation learning. To be specific, first, we propose a novel personalized hashing module, i.e., Personalized Sparse Hashing (PSH). Different personalized subspaces are constructed to reflect category-specific attributes for different clusters, adaptively mapping instances within the same cluster to the same Hamming space. In addition, we deploy sparse constraints for different personalized subspaces to select important features. We also collect the strengths of the other clusters to build the PSH module with avoiding over-fitting. Then, to simultaneously preserve semantic and pairwise similarities in our proposed JPSH, we incorporate the proposed PSH and manifold-based hash learning into the seamless formulation. As such, JPSH not only distinguishes the instances from different clusters but also preserves local neighborhood structures within the cluster. Finally, an alternating optimization algorithm is adopted to iteratively capture analytical solutions of the JPSH model. We apply the proposed representation learning algorithm JPSH to the similarity search task. Extensive experiments on four benchmark datasets verify that the proposed JPSH outperforms several state-of-the-art unsupervised hashing algorithms.

Masked face detection is a challenging task in the surveillance applications due to complex backgrounds. In this paper, we propose a two-stage method for masked face detection: pre-detection and verification. Firstly, a masked face detector based on AdaBoost algorithm and histogram of orientation feature is exploited. It may provide sufficient candidate face regions. Secondly, a two-class classifier is trained by broad learning system, which is an incremental learning algorithm with high efficiency in training. It is used to distinguish realistic masked faces from background. Moreover, this paper proposes a masked face dataset that includes multiple masked faces captured from real-life scenes . It can be used for classifier training and evaluation. Experiments conducted on the dataset indicate the effectiveness of the proposed method with Recall 94.69% and Precision 97.72%.

Recent years have witnessed the rapid development of face image hallucination techniques. However, the previous face hallucination methods are unsupervised and ignore the label information of training samples, leading to undesirable results. This article proposes a locality-constrained and category embedding representation (LCER) method to super-resolve face image in a supervised manner by embedding the label information in data representation. The proposed LCER incorporates the locality prior and category information into one unified framework, which aims to learn both the advantages of locality in preserving the true typologic structure of data manifold and the discriminability in exposing the class subspace information. Such strategy allows the LCER not only to preserve more sharpen image details but also to guarantee the face structure pattern be transferred mainly from the same subject in super-resolution reconstruction. Extensive experiments were conducted to evaluate the proposed LCER, and the comparative results demonstrate that it achieved superior face hallucination performance in both the quantitative measurements and visual impressions compared to several state-of-the-art.

Most of the existing face image super-resolution methods are designed for grayscale images. For color images, these methods just treat each color channel individually or considered the illumination part only, ignoring the relationships among different color channels. To address this concern, in this paper we present a color face image super-resolution method by learning the quaternion graph (LQG) representation. Instead of spatial domain, the color image is represented in the quaternionic domain, which encourages the proposed model to well preserve the correlations among different color channels. Besides, a graph regularization is learned in the quaternion space to ensure the smoothness of encoding feature space. More specifically, by utilizing the graph Laplacian, we present to promote the smoothness of representations by forcing similar training samples to share similar encoding coefficients. This not only helps to stabilize the linear system but also makes the model more robust to noise. Experimental results demonstrated the efficiency of the proposed method in super-resolving color face images.

Recently, the locality-constrained linear coding (LLC) based techniques have been widely exploited for face hallucination. However, for the color face image, the conventional LLC model ignores the neighbor self-similarity prior as well as the relevance of different color channels, resulting in unsatisfactory representations. This paper presents a novel Neighbor locality Representation and inter-Channel Correlation (NRCC) model for color face hallucination. Compared with conventional LLC, NRCC makes full use of neighbor self-similarity prior and takes advantage of the co-manifold structure among RGB channels. The neighbor self-similarity prior and co-manifold structure can make the reconstruction results of eyes and lips generated from the proposed method better than those from other methods. The experimental results in some public face databases indicated the superiority of the proposed method over the prior art face hallucination methods.

In recent years, the collaborative representation (CR)-based techniques have been widely employed for face hallucination. However, the conventional CR model becomes less efficient in handling noisy low-resolution face images. In this paper, an iterative relaxed CR (iRCR) model with adaptive weights learning is presented to enhance the resolution of face images corrupted by noise. The core idea of iRCR is that a diagonal weight matrix is incorporated into the objective function, which helps to debase the influence of noise in representation. Different from existing collaborative methods with reweighting strategy where the weights require manually tuning, the weights in iRCR are adaptively learned to stay more consistent with the model error. Moreover, considering the local manifold structure property and nonlocal prior of small patches, the locality regularization and collaborative regularization are incorporated into a unified framework. This enables the proposed iRCR not only to capture the true topology structure of patch manifold but also to exploit the meaningful patterns among the whole training samples for reconstruction. Experimental results on both face dataset and real-world images demonstrate the superiority of our proposed method over several state-of-the-art face hallucination methods.

Tensor completion (TC), aiming to recover original high-order data from its degraded observations, has recently drawn much attention in hyperspectral images (HSIs) domain. Generally, the widely used TC methods formulate the rank minimization problem with a convex trace norm penalty, which shrinks all singular values equally, and may generate a much biased solution. Besides, these TC methods assume the whole high-order data is of low-rank, which may fail to recover the detail information in high-order data with diverse and complex structures. In this paper, a novel nonlocal low-rank regularization-based TC (NLRR-TC) method is proposed for HSIs, which includes two main steps. In the first step, an initial completion result is generated by the proposed low-rank regularization-based TC (LRR-TC) model, which combines the logarithm of the determinant with the tensor trace norm. This model can more effectively approximate the tensor rank, since the logarithm function values can be adaptively tuned for each input. In the second step, the nonlocal spatial-spectral similarity is integrated into the LRR-TC model, to obtain the final completion result. Specifically, the initial completion result is first divided into groups of nonlocal similar cubes (each group forms a 3-D tensor), and then the LRR-TC is applied to each group. Since similar cubes within each group contain similar structures, each 3-D tensor should have low-rank property, and thus further improves the completion result. Experimental results demonstrate that the proposed NLRR-TC method outperforms state-of-the-art HSIs completion techniques.

In visual recognition and surveillance system, human face is one of the most important factors. Unfortunately, due to the low-cost imaging sensors and the complexity imaging environment, the captured face images are always low-resolution (LR) and corrupted by noise. The noisy LR face images possess limited useful information, which will extremely degrade the performance of face recognition system. To address this issue, in this paper we presented a K-nearest neighbor (K-NN) Regularized Collaborative Representation (K-RCR) method to simultaneously enhance the resolution of face images and suppress the noise. The proposed K-RCR breaks the bottlenecks of patch based face super-resolution methods, which makes it to be a reality that denoising and super-resolution can be achieved in a unified framework. Specifically, the K-NN selection strategy is employed to use the most important K nearest neighbors in the training dataset to collaboratively represent the test patch, leading to a unique and stable solution for the least squares problem. Moreover, a diagonal weight matrix is incorporated into the objective function to equip it more robust to noise. Experimental results on the standard test face dataset, i.e., FEI, demonstrate the superiority of our proposed method over several state-of-the-art face image super-resolution methods.

Recently, robust principal component analysis (RPCA) based methods have been used for hyperspectral images (HSIs) restoration to simultaneously remove several types of noise, including Gaussian noise, impulse noise, stripes, and so on. However, most of these RPCA methods formulate the optimization problem with a convex l1-norm penalty, which over-penalizes large entries of vectors, and results in a biased solution. In this paper, a novel nonconvex sparsity regularizer (NonSR) for measuring the clean HSI low rank structure and noise sparsity structure is proposed, which can effectively approximate rank function and noise sparsity instead of the convex l1-norm. By embedding the sparsity regularizer into the RPCA framework, we formulate a new model, which enhance the capability in simultaneously removing several types of noise. In addition, an iterative algorithm based on the alternative direction multiplier method (ADMM) is developed to effectively solve the proposed model. Experimental results demonstrate that the proposed NonSR method outperforms state-of-the-art HSIs restoration techniques.

In recent years, the sparse coding-based techniques have been widely used for image denoising. However, most of the sparse coding-based mixed noise reduction methods fail to take full advantage of the geometric structure of data samples. In other words, they neglect the common information shared by the similar patches in sparse coding. To address this concern, in this paper, we propose a robust constrained sparse representation (RCSR) method to remove mixed noise. By using the center coefficient of similar patches as the guider which is approximated by the coefficient of query patch in sparse coding, the geometric structure of data can be well preserved. Moreover, different from most existing two-stage mixed noise reduction methods that use explicit detectors to restrain impulse noise, the proposed RCSR adaptively adjusts the contribution of each pixel in the loss function to eliminate the influences of outliers. Experiments on the reconstruction of synthetic data and the removal of mixed noise in real images demonstrate the effectiveness of our proposed method.

Recently, locality-constrained linear coding (LLC) has been drawn great attentions and been widely used in image processing and computer vision tasks. However, the conventional LLC model is always fragile to outliers. In this paper, we present a robust locality-constrained bi-layer representation model to simultaneously hallucinate the face images and suppress noise and outliers with the assistant of a group of training samples. The proposed scheme is not only able to capture the nonlinear manifold structure but also robust to outliers by incorporating a weight vector into the objective function to subtly tune the contribution of each pixel offered in the objective. Furthermore, a high-resolution (HR) layer is employed to compensate the missed information in the low-resolution (LR) space for coding. The use of two layers (the LR layer and the HR layer) is expected to expose the complicated correlation between the LR and HR patch spaces, which helps to obtain the desirable coefficients to reconstruct the final HR face. The experimental results demonstrate that the proposed method outperforms the state-of-the-art image super-resolution methods in terms of both quantitative measurements and visual effects.

Recently, the locality linear coding (LLC) has attracted more and more attentions in the areas of image processing and computer vision. However, the conventional LLC with real setting is just designed for the grayscale image. For the color image, it usually treats each color channel individually or encodes the monochrome image by concatenating all the color channels, which ignores the correlations among different channels. In this paper, we propose a quaternion-based locality-constrained coding (QLC) model for color face hallucination in the quaternion space. In QLC, the face images are represented as quaternion matrices. By transforming the channel images into an orthogonal feature space and encoding the coefficients in the quaternion domain, the proposed QLC is expected to learn the advantages of both quaternion algebra and locality coding scheme. Hence, the QLC cannot only expose the true topology of image patch manifold but also preserve the inherent correlations among different color channels. Experimental results demonstrated that our proposed QLC method achieved superior performance in color face hallucination compared with other state-of-the-art methods.

Joint sparse representation (JSR) has shown great potential in various image processing and computer vision tasks. Nevertheless, the conventional JSR is fragile to outliers. In this paper, we propose a weighted JSR (WJSR) model to simultaneously encode a set of data samples that are drawn from the same subspace but corrupted with noise and outliers. Our model is desirable to exploit the common information shared by these data samples while reducing the influence of outliers. To solve the WJSR model, we further introduce a greedy algorithm called weighted simultaneous orthogonal matching pursuit to efficiently approximate the global optimal solution. Then, we apply the WJSR for mixed noise removal by jointly coding the grouped nonlocal similar image patches. The denoising performance is further improved by incorporating it with the global prior and the sparse errors into a unified framework. Experimental results show that our denoising method is superior to several state-of-the-art mixed noise removal methods.

Many impulse noise (IN) reduction methods suffer from two obstacles, the improper noise detectors and imperfect filters they used. To address such issue, in this paper, a weighted couple sparse representation model is presented to remove IN. In the proposed model, the complicated relationships between the reconstructed and the noisy images are exploited to make the coding coefficients more appropriate to recover the noise-free image. Moreover, the image pixels are classified into clear, slightly corrupted, and heavily corrupted ones. Different data-fidelity regularizations are then accordingly applied to different pixels to further improve the denoising performance. In our proposed method, the dictionary is directly trained on the noisy raw data by addressing a weighted rank-one minimization problem, which can capture more features of the original data. Experimental results demonstrate that the proposed method is superior to several state-of-the-art denoising methods.

Many impulse noise removal algorithms do not reach good denoising performance mainly due to the imperfect filters they adopted. In this paper, the popular used sparse representation model is extended for impulse noise removal by using a fuzzy weight matrix. This fuzzy weight is used to describe the noise-like level of the current pixel, and to determine how much information of this pixel should be used in the sparse land model. Besides, a regularization term which counts the proximity between the reconstructed image and the noisy image is also added into the sparse model. This makes the proposed model more robust to the noise detector which generates the fuzzy weight matrix. Moreover, unlike other sparse model, the dictionary used in our model is trained from some reference images that keep the similar structure information of the original image. Therefore, it is more suitable for reconstructing the original image. Simulation results show that our method is superior to all the tested state-of-the-art impulse noise removal methods.