Effectively leveraging multimodal data such as various images, laboratory tests and clinical information is becoming increasingly attractive in a variety of AI-based medical diagnosis and prognosis tasks. Most existing multi-modal techniques only focus on enhancing their performance by leveraging the differences or shared features from various modalities and fusing feature across different modalities. These approaches are generally not optimal for clinical settings, which pose the additional challenges of limited training data, as well as being rife with redundant data or noisy modality channels, leading to subpar performance. To address this gap, we study the robustness of existing methods to data redundancy and noise and propose a generalized dynamic multimodal information bottleneck framework for attaining a robust fused feature representation. Specifically, our information bottleneck module serves to filter out the task-irrelevant information and noises in the fused feature, and we further introduce a sufficiency loss to prevent dropping of task-relevant information, thus explicitly preserving the sufficiency of prediction information in the distilled feature. We validate our model on an in-house and a public COVID19 dataset for mortality prediction as well as two public biomedical datasets for diagnostic tasks. Extensive experiments show that our method surpasses the state-of-the-art and is significantly more robust, being the only method to remain performance when large-scale noisy channels exist. Our code is publicly available at https://github.com/ayanglab/DMIB.

Abstract. This paper investigates the convergence properties and applications of the three-operator splitting method, also known as the Davis–Yin splitting (DYS) method, integrated with extrapolation and plug-and-play (PnP) denoiser within a nonconvex framework. We first propose an extrapolated DYS method to effectively solve a class of structural nonconvex optimization problems that involve minimizing the sum of three possibly nonconvex functions. Our approach provides an algorithmic framework that encompasses both extrapolated forward–backward splitting and extrapolated Douglas–Rachford splitting methods. To establish the convergence of the proposed method, we rigorously analyze its behavior based on the Kurdyka–Łojasiewicz property, subject to some tight parameter conditions. Moreover, we introduce two extrapolated PnP-DYS methods with convergence guarantee, where the traditional regularization step is replaced by a gradient step–based denoiser. This denoiser is designed using a differentiable neural network and can be reformulated as the proximal operator of a specific nonconvex functional. We conduct extensive experiments on image deblurring and image superresolution problems, where our numerical results showcase the advantage of the extrapolation strategy and the superior performance of the learning-based model that incorporates the PnP denoiser in terms of achieving high-quality recovery images.

Single-particle reconstruction (SPR) in cryogenic electron microscopy (cryo-EM) aims at aligning and averaging two-dimensional micrographs to reconstruct a three-dimensional particle. How to reconstruct micrographs from heavy noise is a crucial point for achieving better micrograph quality, and thus many methods focus on noise removal. However, new problems such as over-smoothing often occur in their results due to failure in handling heavy noise well. This paper proposes a three-dimensional weighted nuclear norm minimization (3DWNNM) model for SPR in the cryo-EM task to address these issues. Specifically, we design a minimization solver based on the forward-backward splitting algorithm to tackle our model efficiently. Under certain conditions, this solution has an energy-decaying feature and performs exceptionally well in reconstruction. Numerical experiments fully demonstrate the effectiveness and the robustness of the proposed method.

Abstract. Spherical image processing has been widely applied in many important fields, such as omnidirectional vision for autonomous cars, global climate modeling, and medical imaging. It is nontrivial to extend an algorithm developed for flat images to the spherical ones. In this work, we focus on the challenging task of spherical image inpainting with a deep learning-based regularizer. Instead of a naive application of existing models for planar images, we employ a fast directional spherical Haar framelet transform and develop a novel optimization framework based on a sparsity assumption of the framelet transform. Furthermore, by employing progressive encoder-decoder architecture, a new and better-performed deep CNN denoiser is carefully designed and works as an implicit regularizer. Finally, we use a plug-and-play method to handle the proposed optimization model, which can be implemented efficiently by training the CNN denoiser prior. Numerical experiments are conducted and show that the proposed algorithms can greatly recover damaged spherical images and achieve the best performance over purely using a deep learning denoiser and a plug-and-play model.

Image enhancement is a technique to enhance the illumination of dark images while keeping the reality and the naturalness of the enhanced images at the same time. For color images, most methods tackle different color channels in a separate way, which overlooks the connection between the color channels. Therefore, in this paper, we consider a quaternion-based model to reserve the color connectivity, which integrates the color information of a pixel by a quaternion number. Moreover, we propose a regularizer based on the gamma-correction function and incorporate it into a screened Poisson equation for the image enhancement task. The uniqueness and existence of the solution of the proposed model are analyzed. The numerical results also prove the superiority of our scheme for color image enhancement.

Color image restoration is a critical task in imaging sciences. Most variational methods regard the color image as a Euclidean vector or the direct combination of three monochrome images and completely ignore the inherent color structures within channels. To better describe the relationship of color channels, we represent the color image as the so-called pure quaternion matrix. Note that the celebrated dictionary learning method has attracted considerable attention for image recovery in the past decade. Following this idea, we propose a novel quaternion-based color image recovery method. This model combines the advantages of dictionary learning and the total variation method for color image restoration. The new strategy used in the proposed model manages to handle the color image restoration problem in the quaternion space. Moreover, the new proposed model can be easily solved by the classical alternating direction method of multipliers (ADMM) algorithm. Numerical results demonstrate clearly that the performance of our proposed dictionary learning method is better than some state-of-the-art color image dictionary learning and total variation methods in terms of some criteria and visual quality.