Accurate and automatic segmentation of medical images plays an essential role in clinical diagnosis and analysis. It has been established that integrating contextual relationships substantially enhances the representational ability of neural networks. Conventionally, Long Short-Term Memory (LSTM) and Self-Attention (SA) mechanisms have been recognized for their proficiency in capturing global dependencies within data. However, these mechanisms have typically been viewed as distinct modules without a direct linkage. This paper presents the integration of LSTM design with SA sparse coding as a key innovation. It uses linear combinations of LSTM states for SA's query, key, and value (QKV) matrices to leverage LSTM's capability for state compression and historical data retention. This approach aims to rectify the shortcomings of conventional sparse coding methods that overlook temporal information, thereby enhancing SA's ability to do sparse coding and capture global dependencies. Building upon this premise, we introduce two innovative modules that weave the SA matrix into the LSTM state design in distinct manners, enabling LSTM to more adeptly model global dependencies and meld seamlessly with SA without accruing extra computational demands. Both modules are separately embedded into the U-shaped convolutional neural network architecture for handling both 2D and 3D medical images. Experimental evaluations on downstream medical image segmentation tasks reveal that our proposed modules not only excel on four extensively utilized datasets across various baselines but also enhance prediction accuracy, even on baselines that have already incorporated contextual modules. Code is available at https://github.com/yeshunlong/SALSTM.

Large volume of labeled data is a cornerstone for deep learning (DL) based segmentation methods. Medical images require domain experts to annotate, and full segmentation annotations of large volumes of medical data are difficult, if not impossible, to acquire in practice. Compared with full annotations, image-level labels are multiple orders of magnitude faster and easier to obtain. Image-level labels contain rich information that correlates with the underlying segmentation tasks and should be utilized in modeling segmentation problems. In this article, we aim to build a robust DL-based lesion segmentation model using only image-level labels (normal v.s. abnormal). Our method consists of three main steps: (1) training an image classifier with image-level labels; (2) utilizing a model visualization tool to generate an object heat map for each training sample according to the trained classifier; (3) based on the generated heat maps (as pseudo-annotations) and an adversarial learning framework, we construct and train an image generator for Edema Area Segmentation (EAS). We name the proposed method Lesion-Aware Generative Adversarial Networks (LAGAN) as it combines the merits of supervised learning (being lesion-aware) and adversarial training (for image generation). Additional technical treatments, such as the design of a multi-scale patch-based discriminator, further enhance the effectiveness of our proposed method. We validate the superior performance of LAGAN via comprehensive experiments on two publicly available datasets (i.e., AI Challenger and RETOUCH).

Recently, hashing has been successfully applied for large-scale remote sensing image retrieval (LSRSIR) due to its advantage in terms of computation and storage. In LSRSIR, existing hashing methods mainly focus on single-source remotely sensed data. They cannot effectively fuse multisource remotely sensed data, which has a large potential for LSRSIR. To fulfill this gap, this letter proposes a novel deep hashing method, dubbed Multiview Graph Convolutional Hashing (MGCH) that can successfully fuse multisource remote sensing image. Since graph convolutional network (GCN) has been applied as an effective means that expresses and integrates relationships into features, MGCH applies a GCN to explore inherent structural similarity among multiview data, which will help to generate discriminative hash codes. An asymmetric scheme is developed that optimizes the proposed deep model in an end-to-end manner to improve training efficiency. We evaluate the proposed method by fusing two different kinds of RS images, i.e., multispectral (MUL) image and panchromatic (PAN) image. The experimental results on the dual-source RS image data set (DSRSID) show that the proposed MGCH outperforms state-of-the-art multiview hashing methods.

Retinal related diseases are the leading cause of vision loss, and severe retinal lesion causes irreversible damage to vision. Therefore, the automatic methods for retinal diseases detection based on medical images is essential for timely treatment. Considering that manual diagnosis and analysis of medical images require a large number of qualified experts, deep learning can effectively diagnosis and locate critical biomarkers. In this paper, we present a novel model by jointly optimize the cycle generative adversarial network (CycleGAN) and the convolutional neural network (CNN) to detect retinal diseases and localize lesion areas with limited training data. The CycleGAN with cycle consistency can generate more realistic and reliable images. The discriminator and the generator achieve a local optimal solution in an adversarial manner, and the generator and the classifier are in a cooperative manner to distinguish the domain of input images. A novel res-guided sampling block is proposed by combining learnable residual features and pixel-adaptive convolutions. A res-guided U-Net is constructed as the generator by substituting the traditional convolution with the res-guided sampling blocks. Our model achieve superior classification and localization performance on LAG, Ichallenge-PM and Ichallenge-AMD datasets. With clear localization for lesion areas, the competitive results reveal great potentials of the joint optimization network. The source code is available at https://github.com/jizexuan/JointOptmization.

Choroidal neovascularization (CNV) volume prediction has an important clinical significance to predict the therapeutic effect and schedule the follow-up. In this paper, we propose a Lesion Attention Maps-Guided Network (LamNet) to automatically predict the CNV volume of next follow-up visit after therapy based on 3-dimentional spectral-domain optical coherence tomography (SD-OCT) images. In particular, the backbone of LamNet is a 3D convolutional neural network (3D-CNN). In order to guide the network to focus on the local CNV lesion regions, we use CNV attention maps generated by an attention map generator to produce the multi-scale local context features. Then, the multi-scale of both local and global feature maps are fused to achieve the high-precision CNV volume prediction. In addition, we also design a synergistic multi-task predictor, in which a trend-consistent loss ensures that the change trend of the predicted CNV volume is consistent with the real change trend of the CNV volume. The experiments include a total of 541 SD-OCT cubes from 68 patients with two types of CNV captured by two different SD-OCT devices. The results demonstrate that LamNet can provide the reliable and accurate CNV volume prediction, which would further assist the clinical diagnosis and design the treatment options.

Deep hashing has greatly improved retrieval performance with the powerful learning capability of deep neural network. However, deep unsupervised hashing can hardly achieve impressive performance due to the lack of the semantic supervision. This letter proposes Unsupervised Discriminative Deep Hashing (UD2H) to fulfill this gap. UD2H is formulated to jointly perform hash code learning and clustering, and trained in an asymmetric manner to improve the efficiency. The cluster labels supervise the training of deep model to enable hash code discriminative. Based on the outputs of the deep model, UD2H adaptively constructs a similarity graph that considers the local and global structures. Experiments on three benchmark datasets show that the proposed UD$^2$H outperforms the state-of-the-art unsupervised deep hashing methods.

This article explores the relationships between image element pairs and label pairs and extends label diffusion to label pair diffusion for the interactive image segmentation task. Compared with label diffusion, more accurate relationships between unlabeled and labeled data can be captured on a tensor product graph (TPG) by using higher order information, and more complex interactions among image elements and finer relationships between image element pairs and label pairs are explored in label pair diffusion (LPD) process. We first establish a prior label estimation framework to measure the label pair prior probability. Then, a probability learning process on TPG is designed to smooth the label prior. The learning process is equivalent to an iterative LPD process on the original graph, which makes the proposed algorithm maintain computational efficiency. Finally, the unary label probabilities can be obtained by a total-probability-theorem-based conversion from the binary relationships. Experiments on popular segmentation data sets demonstrate the superior performance of the proposed method.

This paper presents an interactive image segmen-tation algorithm, in which the segmentation problem is formulated as a global manifold learning process. Based on the principle that the label of each element depends on the influence of all the elements in the image, we extend the conventional local neighborhood or the long range regional relationships to the global relationships over the whole image. A probabilistic framework is established to measure the global effect of each element on all other elements. Based on two different manifold learning styles, the semi-global manifold learning (SGML) and the fully-global manifold learning (FGML) algorithms are proposed to capture the global geometry structure of the data manifold. SGML learns the intrinsic effects of each element separately, equivalent to a matrix diffusion process on an affinity graph. FGML learns the intrinsic effects of all elements together, equivalent to a label pair diffusion process on a higher-order tensor product graph. The global manifold learning helps to overcome the low contrast, weak boundary and texture problems. Extensive experiments on three public interactive segmentation datasets demonstrate the superior performance of the proposed algorithms both in accuracy and efficiency.

Multi-view hashing efficiently integrates multi-view data for learning compact hash codes, and achieves impressive large-scale retrieval performance. In real-world applications, multi-view data are often stored or collected in different locations, where hash code learning is more challenging yet less studied. To fulfill this gap, this paper proposes a novel supervised multi-view distributed hashing (SMvDisH) for hash code learning from multi-view data in a distributed manner. SMvDisH yields the discriminative latent hash codes by joint learning of latent factor model and classifier. With local consistency assumption among neighbor nodes, the distributed learning problem is divided into a set of decentralized sub-problems. The sub-problems can be solved in parallel, and the computational and communication costs are low. Experimental results on three large-scale image datasets demonstrate that SMvDisH achieves competitive retrieval performance and trains faster than state-of-the-art multi-view hashing methods.

Random noise in hyperspectral images (HSIs) may significantly degrade the image quality and further affect the subsequent image applications, such as land cover classification. To improve the performance of the existing classification methods on noisy HSIs, we propose a framework to take full advantages of the superpixel segmentation and the traditional pixel-wise classification methods. First, a novel superpixel model is proposed for HSI segmentation, where a new spectral similarity is defined in wavelet domain to make the superpixel model more robust to random noise; then, a simple but effective fusion strategy is designed to combine the superpixels with the pixel-wise classification results. Experimental results demonstrate the effectiveness of the proposed superpixel model and fusion strategy on noisy HSIs.

Fully polarimetric synthetic aperture radar (PolSAR) images are generally expressed as the complex-valued (CV) matrix, whereas the convolutional neural networks (CNNs) have been successfully utilized for the PolSAR image classification. However, most 2D or 3D CNNs suffer from insufficient exploring the CV features or computationally expensive. To address these issues, this paper presents an efficient complex-valued spatial-scattering separated attention network (CVS3ANet) for PolSAR image classification. The CVS3ANet utilizes CV-3D convolutions to explore the features in both spatial and scattering dimensions of the PolSAR images, and reduces the parameters by factorizing the 3D convolution as a sequential process of 2D spatial convolution followed by 1D scattering convolution. Moreover, a squeeze and fusion attention unit is used to enhance the learning interpretation ability of the network by modeling correlations between channels with respect to attention probability. The experimental results demonstrate that the proposed method can obtain superior results over the state-of-the-art techniques.

Three‐dimensional (3D) driver pose estimation is a promising and challenging problem for computer–human interaction. Recently convolutional neural networks have been introduced into 3D pose estimation, but these methods have the problem of slow running speed and are not suitable for driving scenario. In this study, the proposed method is based on two types of inputs, infrared image and point cloud obtained from time‐of‐flight camera. The authors propose a joint 2D–3D network incorporating image‐based and point‐based feature to promote the performance of 3D human pose estimation and run on a high speed. For point cloud with invalid points, the authors first do preprocess and then design a denoising module to handle this problem. Experiments on private driver data set and public Invariant‐Top View data set show that the proposed method achieves efficient and competitive performance on 3D human pose estimation.

As one of the most critical characteristics in advanced stage of non-exudative Age-related Macular Degeneration (AMD), Geographic Atrophy (GA) is one of the significant causes of sustained visual acuity loss. Automatic localization of retinal regions affected by GA is a fundamental step for clinical diagnosis. In this paper, we present a novel weakly supervised model for GA segmentation in Spectral-Domain Optical Coherence Tomography (SD-OCT) images. A novel Multi-Scale Class Activation Map (MS-CAM) is proposed to highlight the discriminatory significance regions in localization and detail descriptions. To extract available multi-scale features, we design a Scaling and UpSampling (SUS) module to balance the information content between features of different scales. To capture more discriminative features, an Attentional Fully Connected (AFC) module is proposed by introducing the attention mechanism into the fully connected operations to enhance the significant informative features and suppress less useful ones. Based on the location cues, the final GA region prediction is obtained by the projection segmentation of MS-CAM. The experimental results on two independent datasets demonstrate that the proposed weakly supervised model outperforms the conventional GA segmentation methods and can produce similar or superior accuracy comparing with fully supervised approaches. The source code has been released and is available on GitHub: https://github.com/ jizexuan/Multi-Scale-Class-Activation-Map-Tensorflow.

The presence of hyperreflective foci (HFs) is related to retinal disease progression, and the quantity has proven to be a prognostic factor of visual and anatomical outcome in various retinal diseases. However, lack of efficient quantitative tools for evaluating the HFs has deprived ophthalmologist of assessing the volume of HFs. For this reason, we propose an automated quantification algorithm to segment and quantify HFs in spectral domain optical coherence tomography (SD-OCT). The proposed algorithm consists of two parallel processes namely: region of interest (ROI) generation and HFs estimation. To generate the ROI, we use morphological reconstruction to obtain the reconstructed image and histogram constructed for data distributions and clustering. In parallel, we estimate the HFs by extracting the extremal regions from the connected regions obtained from a component tree. Finally, both the ROI and the HFs estimation process are merged to obtain the segmented HFs. The proposed algorithm was tested on 40 3D SD-OCT volumes from 40 patients diagnosed with non-proliferative diabetic retinopathy (NPDR), proliferative diabetic retinopathy (PDR), and diabetic macular edema (DME). The average dice similarity coefficient (DSC) and correlation coefficient (r) are 69.70%, 0.99 for NPDR, 70.31%, 0.99 for PDR, and 71.30%, 0.99 for DME, respectively. The proposed algorithm can provide ophthalmologist with good HFs quantitative information, such as volume, size, and location of the HFs.

We present an image projection network (IPN), which is a novel end-to-end architecture and can achieve 3D-to-2D image segmentation in optical coherence tomography angiography (OCTA) images. Our key insight is to build a projection learning module (PLM) which uses a unidirectional pooling layer to conduct effective features selection and dimension reduction concurrently. By combining multiple PLMs, the proposed network can input 3D OCTA data, and output 2D segmentation results such as retinal vessel segmentation. It provides a new idea for the quantification of retinal indicators: without retinal layer segmentation and without projection maps. We tested the performance of our network for two crucial retinal image segmentation issues: retinal vessel (RV) segmentation and foveal avascular zone (FAZ) segmentation. The experimental results on 316 OCTA volumes demonstrate that the IPN is an effective implementation of 3D-to-2D segmentation networks, and the uses of multi-modality information and volumetric information make IPN perform better than the baseline methods.

3D driver pose estimation is a promising and challenging problem for computer-human interaction. Recently convolutional neural networks (CNNs) have been introduced into 3D pose estimation, but these methods have the problem of slow running speed and are not suitable for driving scenario. In this paper, our method is based on two type of inputs, IR image and point cloud obtained from TOF camera. We propose a Joint 2D-3D network incorporating image-based and point-based feature to promote the performance of 3D human pose estimation and run in a high speed. For point cloud with invalid points, we firstly do preprocess and then design a denoising module to handle this problem. Experiments on private Driver dataset and public ITOP dataset show that our method achieves efficient and competitive performance on 3D human pose estimation.

This paper presents an interactive image segmentation approach in which we formulate segmentation as a probabilistic estimation problem based on the prior user intention. Instead of directly measuring the relationship between pixels and labels, we first estimate the distances between pixel pairs and label pairs using a probabilistic framework. Then, binary probabilities with label pairs are naturally converted to unary probabilities with labels. The higher order relationship helps improve the robustness to user inputs. To improve segmentation accuracy, a likelihood learning framework is proposed to fuse the region and the boundary information of the image by imposing a smoothing constraint on the unary potentials. Furthermore, we establish an equivalence relationship between likelihood learning and likelihood diffusion and propose an iterative diffusion-based optimization strategy to maintain computational efficiency. Experiments on the Berkeley segmentation data set and Microsoft GrabCut database demonstrate that the proposed method can obtain better performance than the state-of-the-art methods.

Retinal swelling due to the accumulation of fluid is associated with the most vision-threatening retinal diseases. Optical coherence tomography (OCT) is the current standard of care in assessing the presence and quantity of retinal fluid and image-guided treatment management. Deep learning methods have made their impact across medical imaging, and many retinal OCT analysis methods have been proposed. However, it is currently not clear how successful they are in interpreting the retinal fluid on OCT, which is due to the lack of standardized benchmarks. To address this, we organized a challenge RETOUCH in conjunction with MICCAI 2017, with eight teams participating. The challenge consisted of two tasks: fluid detection and fluid segmentation. It featured for the first time: all three retinal fluid types, with annotated images provided by two clinical centers, which were acquired with the three most common OCT device vendors from patients with two different retinal diseases. The analysis revealed that in the detection task, the performance on the automated fluid detection was within the inter-grader variability. However, in the segmentation task, fusing the automated methods produced segmentations that were superior to all individual methods, indicating the need for further improvements in the segmentation performance.

Accurate image segmentation is an important issue in image processing, where unsupervised clustering models play an important part and have been proven to be effective. However, most clustering methods suffer from limited segmentation accuracy without considering spatial information or bounded support region for practical data. In this paper, a bounded spatial probability based fuzzy clustering algorithm is proposed for image segmentation. A bounded distribution to fit the bounded data is utilized and a new conditional probability is constructed based on the immediate neighboring probabilities. Then a parameter-free mean template is presented to impose the spatial information more precisely. Finally, the negative logarithmical conditional probability is utilized as the dissimilarity function to describe the observed data. We evaluated our algorithm against several state-of-the-art segmentation approaches on brain magnetic resonance images. Our results suggest that the proposed algorithm is more robust to noise and textures, and can produce more accurate segmentation results.

This paper presents an interactive image segmentation approach where the segmentation problem is formulated as a probabilistic estimation manner. Instead of measuring the distances between unseeded pixels and seeded pixels, we measure the similarities between pixel pairs and seed pairs to improve the robustness to the seeds. The unary prior probability of each pixel belonging to the foreground F and background B can be effectively estimated based on the similarities with label pairs (F, F),(F, B),(B, F) and (B, B). Then a likelihood learning framework is proposed to fuse the region and boundary information of the image by imposing the smoothing constraint on the unary potentials. Experiments on challenging data sets demonstrate that the proposed method can obtain better performance than state-of-the-art methods.

The combination of pixel and superpixel has been widely utilized in the interactive segmentation methods to overcome the sensitivity to the seeds' quantity and quality. However, because of the introduction of more variables and variables' interactions, the pixel-superpixel combination methods are still limited to the segmentation accuracy and computational complexity. To solve these problems, in this paper, we propose an interactive multilabel image segmentation method. In the proposed segmentation model, the multilayer relationships among the pixel layer, superpixel layer, and label layer are fused by the Markov random field framework to further improve the segmentation accuracy. During the optimization stage, the parallel partial optimality strategy is utilized to effectively solve the multilabel submodular energy function. Experimental results on challenging data sets demonstrate the competitiveness of the proposed method comparing with several state-of-the-art interactive algorithms.

Fuzzy clustering algorithms have been widely used in brain magnetic resonance (MR) image segmentation. However, due to the existence of noise and intensity inhomogeneity, many segmentation algorithms suffer from limited accuracy. In this paper, we propose a fuzzy clustering algorithm with robust spatially constraint for accurate and robust brain MR image segmentation. A novel spatial factor is proposed by incorporating the spatial information amongst neighborhood pixels with a simple metric. A new weight factor, which utilizes the intensity information of the original image, is constructed to filter the posterior and prior probabilities in the spatial neighborhood. The proposed method can preserve more details and overcome the over-smoothing disadvantage. Finally, the fuzzy objective function is integrated with the bias field estimation model to overcome the intensity inhomogeneity in the image and segment the brain MR images. Experimental results demonstrate that the proposed algorithm can substantially improve the accuracy of brain MR image segmentation.

Accurate brain tissue segmentation from magnetic resonance (MR) images is an essential step in quantitative brain image analysis. However, due to the existence of noise and intensity inhomogeneity in brain MR images, many segmentation algorithms suffer from limited accuracy. In this paper, we assume that the local image data within each voxel's neighborhood satisfy the Gaussian mixture model (GMM), and thus propose the fuzzy local GMM (FLGMM) algorithm for automated brain MR image segmentation. This algorithm estimates the segmentation result that maximizes the posterior probability by minimizing an objective energy function, in which a truncated Gaussian kernel function is used to impose the spatial constraint and fuzzy memberships are employed to balance the contribution of each GMM. We compared our algorithm to state-of-the-art segmentation approaches in both synthetic and clinical data. Our results show that the proposed algorithm can largely overcome the difficulties raised by noise, low contrast, and bias field, and substantially improve the accuracy of brain MR image segmentation.