Multimodal imaging has become an essential tool in clinical ophthalmology, offering complementary perspectives for disease diagnosis. However, current automated diagnostic approaches often fail to fully exploit the rich, complementary information provided by different imaging modalities. In this paper, to advance automated ophthalmic disease diagnosis through effective multimodal data integration, we propose the OphFusionNet, a novel multimodal learning framework based on uncertainty-driven multi-scale multimodal feature fusion. Drawing inspiration from clinical observations that ophthalmic lesions often appear at multiple spatial scales, we design a multi-scale feature fusion module with sparse self-attention (MSFF-SSA). This module captures hierarchical representations while suppressing redundancy, thereby enhancing both the expressiveness and efficiency of the extracted features. To further improve multimodal fusion, we introduce an uncertainty-aware multimodal fusion module with a game-theoretic selection strategy (UMF-GTSS). This component estimates the uncertainty associated with different features and adaptively weights them based on their relative reliability, yielding more robust and trustworthy diagnostic outcomes. To mitigate the tendency to over-rely on dominant modalities and underutilize the informative potential of subordinate ones, we propose a modality distillation strategy (MDS), which leverages multimodal features to guide and refine the learning of single-modal representations. This strategy enhances generalization and boosts the discriminative capacity of each individual modality. The OphFusionNet was evaluated on four publicly available ophthalmic datasets. Extensive experiments demonstrate that our approach achieves superior multimodal integration, resulting in state-of-the-art (SOTA) performance in multimodal ophthalmic disease diagnosis. The code will be available at: https://github.com/wb66715/OphFusionNet.

Choroidal neovascularization (CNV) classification is a fine-grained classification task. Accurate classification of CNV in optical coherence tomography (OCT) images is crucial for clinical treatment. However, image acquisition noise degrades image quality and exacerbates confirmation bias from class imbalance in medical datasets. Moreover, significant inter-class ambiguity in fine-grained categories can misclassify informative samples (e.g., hard samples or minority class samples) when generating pseudo-labels, leading to sub-optimal classifiers. To address these challenges, we propose a difficulty-aware pseudo-label correction network (DPLC-Net). Specifically, we designed a robust feature mining module using feature similarity loss to maintain consistency between generated adversarial and original samples, enabling noise-resistant feature learning. A difficulty-aware pseudo-label correction module mines and corrects potential noisy pseudo-labels to improve classification performance. Finally, to alleviate data bias and leverage all unlabeled samples, we integrated a hybrid consistency and pseudo-labeling module comprising adaptive weighted consistency loss (AWCL) and class-aware dynamic threshold strategy (CDTS). AWCL adaptively learns weights for unlabeled samples, effectively utilizing all unlabeled data through weighted consistency loss. CDTS dynamically adjusts confidence thresholds based on class distribution and model learning status, improving pseudo-label quantity and quality. Experiments on private and public OCT datasets demonstrate that our method outperforms state-of-the-art methods.

Deep learning (DL) requires large amounts of labeled data, which is extremely time-consuming and labor-intensive to obtain for medical image segmentation tasks. Meta-learning focuses on developing learning strategies that enable quick adaptation to new tasks with limited labeled data. However, rich-class medical image segmentation datasets for constructing meta-learning multi-tasks are currently unavailable. In addition, data collected from various healthcare sites and devices may present significant distribution differences, potentially degrading model's performance. In this paper, we propose a task augmentation-based meta-learning method for retinal image segmentation (TAMS) to meet labor-intensive annotation demand. A retinal Lesion Simulation Algorithm (LSA) is proposed to automatically generate multi-class retinal disease datasets with pixel-level segmentation labels, such that meta-learning tasks can be augmented without collecting data from various sources. In addition, a novel simulation function library is designed to control generation process and ensure interpretability. Moreover, a generative simulation network (GSNet) with an improved adversarial training strategy is introduced to maintain high-quality representations of complex retinal diseases. TAMS is evaluated on three different OCT and CFP image datasets, and comprehensive experiments have demonstrated that TAMS achieves superior segmentation performance than state-of-the-art models.

In clinical practice, obtaining a large amount of labeled CNV data is very difficult. Semi-supervised learning can effectively utilize a large amount of unlabeled CNV data. Since CNV has complex features such as blurred and unevenly distributed pixels on the edges, there are differences in the segmentation difficulty between pixels in the same image. Existing semi-supervised segmentation methods do not consider the segmentation difficulty of pixels, which will reduce the segmentation accuracy. To address this problem, we propose a dual difficulty-aware adaptive pseudo-label learning (D2APL) method for semi-supervised CNV segmentation. The proposed dual difficulty awareness includes segmentation difficulty perception of pixels in labeled and unlabeled data. For labeled data, we propose a classification confidence-guided difficulty perception method. For unlabeled data, we propose a model stability-guided difficulty perception method. Finally, we propose a difficulty-aware self-training method to dynamically adjust the threshold of pseudolabels according to the difficulty, thereby improving the utilization of difficult-to-segment pixels in unlabeled data. Experimental results show that our method outperforms the state-of-the-art method in CNV segmentation.

CT and MR are currently the most common imaging techniques for pancreatic cancer diagnosis. Accurate segmentation of the pancreas in CT and MR images can provide significant help in the diagnosis and treatment of pancreatic cancer. Traditional supervised segmentation methods require a large number of labeled CT and MR training data, which is usually time-consuming and laborious. Meanwhile, due to domain shift, traditional segmentation networks are difficult to be deployed on different imaging modality datasets. Cross-domain segmentation can utilize labeled source domain data to assist unlabeled target domains in solving the above problems. In this paper, a cross-domain pancreas segmentation algorithm is proposed based on Moment-Consistent Contrastive Cycle Generative Adversarial Networks (MC-CCycleGAN). MC-CCycleGAN is a style transfer network, in which the encoder of its generator is used to extract features from real images and style transfer images, constrain feature extraction through a contrastive loss, and fully extract structural features of input images during style transfer while eliminate redundant style features. The multi-order central moments of the pancreas are proposed to describe its anatomy in high dimensions and a contrastive loss is also proposed to constrain the moment consistency, so as to maintain consistency of the pancreatic structure and shape before and after style transfer. Multi-teacher knowledge distillation framework is proposed to transfer the knowledge from multiple teachers to a single student, so as to improve the robustness and performance of the student network. The experimental results have demonstrated the superiority of our framework over state-of-the-art domain adaptation methods.

Benign and malignant classification of clustered microcalcifications (MCs) in digital breast tomosynthesis (DBT) is an essential task in computer-aided diagnosis. However, due to the anisotropic resolution of DBT, three-dimensional (3-D) convolutional neural network (CNN)-based methods cannot extract hierarchical features efficiently. Moreover, the sparse distribution of MC points in the cluster makes it difficult for the CNN to extract discriminative structural information for classification. To comprehensively address these challenges, we propose a novel structure-aware hierarchical network (SAH-Net) for benign and malignant classification of clustered MC in a DBT volume. Specifically, the two-dimensional (2-D) group convolution is used to extract intraslice features. The one-to-one correspondence between group convolutions and slices ensures the independence of hierarchical feature extraction. Then, a partial deformable Transformer-based 3-D structural feature learning module is proposed to capture the long-range dependency between MC points in the cluster. We evaluate the proposed method on an in-house dataset with 495 clustered MCs collected from 462 DBT images. Experimental results confirm the validity of our proposed modules. The results also show that the proposed SAH-Net outperforms several other representative methods on this topic, and achieves the best classification result, with an area under the receiver operation curve (AUC) of 86.87%. The implementation of the proposed model is available at https://github.com/sunhaotian130911/SAHNet.

Unsupervised domain adaptation medical image segmentation is aimed to segment unlabeled target domain images with labeled source domain images. However, different medical imaging modalities lead to large domain shift between their images, in which well-trained models from one imaging modality often fail to segment images from anothor imaging modality. In this paper, to mitigate domain shift between source domain and target domain, a style consistency unsupervised domain adaptation image segmentation method is proposed. First, a local phase-enhanced style fusion method is designed to mitigate domain shift and produce locally enhanced organs of interest. Second, a phase consistency discriminator is constructed to distinguish the phase consistency of domain-invariant features between source domain and target domain, so as to enhance the disentanglement of the domain-invariant and style encoders and removal of domain-specific features from the domain-invariant encoder. Third, a style consistency estimation method is proposed to obtain inconsistency maps from intermediate synthesized target domain images with different styles to measure the difficult regions, mitigate domain shift between synthesized target domain images and real target domain images, and improve the integrity of interested organs. Fourth, style consistency entropy is defined for target domain images to further improve the integrity of the interested organ by the concentration on the inconsistent regions. Comprehensive experiments have been performed with an in-house dataset and a publicly available dataset. The experimental results have demonstrated the superiority of our framework over state-of-the-art methods.

Deep convolutional neural networks (CNNs) have been widely used for fundus image classification and have achieved very impressive performance. However, the explainability of CNNs is poor because of their black-box nature, which limits their application in clinical practice. In this paper, we propose a novel method to search for discriminative regions to increase the confidence of CNNs in the classification of features in specific category, thereby helping users understand which regions in an image are important for a CNN to make a particular prediction. In the proposed method, a set of superpixels is selected in an evolutionary process, such that discriminative regions can be found automatically. Many experiments are conducted to verify the effectiveness of the proposed method. The average drop and average increase obtained with the proposed method are 0 and 77.8%, respectively, in fundus image classification, indicating that the proposed method is very effective in identifying discriminative regions. Additionally, several interesting findings are reported: 1) Some superpixels, which contain the evidence used by humans to make a certain decision in practice, can be identified as discriminative regions via the proposed method; 2) The superpixels identified as discriminative regions are distributed in different locations in an image rather than focusing on regions with a specific instance; and 3) The number of discriminative superpixels obtained via the proposed method is relatively small. In other words, a CNN model can employ a small portion of the pixels in an image to increase the confidence for a specific category.

Breast lesion segmentation from ultrasound images is essential in computer-aided breast cancer diagnosis. To alleviate the problems of blurry lesion boundaries and irregular morphologies, common practices combine CNN and attention to integrate global and local information. However, previous methods use two independent modules to extract global and local features separately, such feature-wise inflexible integration ignores the semantic gap between them, resulting in representation redundancy/insufficiency and undesirable restrictions in clinic practices. Moreover, medical images are highly similar to each other due to the imaging methods and human tissues, but the captured global information by transformer-based methods in the medical domain is limited within images, the semantic relations and common knowledge across images are largely ignored. To alleviate the above problems, in the neighbor view, this paper develops a pixel neighbor representation learning method (NeighborNet) to flexibly integrate global and local context within and across images for lesion morphology and boundary modeling. Concretely, we design two neighbor layers to investigate two properties (i.e., number and distribution) of neighbors. The neighbor number for each pixel is not fixed but determined by itself. The neighbor distribution is extended from one image to all images in the datasets. With the two properties, for each pixel at each feature level, the proposed NeighborNet can evolve into the transformer or degenerate into the CNN for adaptive context representation learning to cope with the irregular lesion morphologies and blurry boundaries. The state-of-the-art performances on three ultrasound datasets prove the effectiveness of the proposed NeighborNet.

Optical coherence tomography (OCT) images are inevitably affected by speckle noise because OCT is based on low-coherence interference. Multi-frame averaging is one of the effective methods to reduce speckle noise. Before averaging, the misalignment between images must be calibrated. In this paper, in order to reduce misalignment between images caused during the acquisition, a novel multi-scale fusion and Transformer based (MsFTMorph) method is proposed for deformable retinal OCT image registration. The proposed method captures global connectivity and locality with convolutional vision transformer and also incorporates a multi-resolution fusion strategy for learning the global affine transformation. Comparative experiments with other state-of-the-art registration methods demonstrate that the proposed method achieves higher registration accuracy. Guided by the registration, subsequent multi-frame averaging shows better results in speckle noise reduction. The noise is suppressed while the edges can be preserved. In addition, our proposed method has strong cross-domain generalization, which can be directly applied to images acquired by different scanners with different modes.

Skin wound segmentation in photographs allows non-invasive analysis of wounds that supports dermatological diagnosis and treatment. In this paper, we propose a novel feature augment network (FANet) to achieve automatic segmentation of skin wounds, and design an interactive feature augment network (IFANet) to provide interactive adjustment on the automatic segmentation results. The FANet contains the edge feature augment (EFA) module and the spatial relationship feature augment (SFA) module, which can make full use of the notable edge information and the spatial relationship information be-tween the wound and the skin. The IFANet, with FANet as the backbone, takes the user interactions and the initial result as inputs, and outputs the refined segmentation result. The pro-posed networks were tested on a dataset composed of miscellaneous skin wound images, and a public foot ulcer segmentation challenge dataset. The results indicate that the FANet gives good segmentation results while the IFANet can effectively improve them based on simple marking. Comprehensive comparative experiments show that our proposed networks outperform some other existing automatic or interactive segmentation methods, respectively.

Choroidal neovascularization (CNV) is a typical symptom of age-related macular degeneration (AMD) and is one of the leading causes for blindness. Accurate segmentation of CNV and detection of retinal layers are critical for eye disease diagnosis and monitoring. In this paper, we propose a novel graph attention U-Net (GA-UNet) for retinal layer surface detection and CNV segmentation in optical coherence tomography (OCT) images. Due to retinal layer deformation caused by CNV, it is challenging for existing models to segment CNV and detect retinal layer surfaces with the correct topological order. We propose two novel modules to address the challenge. The first module is a graph attention encoder (GAE) in a U-Net model that automatically integrates topological and pathological knowledge of retinal layers into the U-Net structure to achieve effective feature embedding. The second module is a graph decorrelation module (GDM) that takes reconstructed features by the decoder of the U-Net as inputs, it then decorrelates and removes information unrelated to retinal layer for improved retinal layer surface detection. In addition, we propose a new loss function to maintain the correct topological order of retinal layers and the continuity of their boundaries. The proposed model learns graph attention maps automatically during training and performs retinal layer surface detection and CNV segmentation simultaneously with the attention maps during inference. We evaluated the proposed model on our private AMD dataset and another public dataset. Experiment results show that the proposed model outperformed the competing methods for retinal layer surface detection and CNV segmentation and achieved new state of the arts on the datasets.

Accurate segmentation of retinal images can assist ophthalmologists to determine the degree of retinopathy and diagnose other systemic diseases. However, the structure of the retina is complex, and different anatomical structures often affect the segmentation of fundus lesions. In this paper, a new segmentation strategy called a dual stream segmentation network embedded into a conditional generative adversarial network is proposed to improve the accuracy of retinal lesion segmentation. First, a dual stream encoder is proposed to utilize the capabilities of two different networks and extract more feature information. Second, a multiple level fuse block is proposed to decode the richer and more effective features from the two different parallel encoders. Third, the proposed network is further trained in a semi-supervised adversarial manner to leverage from labeled images and unlabeled images with high confident pseudo labels, which are selected by the dual stream Bayesian segmentation network. An annotation discriminator is further proposed to reduce the negativity that prediction tends to become increasingly similar to the inaccurate predictions of unlabeled images. The proposed method is cross-validated in 384 clinical fundus fluorescein angiography images and 1040 optical coherence tomography images. Compared to state-of-the-art methods, the proposed method can achieve better segmentation of retinal capillary non-perfusion region and choroidal neovascularization.

Quantitative measurements of corneal sub-basal nerves are biomarkers for many ocular surface disorders and are also important for early diagnosis and assessment of progression of neurodegenerative diseases. This paper aims to develop an automatic method for nerve fiber segmentation from in vivo corneal confocal microscopy (CCM) images, which is fundamental for nerve morphology quantification. A novel multi-discriminator adversarial convolutional network (MDACN) is proposed, where both the generator and the two discriminators emphasize multi-scale feature representations. The generator is a U-shaped fully convolutional network with multi-scale split and concatenate blocks, and the two discriminators have different effective receptive fields, sensitive to features of different scales. A novel loss function is also proposed which enables the network to pay more attention to thin fibers. The MDACN framework was evaluated on four datasets. Experiment results show that our method has excellent segmentation performance for corneal nerve fibers and outperforms some state-of-the-art methods.

Raw optical coherence tomography (OCT) images typically are of low quality because speckle noise blurs retinal structures, severely compromising visual quality and degrading performances of subsequent image analysis tasks. In our previous study (Ma et al., 2018), we have developed a Conditional Generative Adversarial Network (cGAN) for speckle noise removal in OCT images collected by several commercial OCT scanners, which we collectively refer to as scanner T. In this paper, we improve the cGAN model and apply it to our in-house OCT scanner (scanner B) for speckle noise suppression. The proposed model consists of two steps: 1) We train a Cycle-Consistent GAN (CycleGAN) to learn style transfer between two OCT image datasets collected by different scanners. The purpose of the CycleGAN is to leverage the ground truth dataset created in our previous study. 2) We train a mini-cGAN model based on the PatchGAN mechanism with the ground truth dataset to suppress speckle noise in OCT images. After training, we first apply the CycleGAN model to convert raw images collected by scanner B to match the style of the images from scanner T, and subsequently use the mini-cGAN model to suppress speckle noise in the style transferred images. We evaluate the proposed method on a dataset collected by scanner B. Experimental results show that the improved model outperforms our previous method and other state-of-the-art models in speckle noise removal, retinal structure preservation and contrast enhancement.

Learning how to capture long-range dependencies and restore spatial information of down-sampled feature maps are the basis of the encoder-decoder structure networks in medical image segmentation. U-Net based methods use feature fusion to alleviate these two problems, but the global feature extraction ability and spatial information recovery ability of U-Net are still insufficient. In this paper, we propose a Global Feature Reconstruction (GFR) module to efficiently capture global context features and a Local Feature Reconstruction (LFR) module to dynamically up-sample features, respectively. For the GFR module, we first extract the global features with category representation from the feature map, then use the different level global features to reconstruct features at each location. The GFR module establishes a connection for each pair of feature elements in the entire space from a global perspective and transfers semantic information from the deep layers to the shallow layers. For the LFR module, we use low-level feature maps to guide the up-sampling process of high-level feature maps. Specifically, we use local neighborhoods to reconstruct features to achieve the transfer of spatial information. Based on the encoder-decoder architecture, we propose a Global and Local Feature Reconstruction Network (GLFRNet), in which the GFR modules are applied as skip connections and the LFR modules constitute the decoder path. The proposed GLFRNet is applied to four different medical image segmentation tasks and achieves state-of-the-art performance.

Drusen is considered as the landmark for diagnosis of AMD and important risk factor for the development of AMD. Therefore, accurate segmentation of drusen in retinal OCT images is crucial for early diagnosis of AMD. However, drusen segmentation in retinal OCT images is still very challenging due to the large variations in size and shape of drusen, blurred boundaries, and speckle noise interference. Moreover, the lack of OCT dataset with pixel-level annotation is also a vital factor hindering the improvement of drusen segmentation accuracy. To solve these problems, a novel multi-scale transformer global attention network (MsTGANet) is proposed for drusen segmentation in retinal OCT images. In MsTGANet, which is based on U-Shape architecture, a novel multi-scale transformer non-local (MsTNL) module is designed and inserted into the top of encoder path, aiming at capturing multi-scale non-local features with long-range dependencies from different layers of encoder. Meanwhile, a novel multi-semantic global channel and spatial joint attention module (MsGCS) between encoder and decoder is proposed to guide the model to fuse different semantic features, thereby improving the model's ability to learn multi-semantic global contextual information. Furthermore, to alleviate the shortage of labeled data, we propose a novel semi-supervised version of MsTGANet (Semi-MsTGANet) based on pseudo-labeled data augmentation strategy, which can leverage a large amount of unlabeled data to further improve the segmentation performance. Finally, comprehensive experiments are conducted to evaluate the performance of the proposed MsTGANet and Semi-MsTGANet. The experimental results show that our proposed methods achieve better segmentation accuracy than other state-of-the-art CNN-based methods.

Recently, many methods based on hand-designed convolutional neural networks (CNNs) have achieved promising results in automatic retinal vessel segmentation. However, these CNNs remain constrained in capturing retinal vessels in complex fundus images. To improve their segmentation performance, these CNNs tend to have many parameters, which may lead to overfitting and high computational complexity. Moreover, the manual design of competitive CNNs is time-consuming and requires extensive empirical knowledge. Herein, a novel automated design method, called Genetic U-Net, is proposed to generate a U-shaped CNN that can achieve better retinal vessel segmentation but with fewer architecture-based parameters, thereby addressing the above issues. First, we devised a condensed but flexible search space based on a U-shaped encoder-decoder. Then, we used an improved genetic algorithm to identify better-performing architectures in the search space and investigated the possibility of finding a superior network architecture with fewer parameters. The experimental results show that the architecture obtained using the proposed method offered a superior performance with less than 1% of the number of the original U-Net parameters in particular and with significantly fewer parameters than other state-of-the-art models. Furthermore, through in-depth investigation of the experimental results, several effective operations and patterns of networks to generate superior retinal vessel segmentations were identified. The codes of this work are available at https://github.com/96jhwei/Genetic-U-Net.

Retinopathy of prematurity (ROP) is a retinal disease which frequently occurs in premature babies with low birth weight and is considered as one of the major preventable causes of childhood blindness. Although automatic and semi-automatic diagnoses of ROP based on fundus image have been researched, most of the previous studies focused on plus disease detection and ROP screening. There are few studies focusing on ROP staging, which is important for the severity evaluation of the disease. To be consistent with clinical 5-level ROP staging, a novel and effective deep neural network based 5-level ROP staging network is proposed, which consists of multi-stream based parallel feature extractor, concatenation based deep feature fuser and clinical practice based ordinal classifier. First, the three-stream parallel framework including ResNet18, DenseNet121 and EfficientNetB2 is proposed as the feature extractor, which can extract rich and diverse high-level features. Second, the features from three streams are deeply fused by concatenation and convolution to generate a more effective and comprehensive feature. Finally, in the classification stage, an ordinal classification strategy is adopted, which can effectively improve the ROP staging performance. The proposed ROP staging network was evaluated with per-image and per-examination strategies. For per-image ROP staging, the proposed method was evaluated on 635 retinal fundus images from 196 examinations, including 303 Normal, 26 Stage 1, 127 Stage 2, 106 Stage 3, 61 Stage 4 and 12 Stage 5, which achieves 0.9055 for weighted recall, 0.9092 for weighted precision, 0.9043 for weighted F1 score, 0.9827 for accuracy with 1 (ACC1) and 0.9786 for Kappa, respectively. While for per-examination ROP staging, 1173 examinations with a 4-fold cross validation strategy were used to evaluate the effectiveness of the proposed method, which prove the validity and advantage of the proposed method.

Speckle noise is the main cause of poor optical coherence tomography (OCT) image quality. Convolutional neural networks (CNNs) have shown remarkable performances for speckle noise reduction. However, speckle noise denoising still meets great challenges because the deep learning-based methods need a large amount of labeled data whose acquisition is time-consuming or expensive. Besides, many CNNs-based methods design complex structure based networks with lots of parameters to improve the denoising performance, which consume hardware resources severely and are prone to overfitting. To solve these problems, we propose a novel semi-supervised learning based method for speckle noise denoising in retinal OCT images. First, to improve the model's ability to capture complex and sparse features in OCT images, and avoid the problem of a great increase of parameters, a novel capsule conditional generative adversarial network (Caps-cGAN) with small number of parameters is proposed to construct the semi-supervised learning system. Then, to tackle the problem of retinal structure information loss in OCT images caused by lack of detailed guidance during unsupervised learning, a novel joint semi-supervised loss function composed of unsupervised loss and supervised loss is proposed to train the model. Compared with other state-of-the-art methods, the proposed semi-supervised method is suitable for retinal OCT images collected from different OCT devices and can achieve better performance even only using half of the training data.

Medical image registration can be used for studying longitudinal and cross-sectional data, quantitatively monitoring disease progression and guiding computer assisted diagnosis and treatments. However, deformable registration which enables more precise and quantitative comparison has not been well developed for retinal optical coherence tomography (OCT) images. This paper proposes a new 3D registration approach for retinal OCT data called OCTRexpert. To the best of our knowledge, the proposed algorithm is the first full 3D registration approach for retinal OCT images which can be applied to longitudinal OCT images for both normal and serious pathological subjects. In this approach, a pre-processing method is first performed to remove eye motion artifact and then a novel design-detection-deformation strategy is applied for the registration. In the design step, a couple of features are designed for each voxel in the image. In the detection step, active voxels are selected and the point-to-point correspondences between the subject and template images are established. In the deformation step, the image is hierarchically deformed according to the detected correspondences in multi-resolution. The proposed method is evaluated on a dataset with longitudinal OCT images from 20 healthy subjects and 4 subjects diagnosed with serious Choroidal Neovascularization (CNV). Experimental results show that the proposed registration algorithm consistently yields statistically significant improvements in both Dice similarity coefficient and the average unsigned surface error compared with the other registration methods.

Accurate and automatic segmentation of medical images is a crucial step for clinical diagnosis and analysis. The convolutional neural network (CNN) approaches based on the U-shape structure have achieved remarkable performances in many different medical image segmentation tasks. However, the context information extraction capability of single stage is insufficient in this structure, due to the problems such as imbalanced class and blurred boundary. In this paper, we propose a novel Context Pyramid Fusion Network (named CPFNet) by combining two pyramidal modules to fuse global/multi-scale context information. Based on the U-shape structure, we first design multiple global pyramid guidance (GPG) modules between the encoder and the decoder, aiming at providing different levels of global context information for the decoder by reconstructing skip-connection. We further design a scale-aware pyramid fusion (SAPF) module to dynamically fuse multi-scale context information in high-level features. These two pyramidal modules can exploit and fuse rich context information progressively. Experimental results show that our proposed method is very competitive with other state-of-the-art methods on four different challenging tasks, including skin lesion segmentation, retinal linear lesion segmentation, multi-class segmentation of thoracic organs at risk and multi-class segmentation of retinal edema lesions.

In this paper, a hierarchical image matting model is proposed to extract blood vessels from fundus images. More specifically, a hierarchical strategy is integrated into the image matting model for blood vessel segmentation. Normally the matting models require a user specified trimap, which separates the input image into three regions: the foreground, background and unknown regions. However, creating a user specified trimap is laborious for vessel segmentation tasks. In this paper, we propose a method that first generates trimap automatically by utilizing region features of blood vessels, then applies a hierarchical image matting model to extract the vessel pixels from the unknown regions. The proposed method has low calculation time and outperforms many other state-of-art supervised and unsupervised methods. It achieves a vessel segmentation accuracy of 96.0%, 95.7% and 95.1% in an average time of 10.72s, 15.74s and 50.71s on images from three publicly available fundus image datasets DRIVE, STARE, and CHASE DB1, respectively.

OBJECTIVE: Nonrigid image registration with high accuracy and efficiency remains a challenging task for medical image analysis. In this paper, we present the spatially region-weighted correlation ratio (SRWCR) as a novel similarity measure to improve the registration performance. METHODS: SRWCR is rigorously deduced from a three-dimension joint probability density function combining the intensity channels with an extra spatial information channel. SRWCR estimates the optimal functional dependence between the intensities for each spatial bin, in which the spatial distribution modeled by a cubic B-spline function is used to differentiate the contribution of voxels. We also analytically derive the gradient of SRWCR with respect to the transformation parameters and optimize it using a quasi-Newton approach. Furthermore, we propose a GPU-based parallel mechanism to accelerate the computation of SRWCR and its derivatives. RESULTS: The experiments on synthetic images, public four-dimensional thoracic computed tomography (CT) dataset, retinal optical coherence tomography data, and clinical CT and positron emission tomography images confirm that SRWCR significantly outperforms some state-of-the-art techniques such as spatially encoded mutual information and Robust PaTch-based cOrrelation Ration. CONCLUSION: This study demonstrates the advantages of SRWCR in tackling the practical difficulties due to distinct intensity changes, serious speckle noise, or different imaging modalities. SIGNIFICANCE: The proposed registration framework might be more reliable to correct the nonrigid deformations and more potential for clinical applications.

Optical Coherence Tomography (OCT) is beco-ming one of the most important modalities for the noninvasive assessment of retinal eye diseases. As the number of acquired OCT volumes increases, automating the OCT image analysis is becoming increasingly relevant. In this paper, we propose a surrogate-assisted classification method to classify retinal OCT images automatically based on convolutional neural networks (CNNs). Image denoising is first performed to reduce the noise. Thresholding and morphological dilation are applied to extract the masks. The denoised images and the masks are then employed to generate a lot of surrogate images, which are used to train the CNN model. Finally, the prediction for a test image is determined by the average of the outputs from the trained CNN model on the surrogate images. The proposed method has been evaluated on different databases. The results (AUC of 0.9783 in the local database and AUC of 0.9856 in the Duke database) show that the proposed method is a very promising tool for classifying the retinal OCT images automatically.

In this paper, an automatic method is reported for simultaneously segmenting layers and fluid in 3-D OCT retinal images of subjects suffering from central serous retinopathy. To enhance contrast between adjacent layers, multiscale bright and dark layer detection filters are proposed. Due to appearance of serous fluid or pigment epithelial detachment caused fluid, contrast between adjacent layers is often reduced, and also large morphological changes are caused. In addition, 24 features are designed for random forest classifiers. Then, 8 coarse surfaces are obtained based on the trained random forest classifiers. Finally, a hypergraph is constructed based on the smoothed image and the layer structure detection responses. A modified live wire algorithm is proposed to accurately detect surfaces between retinal layers, even though OCT images with fluids are of low contrast and layers are largely deformed. The proposed method was evaluated on 48 spectral domain OCT images with central serous retinopathy. The experimental results showed that the proposed method outperformed the state-of-art methods with regard to layers and fluid segmentation.

The segmentation of lungs with severe pathology is a nontrivial problem in the clinical application. Due to complex structures, pathological changes, individual differences, and low image quality, accurate lung segmentation in clinical 3-D computed tomography (CT) images is still a challenging task. To overcome these problems, a novel dictionary-based approach is introduced to automatically segment pathological lungs in 3-D low-dose CT images. Sparse shape composition is integrated with the eigenvector space shape prior model, called eigenspace sparse shape composition, to reduce local shape reconstruction error caused by the weak and misleading appearance prior information. To initialize the shape model, a landmark recognition method based on discriminative appearance dictionary is introduced to handle lesions and local details. Furthermore, a new vertex search strategy based on the gradient vector flow field is also proposed to drive the shape deformation to the target boundary. The proposed algorithm is tested on 78 3-D low-dose CT images with lung tumors. Compared to the state-of-the-art methods, the proposed approach can robustly and accurately detect pathological lung surface.

Retinal layer segmentation in optical coherence tomography (OCT) images is crucial for the diagnosis and study of retinal diseases. Graph-based methods are commonly used in layer segmentation. However, most of these methods require a lot of human efforts for determining an appropriate model to compute good edge weights. In this paper, we propose a novel automatic method for segmenting retinal layers in macular OCT images. Specially, we propose a new fully convolutional deep learning architecture with a side output layer to directly learn optimal graph-edge weights from raw pixels. The architecture can automatically learn multi-scale and multi-level features to generate accurate boundary probabilities as good edge weights without hand-crafted appropriate models. The boundaries are finalized by using graph segmentation method. The proposed method is evaluated on a dataset with 130 OCT B-scans. The experimental results show the mean absolute boundary positioning differences are 1.48±0.34 pixel.

Age-related macular degeneration is one of the main causes of blindness. However, the internal structures of retinas are complex and difficult to be recognized due to the occurrence of neovascularization. Traditional surface detection methods may fail in the layer segmentation. In this paper, a supervised method is reported for simultaneously segmenting layers and neovascularization. Three spatial features, seven gray-level-based features, and 14 layer-like features are extracted for the neural network classifier. The coarse surfaces of different optical coherence tomography (OCT) images can thus be found. To describe and enhance retinal layers with different thicknesses and abnormalities, multi-scale bright and dark layer detection filters are introduced. A constrained graph search algorithm is also proposed to accurately detect retinal surfaces. The weights of nodes in the graph are computed based on these layer-like responses. The proposed method was evaluated on 42 spectral-domain OCT images with age-related macular degeneration. The experimental results show that the proposed method outperforms state-of-the-art methods.

Automated optic disk (OD) detection plays an important role in developing a computer aided system for eye diseases. In this paper, we propose an algorithm for the OD detection based on structured learning. A classifier model is trained based on structured learning. Then, we use the model to achieve the edge map of OD. Thresholding is performed on the edge map, thus a binary image of the OD is obtained. Finally, circle Hough transform is carried out to approximate the boundary of OD by a circle. The proposed algorithm has been evaluated on three public datasets and obtained promising results. The results (an area overlap and Dices coefficients of 0.8605 and 0.9181, respectively, an accuracy of 0.9777, and a true positive and false positive fraction of 0.9183 and 0.0102) show that the proposed method is very competitive with the state-of-the-art methods and is a reliable tool for the segmentation of OD.

We propose a new multiscale rotation-invariant convolutional neural network (MRCNN) model for classifying various lung tissue types on high-resolution computed tomography. MRCNN employs Gabor-local binary pattern that introduces a good property in image analysis-invariance to image scales and rotations. In addition, we offer an approach to deal with the problems caused by imbalanced number of samples between different classes in most of the existing works, accomplished by changing the overlapping size between the adjacent patches. Experimental results on a public interstitial lung disease database show a superior performance of the proposed method to state of the art.

Branch retinal artery occlusion (BRAO) is an ocular emergency, which could lead to blindness. Quantitative analysis of the BRAO region in the retina is necessary for the assessment of the severity of retinal ischemia. In this paper, a fully automatic framework was proposed to segment BRAO regions based on 3D spectral-domain optical coherence tomography (SD-OCT) images. To the best of our knowledge, this is the first automatic 3D BRAO segmentation framework. First, the input 3D image is automatically classified into BRAO of acute phase and BRAO of chronic phase or normal retina using an AdaBoost classifier based on combining local structural, intensity, textural features with our new feature distribution analyzing strategy. Then, BRAO regions of acute phase and chronic phase are segmented separately. A thickness model is built to segment BRAO in the chronic phase. While for segmenting BRAO in the acute phase, a two-step segmentation strategy is performed: rough initialization and refine segmentation. The proposed method was tested on SD-OCT images of 35 patients (12 BRAO acute phase, 11 BRAO chronic phase, and 12 normal eyes) using the leave-one-out strategy. The classification accuracy for BRAO acute phase, BRAO chronic phase, and normal retina were 100%, 90.9%, and 91.7%, respectively. The overall true positive volume fraction (TPVF) and false positive volume fraction (FPVF) for the acute phase were 91.1% and 5.5% and for the chronic phase were 92.7% and 8.4%, respectively.

In this paper, we propose a novel method for single-channel blind separation of nonoverlapped sources and, to the best of our knowledge, apply it for the first time to automatic segmentation of lung tumors in positron emission tomography (PET) images. Our approach first converts a 3-D PET image into a pseudo-multichannel image. Afterward, regularization free sparseness constrained non-negative matrix factorization is used to separate tumor from other tissues. By using complexity based criterion, we select tumor component as the one with minimal complexity. We have compared the proposed method with threshold based on 40% and 50% maximum standardized uptake value (SUV), graph cuts (GC), random walks (RW), and affinity propagation (AP) algorithms on 18 nonsmall cell lung cancer datasets with respect to ground truth (GT) provided by two radiologists. Dice similarity coefficient averaged with respect to two GTs is: 0.78 ± 0.12 by the proposed algorithm, 0.78 ± 0.1 by GC, 0.77 ± 0.13 by AP, 0.77 ± 0.07 by RW, and 0.75 ± 0.13 by 50% maximum SUV threshold. Since the proposed method achieved performance comparable with interactive methods, considering the unique challenges of lung tumor segmentation from PET images, our findings support possibility of using our fully automated method in routine clinics. The source codes will be available at www.mipav.net/English/research/research.html.

Choroid neovascularization (CNV) is caused by new blood vessels growing in the choroid and penetrating the bruch membrane. It is the major cause of vision disability in many retinal diseases. Though anti-vascular endothelial growth factor injection has proved to be effective for treating CNV, treatment planning is essential to ensure the efficacy while reducing the risk. For this purpose, we propose a CNV growth model based on longitudinal optical coherence tomography (OCT) images. The reaction-diffusion model is applied to simulate the growth and shrinkage of CNV volumes, and is solved by using the finite-element method. A fitted curve of the CNV growth/shrinkage rate is obtained by optimizing the growth parameters. Then, the trained parameters are applied to the predicted image to get the simulated image, which is compared with the validated image to evaluate the accuracy of prediction. The proposed method was tested on a dataset with seven patients in which each patient has 12 longitudinal OCT images. The resulted mean dice coefficient is 76.40% ± 8.20%. The experimental results show a promising step towards the image-guided patient-specific treatment.

In this paper, we investigate the automatic detection of white and brown adipose tissues using Positron Emission Tomography/Computed Tomography (PET/CT) scans, and develop methods for the quantification of these tissues at the whole-body and body-region levels. We propose a patient-specific automatic adiposity analysis system with two modules. In the first module, we detect white adipose tissue (WAT) and its two sub-types from CT scans: Visceral Adipose Tissue (VAT) and Subcutaneous Adipose Tissue (SAT). This process relies conventionally on manual or semi-automated segmentation, leading to inefficient solutions. Our novel framework addresses this challenge by proposing an unsupervised learning method to separate VAT from SAT in the abdominal region for the clinical quantification of central obesity. This step is followed by a context driven label fusion algorithm through sparse 3D Conditional Random Fields (CRF) for volumetric adiposity analysis. In the second module, we automatically detect, segment, and quantify brown adipose tissue (BAT) using PET scans because unlike WAT, BAT is metabolically active. After identifying BAT regions using PET, we perform a co-segmentation procedure utilizing asymmetric complementary information from PET and CT. Finally, we present a new probabilistic distance metric for differentiating BAT from non-BAT regions. Both modules are integrated via an automatic body-region detection unit based on one-shot learning. Experimental evaluations conducted on 151 PET/CT scans achieve state-of-the-art performances in both central obesity as well as brown adiposity quantification.

In the above-named work, the spelling of the second author’s name was incorrect. The correct spelling is given.

In this paper, a fully automatic method is proposed to segment the kidney into multiple components: renal cortex, renal column, renal medulla and renal pelvis, in clinical 3D CT abdominal images. The proposed fast automatic segmentation method of kidney consists of two main parts: localization of renal cortex and segmentation of kidney components. In the localization of renal cortex phase, a method which fully combines 3D Generalized Hough Transform (GHT) and 3D Active Appearance Models (AAM) is applied to localize the renal cortex. In the segmentation of kidney components phase, a modified Random Forests (RF) method is proposed to segment the kidney into four components based on the result from localization phase. During the implementation, a multithreading technology is applied to speed up the segmentation process. The proposed method was evaluated on a clinical abdomen CT data set, including 37 contrast-enhanced volume data using leave-one-out strategy. The overall true-positive volume fraction and false-positive volume fraction were 93.15%, 0.37% for renal cortex segmentation; 83.09%, 0.97% for renal column segmentation; 81.92%, 0.55% for renal medulla segmentation; and 80.28%, 0.30% for renal pelvis segmentation, respectively. The average computational time of segmenting kidney into four components took 20 seconds.

Accurate lung tumor delineation plays an important role in radiotherapy treatment planning. Since the lung tumor has poor boundary in positron emission tomography (PET) images and low contrast in computed tomography (CT) images, segmentation of tumor in the PET and CT images is a challenging task. In this paper, we effectively integrate the two modalities by making fully use of the superior contrast of PET images and superior spatial resolution of CT images. Random walk and graph cut method is integrated to solve the segmentation problem, in which random walk is utilized as an initialization tool to provide object seeds for graph cut segmentation on the PET and CT images. The co-segmentation problem is formulated as an energy minimization problem which is solved by max-flow/min-cut method. A graph, including two sub-graphs and a special link, is constructed, in which one sub-graph is for the PET and another is for CT, and the special link encodes a context term which penalizes the difference of the tumor segmentation on the two modalities. To fully utilize the characteristics of PET and CT images, a novel energy representation is devised. For the PET, a downhill cost and a 3D derivative cost are proposed. For the CT, a shape penalty cost is integrated into the energy function which helps to constrain the tumor region during the segmentation. We validate our algorithm on a data set which consists of 18 PET-CT images. The experimental results indicate that the proposed method is superior to the graph cut method solely using the PET or CT is more accurate compared with the random walk method, random walk co-segmentation method, and non-improved graph cut method.

Liver segmentation is still a challenging task in medical image processing area due to the complexity of the liver's anatomy, low contrast with adjacent organs, and presence of pathologies. This investigation was used to develop and validate an automated method to segment livers in CT images. The proposed framework consists of three steps: 1) preprocessing; 2) initialization; and 3) segmentation. In the first step, a statistical shape model is constructed based on the principal component analysis and the input image is smoothed using curvature anisotropic diffusion filtering. In the second step, the mean shape model is moved using thresholding and Euclidean distance transformation to obtain a coarse position in a test image, and then the initial mesh is locally and iteratively deformed to the coarse boundary, which is constrained to stay close to a subspace of shapes describing the anatomical variability. Finally, in order to accurately detect the liver surface, deformable graph cut was proposed, which effectively integrates the properties and inter-relationship of the input images and initialized surface. The proposed method was evaluated on 50 CT scan images, which are publicly available in two databases Sliver07 and 3Dircadb. The experimental results showed that the proposed method was effective and accurate for detection of the liver surface.

Automated retinal layer segmentation of optical coherence tomography (OCT) images has been successful for normal eyes but becomes challenging for eyes with retinal diseases if the retinal morphology experiences critical changes. We propose a method to automatically segment the retinal layers in 3-D OCT data with serous retinal pigment epithelial detachments (PED), which is a prominent feature of many chorioretinal disease processes. The proposed framework consists of the following steps: fast denoising and B-scan alignment, multi-resolution graph search based surface detection, PED region detection and surface correction above the PED region. The proposed technique was evaluated on a dataset with OCT images from 20 subjects diagnosed with PED. The experimental results showed the following. 1) The overall mean unsigned border positioning error for layer segmentation is 7.87±3.36 μm , and is comparable to the mean inter-observer variability ( 7.81±2.56 μm). 2) The true positive volume fraction (TPVF), false positive volume fraction (FPVF) and positive predicative value (PPV) for PED volume segmentation are 87.1%, 0.37%, and 81.2%, respectively. 3) The average running time is 220 s for OCT data of 512 × 64 × 480 voxels.

In this paper, we propose a novel method based on a strategic combination of the active appearance model (AAM), live wire (LW), and graph cuts (GCs) for abdominal 3-D organ segmentation. The proposed method consists of three main parts: model building, object recognition, and delineation. In the model building part, we construct the AAM and train the LW cost function and GC parameters. In the recognition part, a novel algorithm is proposed for improving the conventional AAM matching method, which effectively combines the AAM and LW methods, resulting in the oriented AAM (OAAM). A multiobject strategy is utilized to help in object initialization. We employ a pseudo-3-D initialization strategy and segment the organs slice by slice via a multiobject OAAM method. For the object delineation part, a 3-D shape-constrained GC method is proposed. The object shape generated from the initialization step is integrated into the GC cost computation, and an iterative GC-OAAM method is used for object delineation. The proposed method was tested in segmenting the liver, kidneys, and spleen on a clinical CT data set and also on the MICCAI 2007 Grand Challenge liver data set. The results show the following: 1) The overall segmentation accuracy of true positive volume fraction TPVF > 94.3% and false positive volume fraction can be achieved; 2) the initialization performance can be improved by combining the AAM and LW; 3) the multiobject strategy greatly facilitates initialization; 4) compared with the traditional 3-D AAM method, the pseudo-3-D OAAM method achieves comparable performance while running 12 times faster; and 5) the performance of the proposed method is comparable to state-of-the-art liver segmentation algorithm. The executable version of the 3-D shape-constrained GC method with a user interface can be downloaded from http://xinjianchen.wordpress.com/research/.

Cardiac CT (CCT) is widely available and has been validated for the detection of focal myocardial scar using a delayed enhancement technique in this paper. CCT, however, has not been previously evaluated for quantification of diffuse myocardial fibrosis. In our investigation, we sought to evaluate the potential of low-dose CCT for the measurement of myocardial whole heart extracellular volume (ECV) fraction. ECV is altered under conditions of increased myocardial fibrosis. A framework consisting of three main steps was proposed for CCT whole heart ECV estimation. First, a shape-constrained graph cut (GC) method was proposed for myocardium and blood pool segmentation on postcontrast image. Second, the symmetric demons deformable registration method was applied to register precontrast to postcontrast images. So the correspondences between the voxels from precontrast to postcontrast images were established. Finally, the whole heart ECV value was computed. The proposed method was tested on 20 clinical low-dose CCT datasets with precontrast and postcontrast images. The preliminary results demonstrated the feasibility and efficiency of the proposed method.

Segmentation of anatomical structures from medical images is a challenging problem, which depends on the accurate recognition (localization) of anatomical structures prior to delineation. This study generalizes anatomy segmentation problem via attacking two major challenges: 1) automatically locating anatomical structures without doing search or optimization, and 2) automatically delineating the anatomical structures based on the located model assembly. For 1), we propose intensity weighted ball-scale object extraction concept to build a hierarchical transfer function from image space to object (shape) space such that anatomical structures in 3-D medical images can be recognized without the need to perform search or optimization. For 2), we integrate the graph-cut (GC) segmentation algorithm with prior shape model. This integrated segmentation framework is evaluated on clinical 3-D images consisting of a set of 20 abdominal CT scans. In addition, we use a set of 11 foot MR images to test the generalizability of our method to the different imaging modalities as well as robustness and accuracy of the proposed methodology. Since MR image intensities do not possess a tissue specific numeric meaning, we also explore the effects of intensity nonstandardness on anatomical object recognition. Experimental results indicate that: 1) effective recognition can make the delineation more accurate; 2) incorporating a large number of anatomical structures via a model assembly in the shape model improves the recognition and delineation accuracy dramatically; 3) ball-scale yields useful information about the relationship between the objects and the image; 4) intensity variation among scenes in an ensemble degrades object recognition performance.

An automated method is reported for segmenting 3-D fluid-associated abnormalities in the retina, so-called symptomatic exudate-associated derangements (SEAD), from 3-D OCT retinal images of subjects suffering from exudative age-related macular degeneration. In the first stage of a two-stage approach, retinal layers are segmented, candidate SEAD regions identified, and the retinal OCT image is flattened using a candidate-SEAD aware approach. In the second stage, a probability constrained combined graph search-graph cut method refines the candidate SEADs by integrating the candidate volumes into the graph cut cost function as probability constraints. The proposed method was evaluated on 15 spectral domain OCT images from 15 subjects undergoing intravitreal anti-VEGF injection treatment. Leave-one-out evaluation resulted in a true positive volume fraction (TPVF), false positive volume fraction (FPVF) and relative volume difference ratio (RVDR) of 86.5%, 1.7%, and 12.8%, respectively. The new graph cut-graph search method significantly outperformed both the traditional graph cut and traditional graph search approaches (p < 0.01, p < 0.04) and has the potential to improve clinical management of patients with choroidal neovascularization due to exudative age-related macular degeneration.

In this paper, we present an automatic renal cortex segmentation approach using the implicit shape registration and novel multiple surfaces graph search. The proposed approach is based on a hierarchy system. First, the whole kidney is roughly initialized using an implicit shape registration method, with the shapes embedded in the space of Euclidean distance functions. Second, the outer and inner surfaces of renal cortex are extracted utilizing multiple surfaces graph searching, which is extended to allow for varying sampling distances and physical constraints to better separate the renal cortex and renal column. Third, a renal cortex refining procedure is applied to detect and reduce incorrect segmentation pixels around the renal pelvis, further improving the segmentation accuracy. The method was evaluated on 17 clinical computed tomography scans using the leave-one-out strategy with five metrics: Dice similarity coefficient (DSC), volumetric overlap error (OE), signed relative volume difference (SVD), average symmetric surface distance (D(avg)), and average symmetric rms surface distance (D(rms)). The experimental results of DSC, OE, SVD, D(avg) , and D(rms) were 90.50% ± 1.19%, 4.38% ± 3.93%, 2.37% ± 1.72%, 0.14 mm ± 0.09 mm , and 0.80 mm ± 0.64 mm, respectively. The results showed the feasibility, efficiency, and robustness of the proposed method.

In the above-named article (ibid., vol. 31, no. 10, pp. 1849-1860, Oct. 2012), the author name Jian Tian should have been Jie Tian.

In this paper, we propose a novel 3D automatic anatomy segmentation method based on the synergistic combination of active appearance models (AAM), live wire (LW) and graph cut (GC). The proposed method consists of three main parts: model building, initialization and segmentation. For the model building part, an AAM model is constructed and the LW cost function is trained. For the initialization part, an improved iterative model refinement algorithm is proposed for the AAM optimization, which synergistically combines the AAM and LW method (OAAM). And a multi-object strategy is applied to help the object initialization. A pseudo 3D initialization strategy is employed to segment the organs slice by slice via multi-object OAAM method. The model constraints are applied to the initialization result. For the segmentation part, the object shape information generated from the initialization step is integrated into the GC cost computation. And an iterative GCOAAM method is proposed for object delineation. This method is a general method and can be applied to any organ segmentation. The proposed method was tested on the clinical liver and kidney CT data sets. The results showed the following: (a) an overall segmentation accuracy of true positive fraction>93.5%, and false positive fraction<0.2% can be achieved. (b) The initialization performance is improved by combining the AAM and LW. (c) The multi-object strategy greatly helps the initialization due to inter-object constraints.

This paper presents a systematic multi-path HMM topology design algorithm to better model online handwriting of East Asian characters. This data-driven algorithm solves three key problems in HMM topology design. First, HMM path number determination is formalized as a clustering problem using subsequence direction histogram vector (SDHV) as feature of both writing order and style. Second, curvature scale space-based (CSS-based) substroke segmentation is used to calculate the optimal state number and initial state parameters. Third, self-rotation restricted corner state and imaginary stroke state are designed to determine state connectivity and Gaussian mixture number in order to achieve better state alignment. Experiments on large character sets demonstrate both a significant relative error reduction rate and high recognition accuracy using the proposed algorithm.

Coping with nonlinear distortions in fingerprint matching is a challenging task. This paper proposes a novel method, a fuzzy feature match (FFM) based on a local triangle feature set to match the deformed fingerprints. The fingerprint is represented by the fuzzy feature set: the local triangle feature set. The similarity between the fuzzy feature set is used to characterize the similarity between fingerprints. A fuzzy similarity measure for two triangles is introduced and extended to construct a similarity vector including the triangle-level similarities for all triangles in two fingerprints. Accordingly, a similarity vector pair is defined to illustrate the similarities between two fingerprints. The FFM method maps the similarity vector pair to a normalized value which quantifies the overall image to image similarity. The proposed algorithm has been evaluated with NIST 24 and FVC2004 fingerprint databases. Experimental results confirm that the proposed FFM based on the local triangle feature set is a reliable and effective algorithm for fingerprint matching with nonlinear distortions.

Coping with nonlinear distortions in fingerprint matching is a challenging task. This paper proposes a novel algorithm, normalized fuzzy similarity measure (NFSM), to deal with the nonlinear distortions. The proposed algorithm has two main steps. First, the template and input fingerprints were aligned. In this process, the local topological structure matching was introduced to improve the robustness of global alignment. Second, the method NFSM was introduced to compute the similarity between the template and input fingerprints. The proposed algorithm was evaluated on fingerprints databases of FVC2004. Experimental results confirm that NFSM is a reliable and effective algorithm for fingerprint matching with nonliner distortions. The algorithm gives considerably higher matching scores compared to conventional matching algorithms for the deformed fingerprints.