In safety-critical domains like medical imaging, where diagnostic errors have severe consequences, AI models must be evaluated beyond average accuracy. A trustworthy model must demonstrate two distinct virtues: high reliability on common, easy cases and high expert capability on challenging, ambiguous, or rare cases. Conventional aggregate metrics fail to distinguish between these, masking a model's fatal flaw-such as misclassifying an easy case-by rewarding its high volume of trivial successes. We present Hardness-Aware Model Evaluation (HaME), a framework that assesses models using an asymmet-ric cost-benefit analysis. HaME identifies challenging instances via foundation models, then evaluates the target model using novel metrics (HaPrecision, HaRecall, HaFt, HaAUC) and the Brittleness Gap (B-Gap). Our formulation uniquely penalizes “easy” errors far more than “hard” failures, while simultaneously rewarding “hard” successes. This shifts the evaluation from “av-erage performance” to “clinical trustworthiness.” Experiments on medical image classification (Dermatology, Pneumonia, Retinal OCT) and segmentation (Nuclei) reveal that HaME uncovers critical reliability gaps and expert-level specializations invisible to standard metrics.

Current successful choroidal vessel segmentation methods rely on large amounts of voxel-level annotations on the 3D optical coherence tomography images, which are hard and time-consuming. Semi-supervised learning solves this issue by enabling model learning from both unlabeled data and a limited amount of labeled data. A challenge is the defective pseudo labels generated for the unlabeled data. In this work, we propose a model-based label-to-image diffusion (MLD) framework for semi-supervised choroidal vessel segmentation. We first generate pseudo labels from unlabeled images with a coarse correspondence using a model-based strategy. Then, we generate precisely corresponding images of pseudo labels by a hierarchical diffusion probabilistic model. We evaluated our method on myopia data with a new topological connectivity metric. The quantitative and qualitative experimental results indicate the effectiveness of the label-to-image diffusion framework and its benefit for enhancing the existing supervised choroidal segmentation methods. The code is available at: https://github.com/nicetomeetu21/MLD.

Accurate retinal layer segmentation on optical coherence tomography (OCT) images is hampered by the challenges of collecting OCT images with diverse pathological characterization and balanced distribution. Current generative models can produce high-realistic images and corresponding labels without quantitative limitations by fitting distributions of real collected data. Nevertheless, the diversity of their generated data is still limited due to the inherent imbalance of training data. To address these issues, we propose an image-label pair generation framework that generates diverse and balanced potential data from imbalanced real samples. Specifically, the framework first generates diverse layer masks, and then generates plausible OCT images corresponding to these layer masks using two customized diffusion probabilistic models respectively. To learn from imbalanced data and facilitate balanced generation, we introduce pathological-related conditions to guide the generation processes. To enhance the diversity of the generated image-label pairs, we propose a potential structure modeling technique that transfers the knowledge of diverse sub-structures from lowly- or non-pathological samples to highly pathological samples. We conducted extensive experiments on two public datasets for retinal layer segmentation. Firstly, our method generates OCT images with higher image quality and diversity compared to other generative methods. Furthermore, based on the extensive training with the generated OCT images, downstream retinal layer segmentation tasks demonstrate improved results. The code is publicly available at: https://github.com/nicetomeetu21/GenPSM.

Large volume of labeled data is a cornerstone for deep learning (DL) based segmentation methods. Medical images require domain experts to annotate, and full segmentation annotations of large volumes of medical data are difficult, if not impossible, to acquire in practice. Compared with full annotations, image-level labels are multiple orders of magnitude faster and easier to obtain. Image-level labels contain rich information that correlates with the underlying segmentation tasks and should be utilized in modeling segmentation problems. In this article, we aim to build a robust DL-based lesion segmentation model using only image-level labels (normal v.s. abnormal). Our method consists of three main steps: (1) training an image classifier with image-level labels; (2) utilizing a model visualization tool to generate an object heat map for each training sample according to the trained classifier; (3) based on the generated heat maps (as pseudo-annotations) and an adversarial learning framework, we construct and train an image generator for Edema Area Segmentation (EAS). We name the proposed method Lesion-Aware Generative Adversarial Networks (LAGAN) as it combines the merits of supervised learning (being lesion-aware) and adversarial training (for image generation). Additional technical treatments, such as the design of a multi-scale patch-based discriminator, further enhance the effectiveness of our proposed method. We validate the superior performance of LAGAN via comprehensive experiments on two publicly available datasets (i.e., AI Challenger and RETOUCH).

Automated layer segmentation plays an important role for retinal disease diagnosis in optical coherence tomography (OCT) images. However, the severe retinal diseases result in the performance degeneration of automated layer segmentation approaches. In this paper, we present a robust semi-supervised layer segmentation network to relieve the model failures on abnormal retinas. We obtain the lesion features from the labeled images with disease-balanced distribution, and utilize the unlabeled images to supplement the layer structure information. Specifically, in our method, the cross-consistency training is utilized over the predictions of different decoders, and we enforce a consistency between different decoder predictions to improve the encoder's representation. Then, we propose a sequence prediction branch based on self-supervised manner, which is designed to predict the position of each jigsaw puzzle to obtain sensory perception of the retinal layer structure. To this task, a layer spatial pyramid pooling (LSPP) module is designed to extract multi-scale layer spatial features. Furthermore, we use the optical coherence tomography angiography (OCTA) to supplement the information damaged by diseases. The experimental results illustrate that our method achieves more robust results compared with current supervised segmentation methods. Meanwhile, advanced segmentation performance can be obtained compared with state-of-the-art semi-supervised segmentation methods.

Retinal related diseases are the leading cause of vision loss, and severe retinal lesion causes irreversible damage to vision. Therefore, the automatic methods for retinal diseases detection based on medical images is essential for timely treatment. Considering that manual diagnosis and analysis of medical images require a large number of qualified experts, deep learning can effectively diagnosis and locate critical biomarkers. In this paper, we present a novel model by jointly optimize the cycle generative adversarial network (CycleGAN) and the convolutional neural network (CNN) to detect retinal diseases and localize lesion areas with limited training data. The CycleGAN with cycle consistency can generate more realistic and reliable images. The discriminator and the generator achieve a local optimal solution in an adversarial manner, and the generator and the classifier are in a cooperative manner to distinguish the domain of input images. A novel res-guided sampling block is proposed by combining learnable residual features and pixel-adaptive convolutions. A res-guided U-Net is constructed as the generator by substituting the traditional convolution with the res-guided sampling blocks. Our model achieve superior classification and localization performance on LAG, Ichallenge-PM and Ichallenge-AMD datasets. With clear localization for lesion areas, the competitive results reveal great potentials of the joint optimization network. The source code is available at https://github.com/jizexuan/JointOptmization.

Choroidal neovascularization (CNV) volume prediction has an important clinical significance to predict the therapeutic effect and schedule the follow-up. In this paper, we propose a Lesion Attention Maps-Guided Network (LamNet) to automatically predict the CNV volume of next follow-up visit after therapy based on 3-dimentional spectral-domain optical coherence tomography (SD-OCT) images. In particular, the backbone of LamNet is a 3D convolutional neural network (3D-CNN). In order to guide the network to focus on the local CNV lesion regions, we use CNV attention maps generated by an attention map generator to produce the multi-scale local context features. Then, the multi-scale of both local and global feature maps are fused to achieve the high-precision CNV volume prediction. In addition, we also design a synergistic multi-task predictor, in which a trend-consistent loss ensures that the change trend of the predicted CNV volume is consistent with the real change trend of the CNV volume. The experiments include a total of 541 SD-OCT cubes from 68 patients with two types of CNV captured by two different SD-OCT devices. The results demonstrate that LamNet can provide the reliable and accurate CNV volume prediction, which would further assist the clinical diagnosis and design the treatment options.

Spectral-domain optical coherence tomography (SD-OCT) images inevitably suffer from multiplicative speckle noise caused by random interference. This study proposes an unsupervised domain adaptation approach for noise reduction by translating the SD-OCT to the corresponding high-quality enhanced depth imaging (EDI)-OCT. We propose a structure-persevered cycle-consistent generative adversarial network for unpaired image-to-image translation, which can be applied to imbalanced unpaired data, and can effectively preserve retinal details based on a structure-specific cross-domain description. It also imposes smoothness by penalizing the intensity variation of the low reflective region between consecutive slices. Our approach was tested on a local data set that consisted of 268 SD-OCT volumes and two public independent validation datasets including 20 SD-OCT volumes and 17 B-scans, respectively. Experimental results show that our method can effectively suppress noise and maintain the retinal structure, compared with other traditional approaches and deep learning methods in terms of qualitative and quantitative assessments. Our proposed method shows good performance for speckle noise reduction and can assist downstream tasks of OCT analysis.

As one of the most critical characteristics in advanced stage of non-exudative Age-related Macular Degeneration (AMD), Geographic Atrophy (GA) is one of the significant causes of sustained visual acuity loss. Automatic localization of retinal regions affected by GA is a fundamental step for clinical diagnosis. In this paper, we present a novel weakly supervised model for GA segmentation in Spectral-Domain Optical Coherence Tomography (SD-OCT) images. A novel Multi-Scale Class Activation Map (MS-CAM) is proposed to highlight the discriminatory significance regions in localization and detail descriptions. To extract available multi-scale features, we design a Scaling and UpSampling (SUS) module to balance the information content between features of different scales. To capture more discriminative features, an Attentional Fully Connected (AFC) module is proposed by introducing the attention mechanism into the fully connected operations to enhance the significant informative features and suppress less useful ones. Based on the location cues, the final GA region prediction is obtained by the projection segmentation of MS-CAM. The experimental results on two independent datasets demonstrate that the proposed weakly supervised model outperforms the conventional GA segmentation methods and can produce similar or superior accuracy comparing with fully supervised approaches. The source code has been released and is available on GitHub: https://github.com/ jizexuan/Multi-Scale-Class-Activation-Map-Tensorflow.

The presence of hyperreflective foci (HFs) is related to retinal disease progression, and the quantity has proven to be a prognostic factor of visual and anatomical outcome in various retinal diseases. However, lack of efficient quantitative tools for evaluating the HFs has deprived ophthalmologist of assessing the volume of HFs. For this reason, we propose an automated quantification algorithm to segment and quantify HFs in spectral domain optical coherence tomography (SD-OCT). The proposed algorithm consists of two parallel processes namely: region of interest (ROI) generation and HFs estimation. To generate the ROI, we use morphological reconstruction to obtain the reconstructed image and histogram constructed for data distributions and clustering. In parallel, we estimate the HFs by extracting the extremal regions from the connected regions obtained from a component tree. Finally, both the ROI and the HFs estimation process are merged to obtain the segmented HFs. The proposed algorithm was tested on 40 3D SD-OCT volumes from 40 patients diagnosed with non-proliferative diabetic retinopathy (NPDR), proliferative diabetic retinopathy (PDR), and diabetic macular edema (DME). The average dice similarity coefficient (DSC) and correlation coefficient (r) are 69.70%, 0.99 for NPDR, 70.31%, 0.99 for PDR, and 71.30%, 0.99 for DME, respectively. The proposed algorithm can provide ophthalmologist with good HFs quantitative information, such as volume, size, and location of the HFs.

We present an image projection network (IPN), which is a novel end-to-end architecture and can achieve 3D-to-2D image segmentation in optical coherence tomography angiography (OCTA) images. Our key insight is to build a projection learning module (PLM) which uses a unidirectional pooling layer to conduct effective features selection and dimension reduction concurrently. By combining multiple PLMs, the proposed network can input 3D OCTA data, and output 2D segmentation results such as retinal vessel segmentation. It provides a new idea for the quantification of retinal indicators: without retinal layer segmentation and without projection maps. We tested the performance of our network for two crucial retinal image segmentation issues: retinal vessel (RV) segmentation and foveal avascular zone (FAZ) segmentation. The experimental results on 316 OCTA volumes demonstrate that the IPN is an effective implementation of 3D-to-2D segmentation networks, and the uses of multi-modality information and volumetric information make IPN perform better than the baseline methods.

Retinal swelling due to the accumulation of fluid is associated with the most vision-threatening retinal diseases. Optical coherence tomography (OCT) is the current standard of care in assessing the presence and quantity of retinal fluid and image-guided treatment management. Deep learning methods have made their impact across medical imaging, and many retinal OCT analysis methods have been proposed. However, it is currently not clear how successful they are in interpreting the retinal fluid on OCT, which is due to the lack of standardized benchmarks. To address this, we organized a challenge RETOUCH in conjunction with MICCAI 2017, with eight teams participating. The challenge consisted of two tasks: fluid detection and fluid segmentation. It featured for the first time: all three retinal fluid types, with annotated images provided by two clinical centers, which were acquired with the three most common OCT device vendors from patients with two different retinal diseases. The analysis revealed that in the detection task, the performance on the automated fluid detection was within the inter-grader variability. However, in the segmentation task, fusing the automated methods produced segmentations that were superior to all individual methods, indicating the need for further improvements in the segmentation performance.

This paper presents an interactive image segmentation approach where the segmentation problem is formulated as a probabilistic estimation manner. Instead of measuring the distances between unseeded pixels and seeded pixels, we measure the similarities between pixel pairs and seed pairs to improve the robustness to the seeds. The unary prior probability of each pixel belonging to the foreground F and background B can be effectively estimated based on the similarities with label pairs (F, F),(F, B),(B, F) and (B, B). Then a likelihood learning framework is proposed to fuse the region and boundary information of the image by imposing the smoothing constraint on the unary potentials. Experiments on challenging data sets demonstrate that the proposed method can obtain better performance than state-of-the-art methods.

The altitude information of single remote sensing image may aid in detecting natural disasters such as landslides or mud-rock flow. Accordingly, in this letter, an approach based on dark channel prior is proposed for the altitude extraction of single remote sensing image, and it also overcomes the effect of mountain shadow. We first detect the mountain shadows based on machine learning, and then adjust the brightness of each shadow with an adaptive adjustment parameter. Next, we estimate the altitude information based on dark channel prior, including atmospheric light calculation and soft matting. The experimental results with real mountain remote sensing images demonstrate that the proposed algorithm can obtain the accurate relative altitude information, which is effective for the extraction of the relative altitude of single mountain remote sensing image with shadows.

The combination of pixel and superpixel has been widely utilized in the interactive segmentation methods to overcome the sensitivity to the seeds' quantity and quality. However, because of the introduction of more variables and variables' interactions, the pixel-superpixel combination methods are still limited to the segmentation accuracy and computational complexity. To solve these problems, in this paper, we propose an interactive multilabel image segmentation method. In the proposed segmentation model, the multilayer relationships among the pixel layer, superpixel layer, and label layer are fused by the Markov random field framework to further improve the segmentation accuracy. During the optimization stage, the parallel partial optimality strategy is utilized to effectively solve the multilabel submodular energy function. Experimental results on challenging data sets demonstrate the competitiveness of the proposed method comparing with several state-of-the-art interactive algorithms.

There are lots of objective image quality assessment (IQA) algorithms to accurately assess the quality of images recently; however, the characteristics of Human Visual System (HVS) as the ultimate receiver of the visual signal are critical factors affecting IQA. Saliency as a feature reflecting quality has been studied deeply. Since HVS has intricate psychovisual structure and nature, the impact of saliency on IQA algorithms needs still further exploration. This paper focuses mainly on the influence of distortion information on saliency applied on IQA. We eliminated interference of algorithms to study the characteristics of different distortion types and degrees simply. We applied three different objective metrics adding natural scene saliency (NSS) with different adding strategies on the LIVE IQA database. Experimental results demonstrate that the variation in saliency highly depends on the distortion type and degree. IQA algorithms will achieve improving performances on their accuracy by applied proper saliency strategy.

Changes of illumination have a huge effect on image quality during imaging process. One method that compares high and low resolution images to compute MTF in image quality assessment requires images to be in the same illumination conditions. Thus, it is necessary to do illumination normalization. This letter presents a novel method by combining gradient domain method and improved singular value equalization, which can achieve a good result of illumination normalization. The gradient domain method can bring the contrasts of multiple images to the same level while the improved singular value equalization can make their intensity means close to each other. We also suggest a parameter named p to assess the illumination consistency quantitatively. Experimental results demonstrate that the proposed method has a good performance in visualization and quantitative assessment.

Accurate brain tissue segmentation from magnetic resonance (MR) images is an essential step in quantitative brain image analysis. However, due to the existence of noise and intensity inhomogeneity in brain MR images, many segmentation algorithms suffer from limited accuracy. In this paper, we assume that the local image data within each voxel's neighborhood satisfy the Gaussian mixture model (GMM), and thus propose the fuzzy local GMM (FLGMM) algorithm for automated brain MR image segmentation. This algorithm estimates the segmentation result that maximizes the posterior probability by minimizing an objective energy function, in which a truncated Gaussian kernel function is used to impose the spatial constraint and fuzzy memberships are employed to balance the contribution of each GMM. We compared our algorithm to state-of-the-art segmentation approaches in both synthetic and clinical data. Our results show that the proposed algorithm can largely overcome the difficulties raised by noise, low contrast, and bias field, and substantially improve the accuracy of brain MR image segmentation.

This paper presents a new interactive image segmentation method. The key idea is to generate the object contour feature image according to a small number of user supplied object contour points, and then adopt the parametric active contour model for object segmentation. An image patch matching with rotation-invariance is utilized to generate object contour feature image. In order to prevent the evolving curve running into local optimal solution, the initial evolving curve of the parametric active contour is constructed by local intensity values of the object contour feature image. Medical image segmentation results indicate that the proposed method is superior to the traditional parametric active contour model, and is an effective semi-automatic image segmentation method.

This letter presents a two-stage object tracking method by combining a region-based method and a contour-based method. First, a kernel-based method is adopted to locate the object region. Then the diffusion snake is used to evolve the object contour in order to improve the tracking precision. In the first object localization stage, the initial target position is predicted and evaluated by the Kalman filter and the Bhattacharyya coefficient, respectively. In the contour evolution stage, the active contour is evolved on the basis of an object feature image generated with the color information in the initial object region. In the process of the evolution, similarities of the target region are compared to ensure that the object contour evolves in the right way. The comparison between our method and the kernel-based method demonstrates that our method can effectively cope with the severe deformation of object contour, so the tracking precision of our method is higher.

Segmentation of left ventricles is one of the important research topics in cardiac magnetic resonance (MR) imaging. The segmentation precision influences the authenticity of ventricular motion reconstruction. In left ventricle MR images, the weak and broken boundary increases the difficulty of segmenting the outer contour precisely. In this paper, we present an improved shape statistics variational approach for the outer contour segmentation of left ventricle MR images. We use the Mumford-Shah model in an object feature space and incorporate the shape statistics and an edge image to the variational framework. The introduction of shape statistics can improve the segmentation with broken boundaries. The edge image can enhance the weak boundary and thus improve the segmentation precision. The generation of the object feature image, which has homogenous "intensities" in the left ventricle, facilitates the application of the Mumford-Shah model. A comparison of mean absolute distance analysis between different contours generated with our algorithm and that generated by hand demonstrated that our method can achieve a higher segmentation precision and a better stability than various approaches. It is a semiautomatic way for the segmentation of the outer contour of the left ventricle in clinical applications.

In this paper, a two-stage approach consisting of coarse and fine segmentation is adopted. For the characters, whose vertical projections combine together, but they don't connected themselves, we confine them in a window and segment them with a curve line which is acquired by connecting all the middle points in sequence. For the characters connected only by one stroke, firstly we find the candidate segmentation points on the candidate stroke in the image of the thinned character, and then determine the optimal point evaluated by the principles which are presented in this paper. The method mentioned above has been applied to the segmentation of Chinese bank check amounts and get good results.