Image segmentation foundation models (SFMs) like Segment Anything Model (SAM) have achieved impressive zero-shot and interactive segmentation across diverse domains. However, they struggle to segment objects with certain structures, particularly those with dense, tree-like morphology and low textural contrast from their surroundings. These failure modes are crucial for understanding the limitations of SFMs in real-world applications. To systematically study this issue, we introduce interpretable metrics quantifying object tree-likeness and textural separability. On carefully controlled synthetic experiments and real-world datasets, we show that SFM performance (e.g., SAM, SAM 2, HQ-SAM) noticeably correlates with these factors. We attribute these failures to SFMs misinterpreting local structure as global texture, resulting in over-segmentation or difficulty distinguishing objects from similar backgrounds. Notably, targeted fine-tuning fails to resolve this issue, indicating a fundamental limitation. Our study provides the first quantitative framework for modeling the behavior of SFMs on challenging structures, offering interpretable insights into their segmentation capabilities.1

Accurately registering breast MR images from different time points enables the alignment of anatomical structures and tracking of tumor progression, supporting more effective breast cancer detection, diagnosis, and treatment planning. However, the complexity of dense tissue and its highly non-rigid nature pose challenges for conventional registration methods, which primarily focus on aligning general structures while overlooking intricate internal details. To address this, we propose GuidedMorph, a novel two-stage registration framework designed to better align dense tissue. In addition to a single-scale network for global structure alignment, we introduce a framework that utilizes dense tissue information to track breast movement. The learned transformation fields are fused by introducing the Dual Spatial Transformer Network (DSTN), improving overall alignment accuracy. A novel warping method based on the Euclidean distance transform (EDT) is also proposed to accurately warp the registered dense tissue and breast masks, preserving fine structural details during deformation. It also operates effectively with the VoxelMorph and TransMorph backbones, offering a versatile solution for breast registration. We validate our method on ISPY2 and internal dataset, demonstrating superior performance in dense tissue, overall breast alignment, and breast structural similarity index measure (SSIM), with notable improvements by over 20.9% in dense tissue Dice, 2.1% in breast Dice, and 3.5% in breast SSIM compared to the best baseline.

Manual annotation of volumetric medical images, such as magnetic resonance imaging (MRI) and computed tomography (CT), is a labor-intensive and time-consuming process. Recent advancements in foundation models for video object segmentation, such as Segment Anything Model 2 (SAM 2), offer a potential opportunity to significantly speed up the annotation process by manually annotating one or a few slices and then propagating target masks across the entire volume. However, the performance of SAM 2 in this context varies. Our experiments show that relying on a single memory bank and attention module is prone to error propagation, particularly at boundary regions where the target is present in the previous slice but absent in the current one. To address this problem, we propose Short-Long Memory SAM 2 (SLM-SAM 2), a novel architecture that integrates distinct short-term and long-term memory banks with separate attention modules to improve segmentation accuracy. We evaluate SLM-SAM 2 on four public datasets covering organs, bones, and muscles across MRI, CT, and ultrasound videos. We show that the proposed method markedly outperforms the default SAM 2, achieving an average Dice Similarity Coefficient improvement of 0.14 and 0.10 in the scenarios when 5 volumes and 1 volume are available for the initial adaptation, respectively. SLM-SAM 2 also exhibits stronger resistance to over-propagation, reducing the time required to correct propagated masks by 60.575% per volume compared to SAM 2, making a notable step toward more accurate automated annotation of medical images for segmentation model development.

Magnetic Resonance Imaging (MRI) is pivotal in radiology, offering non-invasive and high-quality insights into the human body. Precise segmentation of the MRIs into different organs and tissues would be very beneficial as it would allow more accurate measurements, which are essential for accurate diagnosis and effective treatment planning. Specifically, segmenting bones in MRI would allow for more quantitative assessments of musculoskeletal conditions, while such assessments are largely absent in current radiological practice. The difficulty of bone MRI segmentation is illustrated by the fact that limited algorithms are publicly available, and those contained in the literature typically address a specific anatomic area. In our study, we propose a versatile, publicly available deep learning model for bone segmentation in MRI at multiple standard MRI locations. The proposed model can operate in two modes: fully automated segmentation and prompt-based segmentation. Our contributions include (1) collecting and annotating a new MRI dataset across various MRI protocols, encompassing 320 annotated volumes and more than 10k annotated slices across diverse anatomic regions; (2) investigating several standard network architectures and strategies for automated segmentation; (3) introducing SegmentAnyBone, an innovative foundation model-based approach that extends the Segment Anything Model (SAM); (4) comparative analysis of our algorithm and previous approaches; and (5) generalization analysis of our algorithm across different anatomical locations and MRI sequences, as well as three external datasets. We publicly release our model at Github Code.

This paper investigates discrepancies in how neural networks learn from different imaging domains, which are commonly overlooked when adopting computer vision techniques from the domain of natural images to other specialized domains such as medical images. Recent works have found that the generalization error of a trained network typically increases with the intrinsic dimension ($d_{data}$) of its training set. Yet, the steepness of this relationship varies significantly between medical (radiological) and natural imaging domains, with no existing theoretical explanation. We address this gap in knowledge by establishing and empirically validating a generalization scaling law with respect to $d_{data}$, and propose that the substantial scaling discrepancy between the two considered domains may be at least partially attributed to the higher intrinsic ``label sharpness'' ($K_\mathcal{F}$) of medical imaging datasets, a metric which we propose. Next, we demonstrate an additional benefit of measuring the label sharpness of a training set: it is negatively correlated with the trained model's adversarial robustness, which notably leads to models for medical images having a substantially higher vulnerability to adversarial attack. Finally, we extend our $d_{data}$ formalism to the related metric of learned representation intrinsic dimension ($d_{repr}$), derive a generalization scaling law with respect to $d_{repr}$, and show that $d_{data}$ serves as an upper bound for $d_{repr}$. Our theoretical results are supported by thorough experiments with six models and eleven natural and medical imaging datasets over a range of training set sizes. Our findings offer insights into the influence of intrinsic dataset properties on generalization, representation learning, and robustness in deep neural networks. *Code link: https://github.com/mazurowski-lab/intrinsic-properties*

How can we test AI performance? This question seems trivial, but it isn't. Standard benchmarks often have problems such as in-distribution and small-size test sets, oversimplified metrics, unfair comparisons, and short-term outcome pressure. As a consequence, good performance on standard benchmarks does not guarantee success in real-world scenarios. To address these problems, we present Touchstone, a large-scale collaborative segmentation benchmark of 9 types of abdominal organs. This benchmark is based on 5,195 training CT scans from 76 hospitals around the world and 5,903 testing CT scans from 11 additional hospitals. This diverse test set enhances the statistical significance of benchmark results and rigorously evaluates AI algorithms across various out-of-distribution scenarios. We invited 14 inventors of 19 AI algorithms to train their algorithms, while our team, as a third party, independently evaluated these algorithms on three test sets. In addition, we also evaluated pre-existing AI frameworks---which, differing from algorithms, are more flexible and can support different algorithms—including MONAI from NVIDIA, nnU-Net from DKFZ, and numerous other open-source frameworks. We are committed to expanding this benchmark to encourage more innovation of AI algorithms for the medical domain.

Medical image analysis (MedIA) has become an essential tool in medicine and healthcare, aiding in disease diagnosis, prognosis, and treatment planning, and recent successes in deep learning (DL) have made significant contributions to its advances. However, deploying DL models for MedIA in real-world situations remains challenging due to their failure to generalize across the distributional gap between training and testing samples—a problem known as domain shift. Researchers have dedicated their efforts to developing various DL methods to adapt and perform robustly on unknown and out-of-distribution (OOD) data distributions. This article comprehensively reviews domain generalization (DG) studies specifically tailored for MedIA. We provide a holistic view of how DG techniques interact within the broader MedIA system, going beyond methodologies to consider the operational implications on the entire MedIA workflow. Specifically, we categorize DG methods into data-level, feature-level, model-level, and analysis-level methods. We show how those methods can be used in various stages of the MedIA workflow with DL equipped from data acquisition to model prediction and analysis. Furthermore, we critically analyze the strengths and weaknesses of various methods, unveiling future research opportunities.

Anomaly detection (AD) aims to determine if an instance has properties different from those seen in normal cases. The success of this technique depends on how well a neural network learns from normal instances. We observe that the learning difficulty scales exponentially with the input resolution, making it infeasible to apply AD to high-resolution images. Resizing them to a lower resolution is a compromising solution and does not align with clinical practice where the diagnosis could depend on image details. In this work, we propose to train the network and perform inference at the patch level, through the sliding window algorithm. This simple operation allows the network to receive high-resolution images but introduces additional training difficulties, including inconsistent image structure and higher variance. We address these concerns by setting the network's objective to learn augmentation-invariant features. We further study the augmentation function in the context of medical imaging. In particular, we observe that the resizing operation, a key augmentation in general computer vision literature, is detrimental to detection accuracy, and the inverting operation can be beneficial. We also propose a new module that encourages the network to learn from adjacent patches to boost detection performance. Extensive experiments are conducted on breast tomosynthesis and chest X-ray datasets and our method improves 8.03% and 5.66% AUC on image-level classification respectively over the current leading techniques. The experimental results demonstrate the effectiveness of our approach.

Recognizing and organizing different series in an MRI examination is important both for clinical review and research, but it is poorly addressed by the current generation of picture archiving and communication systems (PACSs) and post-processing workstations. In this paper, we study the problem of using deep convolutional neural networks for automatic classification of abdominal MRI series to one of many series types. Our contributions are three-fold. First, we created a large abdominal MRI dataset containing 3717 MRI series including 188,665 individual images, derived from liver examinations. 30 different series types are represented in this dataset. The dataset was annotated by consensus readings from two radiologists. Both the MRIs and the annotations were made publicly available. Second, we proposed a 3D pyramid pooling network, which can elegantly handle abdominal MRI series with varied sizes of each dimension, and achieved state-of-the-art classification performance. Third, we performed the first ever comparison between the algorithm and the radiologists on an additional dataset and had several meaningful findings.

Breast tumor segmentation based on dynamic contrast-enhanced magnetic resonance imaging (DCE-MRI) is a challenging problem and an active area of research. Particular challenges, similarly as in other segmentation problems, include the class-imbalance problem as well as confounding background in DCE-MR images. To address these issues, we propose a mask-guided hierarchical learning (MHL) framework for breast tumor segmentation via fully convolutional networks (FCN). Specifically, we first develop an FCN model to generate a 3D breast mask as the region of interest (ROI) for each image, to remove confounding information from input DCE-MR images. We then design a two-stage FCN model to perform coarse-to-fine segmentation for breast tumors. Particularly, we propose a Dice-Sensitivity-like loss function and a reinforcement sampling strategy to handle the class-imbalance problem. To precisely identify locations of tumors that underwent a biopsy, we further propose an FCN model to detect two landmarks located at two nipples. We finally selected the biopsied tumor based on both identified landmarks and segmentations. We validate our MHL method on 272 patients, achieving a mean Dice similarity coefficient (DSC) of 0.72 which is comparable to mutual DSC between expert radiologists. Using the segmented biopsied tumors, we also demonstrate that the automatically generated masks can be applied to radiogenomics and can identify luminal A subtype from other molecular subtypes with the similar accuracy with the analysis based on semi-manual tumor segmentation.

In this paper, the effect of class imbalance in the case base of a case-based classifier is investigated as it pertains to case base reduction and the resulting classifier performance. A k-nearest neighbor algorithm is used as a classifier and the random mutation hill climbing (RMHC) algorithm is used for case base reduction. The effects at various levels of positive class prevalence are tested in a binary classification problem. The results indicate that class imbalance is detrimental to both case base reduction and classifier performance. Selection with RMHC generally improves the classification performance regardless of the case base prevalence.

In this study, we investigated the relation between two popular classifier performance measures: area under the receiver operator characteristic curve and overall accuracy. We also evaluated the impact of class imbalance and number of examples in test set on this relation. We perform a set of experiments in which we train multiple neural networks and test them in various, well controlled conditions. The experimental results show that given a large and balanced test set, increase in one performance measure is a very good indicator of increase in the other measure. Furthermore increasing the total number of examples, while keeping the positive class prevalence constant generally increases the correlation between the two measures. Our results also indicate that increasing the extent of class imbalance in the test set has a detrimental effect on this correlation.

This paper discusses the framework for virtual collaborative environment for researchers, practitioners, users and learners in the areas of computational intelligence and machine learning (CIML) that is currently developed by our group. It also outlines main features of the community portal under construction that will support communication and sharing of computational resources. In particular, selected aspects of structure of the portal such as common formats of data, models, software, publications and software documentation are discussed.

A knowledge-based computer assisted decision (KB-CAD) system is a case-based reasoning system previously proposed for breast cancer detection. Although it was demonstrated to be very effective for the diagnostic problem, it was also shown to be computationally expensive due to the use of mutual information between images as a similarity measure. Here, the authors propose to alleviate this drawback by reducing the case-base size. The problem is formalized and a genetic algorithm is utilized as an optimization tool. Appropriate for the problem representation and operators are presented and discussed. A clinically relevant index of the area under the receiver operator characteristic curve is used as a measure of the system performance during the optimization and testing stages. Experimental results show that application of the proposed method can significantly reduce the case-base size while the classification performance of the KB-CAD, in fact, increases.

In decentralized control of multi-agent systems each agent is making a decision regarding its action autonomously, based on its own observations. In the light of the formal models of decentralized environments presented in the last decade, finding an optimal solution to a decentralized control problem is computationally prohibitive, even for moderately complicated environments. The problem, however, is of great significance since many of the real world systems can be treated as multi-agent systems with decentralized control. In this article, the authors propose an approximate algorithm for the problem based on a genetic algorithm. First, the problem is formalized using decentralized partially observable Markov decision processes. Then a way of representing a solution (joint policy) in a chromosome is introduced and a genetic algorithm is proposed as a search mechanism. Finally, a multi-agent tiger problem is used as an experimental framework to show the effectiveness of the algorithm.

This paper presents an experimental study on the impact of low class prevalence on the neural network based classifier performance as measured using receiver operator characteristic (ROC) analysis. Two methods of dealing with the problem are investigated: oversampling and undersampling in the context of varying the class prevalence and the size of training datasets with uncorrelated and correlated features. The results show that the class imbalance can significantly decrease the classifier performance especially in the case of small training datasets. Furthermore, the oversampling method is shown to be more effective than the undersampling method in compensating the class imbalance. Statistically significant differences, however, are observed only in the cases with large total number of samples and very low prevalence.

Computer-assisted decision (CAD) systems are becoming increasingly popular for the diagnostic interpretation of radiologic images. These CAD systems often involve the stacked generalization of several different decision models. Combining decision models is a common meta-analysis strategy to improve upon the diagnostic performance of each individual model. This study investigates how different data handling schemes may affect the performance evaluation of CAD systems that rely on stacked generalization. The study is based on a multistage CAD system for the detection of masses in screening mammograms. The CAD system consists of a series of knowledge-based modules that operate at Level 0 capturing morphological as well as multiscale textural information. Then, the knowledge-based predictions are combined with a Level 1 classifier. The study shows that a leave-one-out sampling scheme appears to be an effective and relatively unbiased strategy for the estimation of the overall performance of a CAD system that is based on stacked generalization. However, extra caution should be placed on the complexity of the Level 1 combiner. When the available dataset is relatively small, a relatively simple learning system such as a backpropagation neural network with very few hidden nodes is preferable to avoid optimistically biased estimates of diagnostic performance.

This paper outlines an approximate algorithm for finding an optimal decentralized control in multi-agent systems. Decentralized partially observable Markov decision processes and their extension to infinite state, observation and action spaces are utilized as a theoretical framework. In the presented algorithm, policies of each agent are represented by a feedforward neural network. Then, a search is performed in a joint weight space of all networks. Particle swarm optimization is applied as a search algorithm. Experimental results are provided showing that the algorithm finds good solutions for the classical Tiger problem extended to multi-agent systems, as well as for a multi-agent navigation task involving large state and action spaces