Recent technologies to integrate information of molecular data and images such as spatial transcriptomics (ST) have revolutionized cancer research, enabling deeper insights into tumor heterogeneity and personalized treatment strategies. Existing ST computational models primarily simulate gene expression patterns, but fail to produce realistic, molecularly aligned tumor images at the spot (or tile) level. This study introduces STsim, an innovative computational framework for simulating spot images from gene expression in spatial transcriptomics. STsim consists of a Gene Embedder to improve gene-image correlation in gene embedding, an Image Encoder-Decoder utilizing vector quantized generative adversarial network (VQGAN) for effective image compression and reconstruction, and an innovative diffusion model based on transformers for spot image generation. We evaluate STsim on public breast cancer ST datasets including 298,854 spots from 316 samples. The results demonstrate the ability to generate high-quality simulated images with Fréchet Inception Distance (FID) score of 10.55, significantly better than a baseline model (FID = 123.14) that does not incorporate gene expression in model training. Furthermore, the simulated images by STsim can preserve both morphological characteristics and biological features of the spots, offering a powerful tool for ST research.

Multiple instance learning (MIL) has became a dominant strategy for histopathological whole slide image (WSI) analysis, enabling slide-level prediction from patch-based inputs under weak supervision. While recent transformer- and graph-based MIL methods have improved performance by modeling inter-patch relationships, they face two critical limitations: computational inefficiency when processing ultra-long sequences inherent to gigapixel WSIs, and insufficient interpretability due to the absence of robust quantitative evaluation frameworks. To address these challenges, we propose WSiG, a novel graph neural network framework integrating: (i) a gate-attention distance module that optimizes dynamic graph construction complexity and (ii) a query-aware aggregation mechanism that strengthens feature aggregation and distinction. Extensive experiments on two public datasets (CAMELYON and PANDA) demonstrate WSiG’s superiority, achieving state-of-the-art AUCs of 97.9% and 99.65%, respectively, with minimal computational complexity. In addition, quantitative interpretability analysis demonstrated WSiG’s precise localization of diagnostically relevant regions, further supporting its reliability in practical applications. The code will be publicly available at https://github.com/GaryinDeep/WSiG.

Diffusion probabilistic models (DPMs) have attracted much attention in the field of computer vision by demonstrating superior image generation capabilities. However, in the field of medical images, the scarcity of annotations often poses a challenge for the application of deep learning-like methods. Motivated by the success of DPMs, we present Medical Image Segmentation Augmentation Diffusion Model (MEDSAD), a DPM and mask-based medical image generation model designed for general medical image generation tasks to alleviate the problem of scarcity of images with annotation. MEDSAD leverages a simple annotation to generate paired medical images, thereby enhancing the model’s data exploration capabilities and improving segmentation performance in downstream segmentation tasks. To exert greater control over the texture generated from medical images in MEDSAD, we introduce the Texture Style Injection (TSI) mechanism, which restricts the model from generating textures rather than randomly. Additionally, we propose a Feature Frequency Domain Attention (FFDA) module to mitigate the adverse effects of high-frequency noise components during this process. We validate the performance enhancements achieved by MEDSAD on two medical segmentation tasks involving magnetic resonance (MR) and ultrasound (US) image modalities for breast tumor and brain tumor segmentation. The results demonstrate that MEDSAD outputted high-quantity synthetic pair medical images from given annotation, delivering the most substantial performance improvement even with a limited number of samples in the downstream segmentation task. These findings underscore the generalization and efficacy of the proposed model.

Within Neural Networks (NN), the parameters of Adaptive Activation Functions (AAF) control the shapes of activation functions. These parameters are trained along with other parameters in the NN. AAFs have improved performance of Convolutional Neural Networks (CNN) in multiple classification tasks. In this paper, we propose and apply AAFs on CNNs for regression tasks. We argue that applying AAFs in the regression (second-to-last) layer of a NN can significantly decrease the bias of the regression NN. However, using existing AAFs may lead to overfitting. To address this problem, we propose a Smooth Adaptive Activation Function (SAAF) with a piecewise polynomial form which can approximate any continuous function to arbitrary degree of error, while having a bounded Lipschitz constant for given bounded model parameters. As a result, NNs with SAAF can avoid overfitting by simply regularizing model parameters. We empirically evaluated CNNs with SAAFs and achieved state-of-the-art results on age and pose estimation datasets.

Motivation: Sensitivity analysis and parameter tuning are important processes in large-scale image analysis. They are very costly because the image analysis workflows are required to be executed several times to systematically correlate output variations with parameter changes or to tune parameters. An integrated solution with minimum user interaction that uses effective methodologies and high performance computing is required to scale these studies to large imaging datasets and expensive analysis workflows. Results: The experiments with two segmentation workflows show that the proposed approach can (i) quickly identify and prune parameters that are non-influential; (ii) search a small fraction (about 100 points) of the parameter search space with billions to trillions of points and improve the quality of segmentation results (Dice and Jaccard metrics) by as much as 1.42× compared to the results from the default parameters; (iii) attain good scalability on a high performance cluster with several effective optimizations. Conclusions: Our work demonstrates the feasibility of performing sensitivity analyses, parameter studies and auto-tuning with large datasets. The proposed framework can enable the quantification of error estimations and output variations in image segmentation pipelines. Availability and Implementation: Source code: https://github.com/SBU-BMI/region-templates/ . Contact: [email protected]. Supplementary information: Supplementary data are available at Bioinformatics online.

Convolutional Neural Networks (CNN) are state-of-the-art models for many image classification tasks. However, to recognize cancer subtypes automatically, training a CNN on gigapixel resolution Whole Slide Tissue Images (WSI) is currently computationally impossible. The differentiation of cancer subtypes is based on cellular-level visual features observed on image patch scale. Therefore, we argue that in this situation, training a patch-level classifier on image patches will perform better than or similar to an image-level classifier. The challenge becomes how to intelligently combine patch-level classification results and model the fact that not all patches will be discriminative. We propose to train a decision fusion model to aggregate patch-level predictions given by patch-level CNNs, which to the best of our knowledge has not been shown before. Furthermore, we formulate a novel Expectation-Maximization (EM) based method that automatically locates discriminative patches robustly by utilizing the spatial relationships of patches. We apply our method to the classification of glioma and non-small-cell lung carcinoma cases into subtypes. The classification accuracy of our method is similar to the inter-observer agreement between pathologists. Although it is impossible to train CNNs on WSIs, we experimentally demonstrate using a comparable non-cancer dataset of smaller images that a patch-based CNN can outperform an image-based CNN.

In multiatlas segmentation, one typically registers several atlases to the novel image, and their respective segmented label images are transformed and fused to form the final segmentation. In this work, we provide a new dynamical system perspective for multiatlas segmentation, inspired by the following fact: The transformation that aligns the current atlas to the novel image can be not only computed by direct registration but also inferred from the transformation that aligns the previous atlas to the image together with the transformation between the two atlases. This process is similar to the global positioning system on a vehicle, which gets position by inquiring from the satellite and by employing the previous location and velocity-neither answer in isolation being perfect. To solve this problem, a dynamical system scheme is crucial to combine the two pieces of information; for example, a Kalman filtering scheme is used. Accordingly, in this work, a Kalman multiatlas segmentation is proposed to stabilize the global/affine registration step. The contributions of this work are twofold. First, it provides a new dynamical systematic perspective for standard independent multiatlas registrations, and it is solved by Kalman filtering. Second, with very little extra computation, it can be combined with most existing multiatlas segmentation schemes for better registration/segmentation accuracy.

Groupwise statistical analysis of the morphometry of brain structures plays an important role in neuroimaging studies. Nevertheless, most morphometric measurements are often limited to volume and surface area, as further morphological characterization of anatomical structures poses a significant challenge. In this paper, we present a method that allows the detection, localization, and quantification of statistically significant morphological differences in complex brain structures between populations. This is accomplished by a novel level-set framework for shape morphing and a multishape dissimilarity-measure derived by a modified version of the Hausdorff distance. The proposed method does not require explicit one-to-one point correspondences and is fast, robust, and easy to implement regardless of the topological complexity of the anatomical surface under study. The proposed model has been applied to well-defined regions of interest using both synthetic and real data sets. This includes the corpus callosum, striatum, caudate, amygdala-hippocampal complex, and superior temporal gyrus. These structures were selected for their importance with respect to brain regions implicated in a variety of neurological disorders. The synthetic databases allowed quantitative evaluations of the method. Results obtained with real clinical data of Williams syndrome and schizophrenia patients agree with published findings in the psychiatry literature.

The left ventricular myocardium plays a key role in the entire circulation system and an automatic delineation of the myocardium is a prerequisite for most of the subsequent functional analysis. In this paper, we present a complete system for an automatic segmentation of the left ventricular myocardium from cardiac computed tomography (CT) images using the shape information from images to be segmented. The system follows a coarse-to-fine strategy by first localizing the left ventricle and then deforming the myocardial surfaces of the left ventricle to refine the segmentation. In particular, the blood pool of a CT image is extracted and represented as a triangulated surface. Then, the left ventricle is localized as a salient component on this surface using geometric and anatomical characteristics. After that, the myocardial surfaces are initialized from the localization result and evolved by applying forces from the image intensities with a constraint based on the initial myocardial surface locations. The proposed framework has been validated on 34-human and 12-pig CT images, and the robustness and accuracy are demonstrated.

In image segmentation, we are often interested in using certain quantities to characterize the object, and perform the classification based on criteria such as mean intensity, gradient magnitude, and responses to certain predefined filters. Unfortunately, in many cases such quantities are not adequate to model complex textured objects. Along a different line of research, the sparse characteristic of natural signals has been recognized and studied in recent years. Therefore, how such sparsity can be utilized, in a non-parametric way, to model the object texture and assist the textural image segmentation process is studied in this paper, and a segmentation scheme based on the sparse representation of the texture information is proposed. More explicitly, the texture is encoded by the dictionaries constructed from the user initialization. Then, an active contour is evolved to optimize the fidelity of the representation provided by the dictionary of the target. In doing so, not only a non-parametric texture modeling technique is provided, but also the sparsity of the representation guarantees the computation efficiency. The experiments are carried out on the publicly available image data sets which contain a large variety of texture images, to analyze the user interaction, performance statistics, and to highlight the algorithm's capability of robustly extracting textured regions from an image.

The planning and evaluation of left atrial ablation procedures are commonly based on the segmentation of the left atrium, which is a challenging task due to large anatomical variations. In this paper, we propose an automatic approach for segmenting the left atrium from magnetic resonance imagery. The segmentation problem is formulated as a problem in variational region growing. In particular, the method starts locally by searching for a seed region of the left atrium from an MR slice. A global constraint is imposed by applying a shape prior to the left atrium represented by Zernike moments. The overall growing process is guided by the robust statistics of intensities from the seed region along with the shape prior to capture the entire atrial region. The robustness and accuracy of our approach are demonstrated by experimental results from 64 human MR images.

Extracting anatomical and functional significant structures renders one of the important tasks for both the theoretical study of the medical image analysis, and the clinical and practical community. In the past, much work has been dedicated only to the algorithmic development. Nevertheless, for clinical end users, a well designed algorithm with an interactive software is necessary for an algorithm to be utilized in their daily work. Furthermore, the software would better be open sourced in order to be used and validated by not only the authors but also the entire community. Therefore, the contribution of the present work is twofolds: first, we propose a new robust statistics based conformal metric and the conformal area driven multiple active contour framework, to simultaneously extract multiple targets from MR and CT medical imagery in 3D. Second, an open source graphically interactive 3D segmentation tool based on the aforementioned contour evolution is implemented and is publicly available for end users on multiple platforms. In using this software for the segmentation task, the process is initiated by the user drawn strokes (seeds) in the target region in the image. Then, the local robust statistics are used to describe the object features, and such features are learned adaptively from the seeds under a non-parametric estimation scheme. Subsequently, several active contours evolve simultaneously with their interactions being motivated by the principles of action and reaction-this not only guarantees mutual exclusiveness among the contours, but also no longer relies upon the assumption that the multiple objects fill the entire image domain, which was tacitly or explicitly assumed in many previous works. In doing so, the contours interact and converge to equilibrium at the desired positions of the desired multiple objects. Furthermore, with the aim of not only validating the algorithm and the software, but also demonstrating how the tool is to be used, we provide the reader reproducible experiments that demonstrate the capability of the proposed segmentation tool on several public available data sets.

The registration of a pair of point sets as well as the estimation of their pointwise correspondences is a challenging and important task in computer vision. In this paper, we present a method to estimate the diffeomorphic deformation, together with the pointwise correspondences, between a pair of point sets. Many of the registration problems are iteratively solved by estimating the correspondence, locally optimizing certain cost functionals over the rigid or similarity or affine transformation group, then estimating the correspondence again, and so on. This type of approach, however, is well-known to be susceptible to suboptimal local solutions. In this paper, we first adopt the perspective of treating the registration as a posterior estimation optimization problem and solve it accordingly via a particle-filtering framework. Second, within such a framework, the diffeomorphic registration is performed to correct the nonlinear deformation of the points. In doing so, we provide a solution less susceptible to local minima. We provide the experimental results, which include challenging medical data sets where the two point sets differ by 180 (°) rotation as well as local deformations, to highlight the algorithm's capability of robustly finding the more globally optimal solution for the registration task.

Extracting the prostate from magnetic resonance (MR) imagery is a challenging and important task for medical image analysis and surgical planning. We present in this work a unified shape-based framework to extract the prostate from MR prostate imagery. In many cases, shape-based segmentation is a two-part problem. First, one must properly align a set of training shapes such that any variation in shape is not due to pose. Then segmentation can be performed under the constraint of the learnt shape. However, the general registration task of prostate shapes becomes increasingly difficult due to the large variations in pose and shape in the training sets, and is not readily handled through existing techniques. Thus, the contributions of this paper are twofold. We first explicitly address the registration problem by representing the shapes of a training set as point clouds. In doing so, we are able to exploit the more global aspects of registration via a certain particle filtering based scheme. In addition, once the shapes have been registered, a cost functional is designed to incorporate both the local image statistics as well as the learnt shape prior. We provide experimental results, which include several challenging clinical data sets, to highlight the algorithm's capability of robustly handling supine/prone prostate registration and the overall segmentation task.