Medical scans often include patient identifiers and clinical annotations that must be removed prior to data sharing or use in downstream model training. With machine learning now central to clinical imaging analysis, reliable removal of such non-imaging artifacts is essential for preserving patient privacy, reducing bias, and improving data quality. However, this crucial curation step is frequently overlooked or addressed heuristically.We present a deep learning framework that automatically detects and removes overlaid text, markers, and other non-imaging elements from clinical scans while restoring the underlying image content. The model comprises two components: a detection module that localizes non-imaging regions, and a dual-generator architecture for unsupervised image decomposition, where one generator reconstructs the imaging content and the other produces the non-imaging components. Unlike conventional inpainting, our method bypasses explicit segmentation by leveraging explainable AI (XAI) maps from the detection module to guide artifact masking and restoration.We demonstrate robust curation performance on three datasets, one MRI and two ultrasound, for both public and private sources. Results show high visual quality (Turing-test validated) and strong quantitative scores. Importantly, training downstream classification and segmentation models with scans curated by our method substantially improves results compared to models trained on data containing overlaid annotations. In fact, our performance on various metrics (e.g., accuracy, F1 score, IoU, and Dice) is comparable to those obtained with clean, marker-free training data. Our code and resources are available at: https://github.com/YaelElkin/DeepImageCuration

Modern deep neural networks have now reached human-level performance across a variety of tasks. However, unlike humans they lack the ability to explain their decisions by showing where and telling what concepts guided them. In this work, we present a unified framework for transforming any vision neural network into a spatially and conceptually interpretable model. We introduce a spatially-aware concept bottleneck layer that projects "black-box" features of pre-trained backbone models into interpretable concept maps, without requiring human labels. By training a classification layer over this bottleneck, we obtain a self-explaining model that articulates which concepts most influenced its prediction, along with heatmaps that ground them in the input image. Accordingly, we name this method "Spatially-Aware and Label-Free Concept Bottleneck Model" (SALF-CBM). Our results show that the proposed SALF-CBM: (1) Outperforms non-spatial CBM methods, as well as the original backbone, on a variety of classification tasks; (2) Produces high-quality spatial explanations, outperforming widely used heatmap-based methods on a zero-shot segmentation task; (3) Facilitates model exploration and debugging, enabling users to query specific image regions and refine the model’s decisions by locally editing its concept maps.

White matter tractography is an advanced neuroimaging technique that reconstructs the 3D white matter pathways of the brain from diffusion MRI data. It can be framed as a pathfinding problem aiming to infer neural fiber trajectories from noisy and ambiguous measurements, facing challenges such as crossing, merging, and fanning white-matter configurations. In this paper, we propose a novel tractography method that leverages Transformers to model the sequential nature of white matter streamlines, enabling the prediction of fiber directions by integrating both the trajectory context and current diffusion MRI measurements. To incorporate spatial information, we utilize CNNs that extract microstructural features from local neighborhoods around each voxel. By combining these complementary sources of information, our approach improves the precision and completeness of neural pathway mapping compared to traditional tractography models. We evaluate our method with the Tractometer toolkit, achieving competitive performance against state-of-the-art approaches, and present qualitative results on the TractoInferno dataset, demonstrating strong generalization to real-world data. Our code is publicly available at https://github.com/ItzikWaizman/TractoTransformer.

The integration of diverse clinical modalities such as medical imaging and the tabular data extracted from patients’ Electronic Health Records (EHRs) is a crucial aspect of modern healthcare. Integrative analysis of multiple sources can provide a comprehensive understanding of the clinical condition of a patient, improving diagnosis and treatment decision. Deep Neural Networks (DNNs) consistently demonstrate outstanding performance in a wide range of multimodal tasks in the medical domain. However, the complex endeavor of effectively merging medical imaging with clinical, demographic and genetic information represented as numerical tabular data remains a highly active and ongoing research pursuit. We present a novel framework based on hypernetworks to fuse clinical imaging and tabular data by conditioning the image processing on the EHR’s values and measurements. This approach aims to leverage the complementary information present in these modalities to enhance the accuracy of various medical applications. We demonstrate the strength and generality of our method on two different brain Magnetic Resonance Imaging (MRI) analysis tasks, namely, brain age prediction conditioned by subject’s sex and multi-class Alzheimer’s Disease (AD) classification conditioned by tabular data. We show that our framework outperforms both single-modality models and state-of-the-art MRI tabular data fusion methods. A link to our code can be found at https://github.com/daniel4725/HyperFusion . • We present a HyperFusion network - a novel hypernetwork for medical imaging and tabular data fusion. • A hypernetwork controls a primary network by producing parameters to predefined layers. • This mechanism is exploited to condition image processing predictions by tabular data. • The HyperFusion outperforms existing imaging-tabular fusion methods for Alzheimer’s disease classification. • The HyperFusion versatility is demonstrated for brain age prediction conditioned by sex.

We introduce the HueNet - a novel deep learning framework for a differentiable construction of intensity (1D) and joint (2D) histograms and present its applicability to paired and unpaired image-to-image translation problems. The key idea is an innovative technique for augmenting a generative neural network by histogram layers appended to the image generator. These histogram layers allow us to define two new histogram-based loss functions for constraining the structural appearance of the synthesized output image and its color distribution. Specifically, the color similarity loss is defined by the Earth Mover's Distance between the intensity histograms of the network output and a color reference image. The structural similarity loss is determined by the mutual information between the output and a content reference image based on their joint histogram. Although the HueNet can be applied to a variety of image-to-image translation problems, we chose to demonstrate its strength on the tasks of color transfer, exemplar-based image colorization, and edges → photo, where the colors of the output image are predefined. The code is available at https://github.com/mor-avi-aharon-bgu/HueNet.git.

We present a novel deep learning approach to categorical segmentation of lung CTs of COVID-19 patients. Specifically, we partition the scans into healthy lung tissues, non-lung regions, and two different, yet visually similar, pathological lung tissues, namely, ground-glass opacity and consolidation. This is accomplished via a unique, end-to-end hierarchical network architecture and ensemble learning, which contribute to the segmentation and provide a measure for segmentation uncertainty.The proposed framework achieves competitive results and outstanding generalization capabilities for three COVID-19 datasets. Our method is ranked second in a public Kaggle competition for COVID-19 CT images segmentation. Moreover, segmentation uncertainty regions are shown to correspond to the disagreements between the manual annotations of two different radiologists. Finally, preliminary promising correspondence results are shown for our private dataset when comparing the patients’ COVID-19 severity scores (based on clinical measures), and the segmented lung pathologies. Code and data are available at our repository1.

Convolutional Neural Networks (CNNs) are considered state of the art segmentation methods for biomedical images in general and microscopy sequences of living cells, in particular. The success of the CNNs is attributed to their ability to capture the structural properties of the data, which enables accommodating complex spatial structures of the cells, low contrast, and unclear boundaries. However, in their standard form CNNs do not exploit the temporal information available in time-lapse sequences, which can be crucial to separating touching and partially overlapping cell instances. In this work, we exploit cell dynamics using a novel CNN architecture which allows multi-scale spatio-temporal feature extraction. Specifically, a novel recurrent neural network (RNN) architecture is proposed based on the integration of a Convolutional Long Short Term Memory (ConvLSTM) network with the U-Net. The proposed ConvLSTM-UNet network is constructed as a dual-task network to enable training with weakly annotated data, in the form of approximate cell centers, termed markers, when the complete cells' outlines are not available. We further use the fast marching method to facilitate the partitioning of clustered cells into individual connected components. Finally, we suggest an adaptation of the method for 3D microscopy sequences without drastically increasing the computational load. The method was evaluated on the Cell Segmentation Benchmark and was ranked among the top three methods on six submitted datasets. Exploiting the proposed built-in marker estimator we also present state-of-the-art cell detection results for an additional, publicly available, weekly annotated dataset. The source code is available at https://gitlab.com/shaked0/lstmUnet.

MOTIVATION: Cell microscopy datasets have great diversity due to variability in cell types, imaging techniques and protocols. Existing methods are either tailored to specific datasets or are based on supervised learning, which requires comprehensive manual annotations. Using the latter approach, however, poses a significant difficulty due to the imbalance between the number of mitotic cells with respect to the entire cell population in a time-lapse microscopy sequence. RESULTS: We present a fully unsupervised framework for both mitosis detection and mother-daughters association in fluorescence microscopy data. The proposed method accommodates the difficulty of the different cell appearances and dynamics. Addressing symmetric cell divisions, a key concept is utilizing daughters' similarity. Association is accomplished by defining cell neighborhood via a stochastic version of the Delaunay triangulation and optimization by dynamic programing. Our framework presents promising detection results for a variety of fluorescence microscopy datasets of different sources, including 2D and 3D sequences from the Cell Tracking Challenge. AVAILABILITY AND IMPLEMENTATION: Code is available in github (github.com/topazgl/mitodix). SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.

Image segmentation is often considered a deterministic process with a single ground truth. Nevertheless, in practice, and in particular, when medical imaging analysis is considered, the extraction of regions of interest (ROIs) is ill-posed and the concept of `most probable' segmentation is model-dependent. In this paper, a measure for segmentation uncertainty in the form of segmentation error margins is introduced. This measure provides a goodness quantity and allows a `fully informed' comparison between extracted boundaries of related ROIs as well as more meaningful statistical analysis. The tool we present is based on a novel technique for segmentation sampling in the Fourier domain and Markov Chain Monte Carlo (MCMC). The method was applied to cortical and sub-cortical structure segmentation in MRI. Since the accuracy of segmentation error margins cannot be validated, we use receiver operating characteristic (ROC) curves to support the proposed method. Precision and recall scores with respect to expert annotation suggest this method as a promising tool for a variety of medical imaging applications including user-interactive segmentation, patient follow-up, and cross-sectional analysis.

We present a novel computational approach to detect white-matter brain impairments following stroke or Traumatic Brain Injuries (TBI). A key assumption in our study is that the two hemispheres are not affected identically. The pathology of white matter (WM) tracts can be thus identified according to the asymmetry level of brain diffusivity measures, such as Fractional Anisotropy (FA) and Mean Diffusivity (MD), extracted from Diffusion Tensor Imaging (DTI). The proposed methodological contribution is based on the construction of a sequence of isosurfaces of these scalar measures and their symmetrical counterparts, obtained by reflecting the original isosurfaces. The modified Hausdorff distance is then used to measure the dissimilarity between each corresponding pair of aligned surfaces. The proposed method is assessed using datasets of normal controls (NCs), stroke patients and longitudinal brain scans of football players that might have been exposed to mild head traumas. Increased asymmetry with respect to NCs is shown for the stroke patients and some of the players indicating possible WM injuries.

We introduce a generative probabilistic model for segmentation of brain lesions in multi-dimensional images that generalizes the EM segmenter, a common approach for modelling brain images using Gaussian mixtures and a probabilistic tissue atlas that employs expectation-maximization (EM), to estimate the label map for a new image. Our model augments the probabilistic atlas of the healthy tissues with a latent atlas of the lesion. We derive an estimation algorithm with closed-form EM update equations. The method extracts a latent atlas prior distribution and the lesion posterior distributions jointly from the image data. It delineates lesion areas individually in each channel, allowing for differences in lesion appearance across modalities, an important feature of many brain tumor imaging sequences. We also propose discriminative model extensions to map the output of the generative model to arbitrary labels with semantic and biological meaning, such as "tumor core" or "fluid-filled structure", but without a one-to-one correspondence to the hypo- or hyper-intense lesion areas identified by the generative model. We test the approach in two image sets: the publicly available BRATS set of glioma patient scans, and multimodal brain images of patients with acute and subacute ischemic stroke. We find the generative model that has been designed for tumor lesions to generalize well to stroke images, and the extended discriminative -discriminative model to be one of the top ranking methods in the BRATS evaluation.

In this paper we report the set-up and results of the Multimodal Brain Tumor Image Segmentation Benchmark (BRATS) organized in conjunction with the MICCAI 2012 and 2013 conferences. Twenty state-of-the-art tumor segmentation algorithms were applied to a set of 65 multi-contrast MR scans of low- and high-grade glioma patients-manually annotated by up to four raters-and to 65 comparable scans generated using tumor image simulation software. Quantitative evaluations revealed considerable disagreement between the human raters in segmenting various tumor sub-regions (Dice scores in the range 74%-85%), illustrating the difficulty of this task. We found that different algorithms worked best for different sub-regions (reaching performance comparable to human inter-rater variability), but that no single algorithm ranked in the top for all sub-regions simultaneously. Fusing several good algorithms using a hierarchical majority vote yielded segmentations that consistently ranked above all individual algorithms, indicating remaining opportunities for further methodological improvements. The BRATS image data and manual annotations continue to be publicly available through an online evaluation system as an ongoing benchmarking resource.

Groupwise statistical analysis of the morphometry of brain structures plays an important role in neuroimaging studies. Nevertheless, most morphometric measurements are often limited to volume and surface area, as further morphological characterization of anatomical structures poses a significant challenge. In this paper, we present a method that allows the detection, localization, and quantification of statistically significant morphological differences in complex brain structures between populations. This is accomplished by a novel level-set framework for shape morphing and a multishape dissimilarity-measure derived by a modified version of the Hausdorff distance. The proposed method does not require explicit one-to-one point correspondences and is fast, robust, and easy to implement regardless of the topological complexity of the anatomical surface under study. The proposed model has been applied to well-defined regions of interest using both synthetic and real data sets. This includes the corpus callosum, striatum, caudate, amygdala-hippocampal complex, and superior temporal gyrus. These structures were selected for their importance with respect to brain regions implicated in a variety of neurological disorders. The synthetic databases allowed quantitative evaluations of the method. Results obtained with real clinical data of Williams syndrome and schizophrenia patients agree with published findings in the psychiatry literature.

We introduce a novel variational method for the extraction of objects with either bilateral or rotational symmetry in the presence of perspective distortion. Information on the symmetry axis of the object and the distorting transformation is obtained as a by--product of the segmentation process. The key idea is the use of a flip or a rotation of the image to segment as if it were another view of the object. We call this generated image the symmetrical counterpart image. We show that the symmetrical counterpart image and the source image are related by planar projective homography. This homography is determined by the unknown planar projective transformation that distorts the object symmetry. The proposed segmentation method uses a level-set-based curve evolution technique. The extraction of the object boundaries is based on the symmetry constraint and the image data. The symmetrical counterpart of the evolving level-set function provides a dynamic shape prior. It supports the segmentation by resolving possible ambiguities due to noise, clutter, occlusions, and assimilation with the background. The homography that aligns the symmetrical counterpart to the source level-set is recovered via a registration process carried out concurrently with the segmentation. Promising segmentation results of various images of approximately symmetrical objects are shown.

Video surveillance systems produce huge amounts of data for storage and display. Long-term human monitoring of the acquired video is impractical and ineffective. Automatic abnormal motion detection system which can effectively attract operator attention and trigger recording is therefore the key to successful video surveillance in dynamic scenes, such as airport terminals. This paper presents a novel solution for real-time abnormal motion detection. The proposed method is well-suited for modern video-surveillance architectures, where limited computing power is available near the camera for compression and communication. The algorithm uses the macroblock motion vectors that are generated in any case as part of the video compression process. Motion features are derived from the motion vectors. The statistical distribution of these features during normal activity is estimated by training. At the operational stage, improbable-motion feature values indicate abnormal motion. Experimental results demonstrate reliable real-time operation.

Shape symmetry is an important cue for image understanding. In the absence of more detailed prior shape information, segmentation can be significantly facilitated by symmetry. However, when symmetry is distorted by perspectivity, the detection of symmetry becomes non-trivial, thus complicating symmetry-aided segmentation. We present an original approach for segmentation of symmetrical objects accommodating perspective distortion. The key idea is the use of the replicative form induced by the symmetry for challenging segmentation tasks. This is accomplished by dynamic extraction of the object boundaries, based on the image gradients, gray levels or colors, concurrently with registration of the image symmetrical counterpart (e.g. reflection) to itself. The symmetrical counterpart of the evolving object contour supports the segmentation by resolving possible ambiguities due to noise, clutter, distortion, shadows, occlusions and assimilation with the background. The symmetry constraint is integrated in a comprehensive level-set functional for segmentation that determines the evolution of the delineating contour. The proposed framework is exemplified on various images of skewsymmetrical objects and its superiority over state of the art variational segmentation techniques is demonstrated.

Object detection and segmentation can be facilitated by the availability of a reference object. However, accounting for possible transformations between the different object views, as part of the segmentation process, remains a challenge. Recent works address this problem by using comprehensive training data. Other approaches are applicable only to limited object classes or can only accommodate similarity transformations. We suggest a novel variational approach to prior-based segmentation, which accounts for planar projective transformation, using a single reference object. The prior shape is registered concurrently with the segmentation process, without point correspondence. The algorithm detects the object of interest and correctly extracts its boundaries. The homography between the two object views is accurately recovered as well. Extending the Chan-Vese level set framework, we propose a region-based segmentation functional that includes explicit representation of the projective homography between the prior shape and the shape to segment. The formulation is derived from two-view geometry. Segmentation of a variety of objects is demonstrated and the recovered transformation is verified.

The paper addresses the problem of "class-based" image-based recognition and rendering with varying illumination. The rendering problem is defined as follows: Given a single input image of an object and a sample of images with varying illumination conditions of other objects of the same general class, re-render the input image to simulate new illumination conditions. The class-based recognition problem is similarly defined: Given a single image of an object in a database of images of other objects, some of them multiply sampled under varying illumination, identify (match) any novel image of that object under varying illumination with the single image of that object in the database. We focus on Lambertian surface classes and, in particular, the class of human faces. The key result in our approach is based on a definition of an illumination invariant signature image which enables an analytic generation of the image space with varying illumination. We show that a small database of objects-in our experiments as few as two objects-is sufficient for generating the image space with varying illumination of any new object of the class from a single input image of that object. In many cases, the recognition results outperform by far conventional methods and the re-rendering is of remarkable quality considering the size of the database of example images and the mild preprocess required for making the algorithm work.

The paper addresses the problem of "class-based" recognition and image-synthesis with varying illumination. The class-based synthesis and recognition tasks are defined as follows: given a single input image of an object, and a sample of images with varying illumination conditions of other objects of the same general class, capture the equivalence relationship (by generation of new images or by invariants) among all images of the object corresponding to new illumination conditions. The key result in our approach is based on a definition of an illumination invariant signature image, we call the "quotient" image, which enables an analytic generation of the image space with varying illumination from a single input image and a very small sample of other objects of the class-in our experiments as few as two objects. In many cases the recognition results outperform by far conventional methods and the image-synthesis is of remarkable quality considering the size of the database of example images and the mild pre-process required for making the algorithm work.