Speed-of-sound (SoS) is an emerging ultrasound contrast modality, where pulse-echo techniques using conventional transducers offer multiple benefits. For estimating tissue SoS distributions, spatial domain reconstruction from relative speckle shifts between different beamforming sequences is a promising approach. This operates based on a forward model that relates the sought local values of SoS to observed speckle shifts, for which the associated image reconstruction inverse problem is solved. The reconstruction accuracy thus highly depends on the hand-crafted forward imaging model. In this work, we propose to learn the SoS imaging model based on data. We introduce a convolutional formulation of the pulse-echo SoS imaging problem such that the entire field-of-view requires a single unified kernel, the learning of which is then tractable and robust. We present least-squares estimation of such convolutional kernel, which can further be constrained and regularized for numerical stability. In experiments, we show that a forward model learned from k-Wave simulations reduces the contrast error of SoS reconstructions by 38%, compared to a conventional hand-crafted line-based wave-path model. This simulation-learned model generalizes successfully to acquired phantom data, reducing the contrast error compared to the conventional hand-crafted alternative. We successfully demonstrate the feasibility of learning machine-specific kernels as well as one-shot learning from a single image. On in-vivo data of a cancerous breast tumor, the phantom-learned model exhibits an SoS contrast of 34.6m/s, as an impressive improvement over the conventional model contrast of merely 3.4m/s.

Methods for unsupervised domain adaptation (UDA) help to improve the performance of deep neural networks on unseen domains without any labeled data. Especially in medical disciplines such as histopathology, this is crucial since large datasets with detailed annotations are scarce. While the majority of existing UDA methods focus on the adaptation from a labeled source to a single unlabeled target domain, many real-world applications with a long life cycle involve more than one target domain. Thus, the ability to sequentially adapt to multiple target domains becomes essential. In settings where the data from previously seen domains cannot be stored, e.g., due to data protection regulations, the above becomes a challenging continual learning problem. To this end, we propose to use generative feature-driven image replay in conjunction with a dual-purpose discriminator that not only enables the generation of images with realistic features for replay, but also promotes feature alignment during domain adaptation. We evaluate our approach extensively on a sequence of three histopathological datasets for tissue-type classification, achieving state-of-the-art results. We present detailed ablation experiments studying our proposed method components and demonstrate a possible use-case of our continual UDA method for an unsupervised patch-based segmentation task given high-resolution tissue images. Our code is available at: https://github.com/histocartography/multi-scale-feature-alignment.

Deep learning models can achieve high accuracy when trained on large amounts of labeled data. However, real-world scenarios often involve several challenges: Training data may become available in installments, may originate from multiple different domains, and may not contain labels for training. Certain settings, for instance medical applications, often involve further restrictions that prohibit retention of previously seen data due to privacy regulations. In this work, to address such challenges, we study unsupervised segmentation in continual learning scenarios that involve domain shift. To that end, we introduce GarDA (Generative Appearance Replay for continual Domain Adaptation), a generative-replay based approach that can adapt a segmentation model sequentially to new domains with unlabeled data. In contrast to single-step unsupervised domain adaptation (UDA), continual adaptation to a sequence of domains enables leveraging and consolidation of information from multiple domains. Unlike previous approaches in incremental UDA, our method does not require access to previously seen data, making it applicable in many practical scenarios. We evaluate GarDA on three datasets with different organs and modalities, where it substantially outperforms existing techniques. Our code is available at: https://github.com/histocartography/generative-appearance-replay.

Cancer diagnosis, prognosis, and therapy response predictions from tissue specimens highly depend on the phenotype and topological distribution of constituting histological entities. Thus, adequate tissue representations for encoding histological entities is imperative for computer aided cancer patient care. To this end, several approaches have leveraged cell-graphs, capturing the cell-microenvironment, to depict the tissue. These allow for utilizing graph theory and machine learning to map the tissue representation to tissue functionality, and quantify their relationship. Though cellular information is crucial, it is incomplete alone to comprehensively characterize complex tissue structure. We herein treat the tissue as a hierarchical composition of multiple types of histological entities from fine to coarse level, capturing multivariate tissue information at multiple levels. We propose a novel multi-level hierarchical entity-graph representation of tissue specimens to model the hierarchical compositions that encode histological entities as well as their intra- and inter-entity level interactions. Subsequently, a hierarchical graph neural network is proposed to operate on the hierarchical entity-graph and map the tissue structure to tissue functionality. Specifically, for input histology images, we utilize well-defined cells and tissue regions to build HierArchical Cell-to-Tissue (HACT) graph representations, and devise HACT-Net, a message passing graph neural network, to classify the HACT representations. As part of this work, we introduce the BReAst Carcinoma Subtyping (BRACS) dataset, a large cohort of Haematoxylin & Eosin stained breast tumor regions-of-interest, to evaluate and benchmark our proposed methodology against pathologists and state-of-the-art computer-aided diagnostic approaches. Through comparative assessment and ablation studies, our proposed method is demonstrated to yield superior classification results compared to alternative methods as well as individual pathologists. The code, data, and models can be accessed at https://github.com/histocartography/hact-net.

Explainability of deep learning methods is imperative to facilitate their clinical adoption in digital pathology. However, popular deep learning methods and explainability techniques (explainers) based on pixel-wise processing disregard biological entities’ notion, thus complicating comprehension by pathologists. In this work, we address this by adopting biological entity-based graph processing and graph explainers enabling explanations accessible to pathologists. In this context, a major challenge becomes to discern meaningful explainers, particularly in a standardized and quantifiable fashion. To this end, we propose herein a set of novel quantitative metrics based on statistics of class separability using pathologically measurable concepts to characterize graph explainers. We employ the proposed metrics to evaluate three types of graph explainers, namely the layer-wise relevance propagation, gradient-based saliency, and graph pruning approaches, to explain Cell-Graph representations for Breast Cancer Subtyping. The proposed metrics are also applicable in other domains by using domain-specific intuitive concepts. We validate the qualitative and quantitative findings on the BRACS dataset, a large cohort of breast cancer RoIs, by expert pathologists. The code, data, and models can be accessed here1.

We present a novel texture synthesis framework, enabling the generation of infinite, high-quality 3D textures given a 2D exemplar image. Inspired by recent advances in natural texture synthesis, we train deep neural models to generate textures by non-linearly combining learned noise frequencies. To achieve a highly realistic output conditioned on an exemplar patch, we propose a novel loss function that combines ideas from both style transfer and generative adversarial networks. In particular, we train the synthesis network to match the Gram matrices of deep features from a discriminator network. In addition, we propose two architectural concepts and an extrapolation strategy that significantly improve generalization performance. In particular, we inject both model input and condition into hidden network layers by learning to scale and bias hidden activations. Quantitative and qualitative evaluations on a diverse set of exemplars motivate our design decisions and show that our system performs superior to previous state of the art. Finally, we conduct a user study that confirms the benefits of our framework.

Abstract Ray‐based simulations have been shown to generate impressively realistic ultrasound images in interactive frame rates. Recent efforts used GPU‐based surface raytracing to simulate complex ultrasound interactions such as multiple reflections and refractions. These methods are restricted to perfectly specular reflections (i.e. following only a single reflective/refractive ray), whereas real tissue exhibits roughness of varying degree at tissue interfaces, causing partly diffuse reflections and refractions. Such surface interactions are significantly more complex and can in general not be handled by conventional deterministic raytracing approaches. However, these can be efficiently computed by Monte‐Carlo sampling techniques, where many ray paths are generated with respect to a probability distribution. In this paper, we introduce Monte‐Carlo raytracing for ultrasound simulation. This enables the realistic simulation of ultrasound‐tissue interactions such as soft shadows and fuzzy reflections. We discuss how to properly weight the contribution of each ray path in order to simulate the behaviour of a beamformed ultrasound signal. Tracing many individual rays per transducer element is easily parallelizable on modern GPUs, as opposed to previous approaches based on recursive binary raytracing. We further propose a significant performance optimization based on adaptive sampling.

Numerical simulation of ultrasound images can facilitate the training of sonographers. An efficient and realistic model for the simulation of ultrasonic speckle is the convolution of the ultrasound point-spread function with a distribution of point scatterers. Nevertheless, for a given arbitrary tissue type, a scatterer map that would generate a realistic appearance of that tissue is not known a priori. In this paper, we introduce a principled approach to estimate (reconstruct) such a scatterer map from images, by solving the inverse-problem of ultrasound speckle formation, such that images from arbitrary view angles and transducer settings can be generated from those scatterer maps later in simulations. Robust reconstructions are achieved by using multiple measurements of the same tissue with different viewing parameters. For this purpose, a novel use of beam-steering to rapidly and conveniently acquire multiple images of the same scene is proposed. We demonstrate in numerical and physical phantoms and images that the appearance of synthesized images closely match real images for a range of viewing parameters and probe settings. We also present a scene editing scenario exploiting these scatterer representations to create realistic images of augmented anatomy.

Biomedical parameters of tissue can be important indicators for clinical diagnosis. One such parameter that reflects tissue stiffness is elasticity, the imaging of which is called elastography. In this paper, we use displacements from harmonic excitations to solve the inverse problem of elasticity based on a finite-element method (FEM) formulation. This leads to iterative solution of nonlinear and nonconvex problems. In this paper, we show the importance and selection of viable initializations in numerical simulation studies and propose techniques for the fusion of multiple initializations for ideal reconstructions of unknown tissue as well as combining information from excitations at multiple frequencies. Results show that our method leads up to 76% decrease in root-mean-squared error (RMSE) and 9.9 dB increase in contrast-to-noise ratio (CNR) in simulations with noise, when compared to conventional iterative FEM without multiple initializations and frequencies. As the wave patterns in individually selected frequencies may introduce artifacts, a joint inverse-problem solution of multi-frequency excitations is introduced as a robust solution, where CNR improvements of up to 11.9 dB are observed. We also present the methods on a tissue-mimicking gelatin phantom study using mechanical excitation and ultrafast plane-wave ultrasound imaging, where the RMSE was improved by up to 51%. An experiment of ablation via heating an ex-vivo bovine liver shows that reconstruction artifacts are reduced with our proposed method.

We present a novel technique for real-time deformable registration of 3-D to 2.5-D transrectal ultrasound (TRUS) images for image-guided, robot-assisted laparoscopic radical prostatectomy (RALRP). For RALRP, a pre-operatively acquired 3-D TRUS image is registered to thin-volumes comprised of consecutive intra-operative 2-D TRUS images, where the optimal transformation is found using a gradient descent method based on analytical first and second order derivatives. Our method relies on an efficient algorithm for real-time extraction of arbitrary slices from a 3-D image deformed given a discrete mesh representation. We also propose and demonstrate an evaluation method that generates simulated models and images for RALRP by modeling tissue deformation through patient-specific finite-element models (FEM). We evaluated our method on in-vivo data from 11 patients collected during RALRP and focal therapy interventions. In the presence of an average landmark deformation of 3.89 and 4.62 mm, we achieved accuracies of 1.15 and 0.72 mm, respectively, on the synthetic and in-vivo data sets, with an average registration computation time of 264 ms, using MATLAB on a conventional PC. The results show that the real-time tracking of the prostate motion and deformation is feasible, enabling a real-time augmented reality-based guidance system for RALRP.].

-norm) is unable to correctly represent non-smooth displacement fields, that can, for example, occur at sliding interfaces in the thorax and abdomen in image time-series during respiration. In this paper, isotropic Total Variation (TV) regularization is used to enable accurate registration near such interfaces. We further develop the TV-regularization for parametric displacement fields and provide an efficient numerical solution scheme using the Alternating Directions Method of Multipliers (ADMM). The proposed method was successfully applied to four clinical databases which capture breathing motion, including CT lung and MR liver images. It provided accurate registration results for the whole volume. A key strength of our proposed method is that it does not depend on organ masks that are conventionally required by many algorithms to avoid errors at sliding interfaces. Furthermore, our method is robust to parameter selection, allowing the use of the same parameters for all tested databases. The average target registration error (TRE) of our method is superior (10% to 40%) to other techniques in the literature. It provides precise motion quantification and sliding detection with sub-pixel accuracy on the publicly available breathing motion databases (mean TREs of 0.95 mm for DIR 4D CT, 0.96 mm for DIR COPDgene, 0.91 mm for POPI databases).

Variations in the shape and appearance of anatomical structures in medical images are often relevant radiological signs of disease. Automatic tools can help automate parts of this manual process. A cloud-based evaluation framework is presented in this paper including results of benchmarking current state-of-the-art medical imaging algorithms for anatomical structure segmentation and landmark detection: the VISCERAL Anatomy benchmarks. The algorithms are implemented in virtual machines in the cloud where participants can only access the training data and can be run privately by the benchmark administrators to objectively compare their performance in an unseen common test set. Overall, 120 computed tomography and magnetic resonance patient volumes were manually annotated to create a standard Gold Corpus containing a total of 1295 structures and 1760 landmarks. Ten participants contributed with automatic algorithms for the organ segmentation task, and three for the landmark localization task. Different algorithms obtained the best scores in the four available imaging modalities and for subsets of anatomical structures. The annotation framework, resulting data set, evaluation setup, results and performance analysis from the three VISCERAL Anatomy benchmarks are presented in this article. Both the VISCERAL data set and Silver Corpus generated with the fusion of the participant algorithms on a larger set of non-manually-annotated medical images are available to the research community.

In surface-based registration for image-guided interventions, the presence of missing data can be a significant issue. This often arises with real-time imaging modalities such as ultrasound, where poor contrast can make tissue boundaries difficult to distinguish from surrounding tissue. Missing data poses two challenges: ambiguity in establishing correspondences; and extrapolation of the deformation field to those missing regions. To address these, we present a novel non-rigid registration method. For establishing correspondences, we use a probabilistic framework based on a Gaussian mixture model (GMM) that treats one surface as a potentially partial observation. To extrapolate and constrain the deformation field, we incorporate biomechanical prior knowledge in the form of a finite element model (FEM). We validate the algorithm, referred to as GMM-FEM, in the context of prostate interventions. Our method leads to a significant reduction in target registration error (TRE) compared to similar state-of-the-art registration algorithms in the case of missing data up to 30%, with a mean TRE of 2.6 mm. The method also performs well when full segmentations are available, leading to TREs that are comparable to or better than other surface-based techniques. We also analyze robustness of our approach, showing that GMM-FEM is a practical and reliable solution for surface-based registration.

In this paper, a novel Markov random field (MRF)-based approach is presented for segmenting medical images while simultaneously registering an atlas nonrigidly. In the literature, both segmentation and registration have been studied extensively. For applications that involve both, such as segmentation via atlas-based registration, earlier studies proposed addressing these problems iteratively by feeding the output of each to initialize the other. This scheme, however, cannot guarantee an optimal solution for the combined task at hand, since these two individual problems are then treated separately. In this paper, we formulate simultaneous registration and segmentation (SRS) as a maximum a-posteriori (MAP) problem. We decompose the resulting probabilities such that the MAP inference can be done using MRFs. An efficient hierarchical implementation is employed, allowing coarse-to-fine registration while estimating segmentation at pixel level. The method is evaluated on two clinical data sets: 1) mandibular bone segmentation in 3D CT and 2) corpus callosum segmentation in 2D midsaggital slices of brain MRI. A video tracking example is also given. Our implementation allows us to directly compare the proposed method with the individual segmentation/registration and the iterative approach using the exact same potential functions. In a leave-one-out evaluation, SRS demonstrated more accurate results in terms of dice overlap and surface distance metrics for both data sets. We also show quantitatively that the SRS method is less sensitive to the errors in the registration as opposed to the iterative approach.

Interaction with virtual deformable models is common in several haptic contexts, such as in medical training simulators. This paper presents a methodological procedure for the creation of such virtual models from their real-life counterparts. Both the surface geometry and the elastic parametrization of an object are reconstructed from position/force readings during an operator-assisted exploration of the object. A 3D mesh model is then generated from the surface contact points. The internal elastic modulus is found using the 3D finite element method. This modeling method is compared with two common 1D elastic models, namely Kelvin-Voigt and Hunt-Crossley. Results using three deformable homogeneous silicone samples show successful geometry reconstruction. 1D model parameterizations exhibit high variation dependent on geometry and contact location. In contrast, elastic modulus reconstruction yields a global model parameterization independent of geometry. Elastic moduli estimated in experiments correlated with their known values, and were shown to be reproducible among samples with different geometries.

The finite element method is commonly used to model tissue deformation in order to solve for unknown parameters in the inverse problem of viscoelasticity. Typically, a (regular-grid) structured mesh is used since the internal geometry of the domain to be identified is not known a priori. In this work, the generation of problem-specific meshes is studied and such meshes are shown to significantly improve inverse-problem elastic parameter reconstruction. Improved meshes are generated from axial strain images, which provide an approximation to the underlying structure, using an optimization-based mesh adaptation approach. Such strain-based adapted meshes fit the underlying geometry even at coarse mesh resolutions, therefore improving the effective resolution of the reconstruction at a given mesh size/complexity. Elasticity reconstructions are then performed iteratively using the reflective trust-region method for optimizing the fit between estimated and observed displacements. This approach is studied for Young's modulus reconstruction at various mesh resolutions through simulations, yielding 40%-72% decrease in root-mean-square reconstruction error and 4-52 times improvement in contrast-to-noise ratio in simulations of a numerical phantom with a circular inclusion. A noise study indicates that conventional structured meshes with no noise perform considerably worse than the proposed adapted meshes with noise levels up to 20% of the compression amplitude. A phantom study and preliminary in vivo results from a breast tumor case confirm the benefit of the proposed technique. Not only conventional axial strain images but also other elasticity approximations can be used to adapt meshes. This is demonstrated on images generated by combining axial strain and axial-shear strain, which enhances lateral image contrast in particular settings, consequently further improving mesh-adapted reconstructions.

This paper presents a haptic simulator for prostate brachytherapy. Both needle insertion and the manipulation of the transrectal ultrasound (TRUS) probe are controlled via haptic devices. These are used to render tissue interaction forces computed using a deformable tissue model based on the finite element method (FEM). Needle flexibility and lateral needle bevel forces are also simulated. The TRUS-tissue simulation allows a trainee to practice the 3D intra-operative placement of the TRUS probe for registration with the pre-operative volume study. The needle-tissue simulation allows a trainee to practice needle insertion and targeting. The TRUS probe and the needle can be maneuvered simultaneously. Approaches to computational acceleration for real-time haptic performance are presented. Trade-offs between accuracy and speed are discussed. A graphics-card implementation of the numerically intensive mesh-adaptation operation is also presented.

In medical simulations involving tissue deformation, the finite element method (FEM) is a widely used technique, where the size, shape, and placement of the elements in a model are important factors that affect the interpolation and numerical errors of a solution. Conventional model generation schemes for FEM consist of a segmentation step delineating the anatomy followed by a meshing step generating elements conforming to this segmentation. In this paper, a single-step model generation technique is proposed based on optimization. Starting from an initial mesh covering the domain of interest, the mesh nodes are adjusted to minimize an objective function which penalizes intra-element intensity variations and poor element geometry for the entire mesh. Trade-offs between mesh geometry quality and intra-element variance are achieved by adjusting the relative weights of the geometric and intensity variation components of the cost function. This meshing approach enables a more accurate rendering of shapes with fewer elements and provides more accurate models for deformation simulation, especially when the image intensities represent a mechanical feature of the tissue such as the elastic modulus. The use of the proposed mesh optimization is demonstrated in 2-D and 3-D on synthetic phantoms, MR images of the brain, and CT images of the kidney. A comparison with previous meshing techniques that do not account for image intensity is also provided demonstrating the benefits of our approach.

This paper presents an algorithm for fast image synthesis inside deformed volumes. Given the node displacements of a mesh and a reference 3-D image dataset of a predeformed volume, the method first maps the image pixels that need to be synthesized from the deformed configuration to the nominal predeformed configuration, where the pixel intensities are obtained easily through interpolation in the regular-grid structure of the reference voxel volume. This mapping requires the identification of the mesh element enclosing each pixel for every image frame. To accelerate this point location operation, a fast method of projecting the deformed mesh on image pixels is introduced in this paper. The method presented was implemented for ultrasound B-mode image simulation of a synthetic tissue phantom. The phantom deformation as a result of ultrasound probe motion was modeled using the finite element method. Experimental images of the phantom under deformation were then compared with the corresponding synthesized images using sum of squared differences and mutual information metrics. Both this quantitative comparison and a qualitative assessment show that realistic images can be synthesized using the proposed technique. An ultrasound examination system was also implemented to demonstrate that real-time image synthesis with the proposed technique can be successfully integrated into a haptic simulation.