Abstract Deep neural networks are widely used for image segmentation, also in sensitive applications such as medical imaging or autonomous driving. However, few explainable AI methods are available that help developers understand such networks beyond classification. We demonstrate that studying how segmentation networks behave in scale space, i.e., when increasingly blurring their input, conveys relevant insight into whether detections require features such as sharp edges or high‐frequency textures, and what priors have been learned implicitly. We introduce a visual analytics framework that supports a systematic qualitative and quantitative investigation of how segmentations evolve across scale space. In particular, summary visualizations for individual images support the formation of hypotheses that are subsequently formalized via customizable feature representations and evaluated on larger sets of test images using interactive embeddings and quantitative plots. On a cardiac MRI dataset, we demonstrate that this framework reveals important differences between a convolutional and a transformer‐based neural network that, at first glance, appear to produce similar results. We confirm the practical relevance of those insights on out‐of‐distribution images from an MR scanner from which no images were included in the training data. Beyond this primary analysis, we provide extended results on the Adverse Conditions Dataset with Correspondences (ACDC) for semantic driving scene understanding.

Abstract Diffusion magnetic resonance imaging (dMRI) tractography has the unique ability to reconstruct major white matter tracts non‐invasively and is, therefore, widely used in neurosurgical planning and neuroscience. In this work, we reduce two sources of uncertainty within the tractography pipeline. The first one is the model uncertainty that arises in crossing fibre tractography, from having to estimate the number of relevant fibre compartments in each voxel. We propose a mathematical framework to estimate model uncertainty, and we reduce this type of uncertainty with a model averaging approach that combines the fibre direction estimates from all candidate models, weighted by the posterior probability of the respective model. The second source of uncertainty is measurement noise. We use bootstrapping to estimate this data uncertainty, and consolidate the fibre direction estimates from all bootstraps into a consensus model. We observe that, in most voxels, a traditional model selection strategy selects different models across bootstraps. In this sense, the bootstrap consensus also reduces model uncertainty. Either approach significantly increases the accuracy of crossing fibre tractography in multiple subjects, and combining them provides an additional benefit. However, model averaging is much more efficient computationally.

Annotated datasets are essential for supervised learning. However, annotating large datasets is a tedious and time-intensive task. This paper addresses active learning in the context of semantic segmentation with the goal of reducing the human labeling effort. Our application is agricultural robotics and we focus on the task of distinguishing between crop and weed plants from image data. A key challenge in this application is the transfer of an existing semantic segmentation CNN to a new field, in which growth stage, weeds, soil, and weather conditions differ. We propose a novel approach that, given a trained model on one field together with rough foreground segmentation, refines the network on a substantially different field providing an effective method of selecting samples to annotate for supporting the transfer. We evaluated our approach on two challenging datasets from the agricultural robotics domain and show that we achieve a higher accuracy with a smaller number of samples compared to random sampling as well as entropy based sampling, which consequently reduces the required human labeling effort.

DT-MRI streamsurfaces, defined as surfaces that are everywhere tangential to the major and medium eigenvector fields, have been proposed as a tool for visualizing regions of predominantly planar behavior in diffusion tensor MRI. Even though it has long been known that their construction assumes that the involved eigenvector fields satisfy an integrability condition, it has never been tested systematically whether this condition is met in real-world data. We introduce a suitable and efficiently computable test to the visualization literature, demonstrate that it can be used to distinguish integrable from nonintegrable configurations in simulations, and apply it to whole-brain datasets of 15 healthy subjects. We conclude that streamsurface integrability is approximately satisfied in a substantial part of the brain, but not everywhere, including some regions of planarity. As a consequence, algorithms for streamsurface extraction should explicitly test local integrability. Finally, we propose a novel patch-based approch to streamsurface visualization that reduces visual artifacts, and is shown to more fully sample the extent of streamsurfaces.

Diffusion MRI (dMRI) provides rich information on the white matter of the human brain, enabling insight into neurological disease, normal aging, and neuroplasticity. We present BundleMAP, an approach to extracting features from dMRI data that can be used for supervised classification, regression, and hypothesis testing. Our features are based on aggregating measurements along nerve fiber bundles, enabling visualization and anatomical interpretation. The main idea behind BundleMAP is to use the ISOMAP manifold learning technique to jointly parametrize nerve fiber bundles. We combine this idea with mechanisms for outlier removal and feature selection to obtain a practical machine learning pipeline. We demonstrate that it increases accuracy of disease detection and estimation of disease activity, and that it improves the power of statistical tests.

In regression problems over $\mathbb{R}^d$, the unknown function $f$ often varies more in some coordinates than in others. We show that weighting each coordinate $i$ according to an estimate of the variation of $f$ along coordinate $i$ -- e.g. the $L_1$ norm of the $i$th-directional derivative of $f$ -- is an efficient way to significantly improve the performance of distance-based regressors such as kernel and $k$-NN regressors. The approach, termed Gradient Weighting (GW), consists of a first pass regression estimate $f_n$ which serves to evaluate the directional derivatives of $f$, and a second-pass regression estimate on the re-weighted data. The GW approach can be instantiated for both regression and classification, and is grounded in strong theoretical principles having to do with the way regression bias and variance are affected by a generic feature-weighting scheme. These theoretical principles provide further technical foundation for some existing feature-weighting heuristics that have proved successful in practice. We propose a simple estimator of these derivative norms and prove its consistency. The proposed estimator computes efficiently and easily extends to run online. We then derive a classification version of the GW approach which evaluates on real-worlds datasets with as much success as its regression counterpart.

Despite the widely recognized importance of symmetric second order tensor fields in medicine and engineering, the visualization of data uncertainty in tensor fields is still in its infancy. A recently proposed tensorial normal distribution, involving a fourth order covariance tensor, provides a mathematical description of how different aspects of the tensor field, such as trace, anisotropy, or orientation, vary and covary at each point. However, this wealth of information is far too rich for a human analyst to take in at a single glance, and no suitable visualization tools are available. We propose a novel approach that facilitates visual analysis of tensor covariance at multiple levels of detail. We start with a visual abstraction that uses slice views and direct volume rendering to indicate large-scale changes in the covariance structure, and locations with high overall variance. We then provide tools for interactive exploration, making it possible to drill down into different types of variability, such as in shape or orientation. Finally, we allow the analyst to focus on specific locations of the field, and provide tensor glyph animations and overlays that intuitively depict confidence intervals at those points. Our system is demonstrated by investigating the effects of measurement noise on diffusion tensor MRI, and by analyzing two ensembles of stress tensor fields from solid mechanics.

Large image deformations pose a challenging problem for the visualization and statistical analysis of 3D image ensembles which have a multitude of applications in biology and medicine. Simple linear interpolation in the tangent space of the ensemble introduces artifactual anatomical structures that hamper the application of targeted visual shape analysis techniques. In this work we make use of the theory of stationary velocity fields to facilitate interactive non-linear image interpolation and plausible extrapolation for high quality rendering of large deformations and devise an efficient image warping method on the GPU. This does not only improve quality of existing visualization techniques, but opens up a field of novel interactive methods for shape ensemble analysis. Taking advantage of the efficient non-linear 3D image warping, we showcase four visualizations: 1) browsing on-the-fly computed group mean shapes to learn about shape differences between specific classes, 2) interactive reformation to investigate complex morphologies in a single view, 3) likelihood volumes to gain a concise overview of variability and 4) streamline visualization to show variation in detail, specifically uncovering its component tangential to a reference surface. Evaluation on a real world dataset shows that the presented method outperforms the state-of-the-art in terms of visual quality while retaining interactive frame rates. A case study with a domain expert was performed in which the novel analysis and visualization methods are applied on standard model structures, namely skull and mandible of different rodents, to investigate and compare influence of phylogeny, diet and geography on shape. The visualizations enable for instance to distinguish (population-)normal and pathological morphology, assist in uncovering correlation to extrinsic factors and potentially support assessment of model quality.

Three-dimensional symmetric tensor fields have a wide range of applications in solid and fluid mechanics. Recent advances in the (topological) analysis of 3D symmetric tensor fields focus on degenerate tensors which form curves. In this paper, we introduce a number of feature surfaces, such as neutral surfaces and traceless surfaces, into tensor field analysis, based on the notion of eigenvalue manifold. Neutral surfaces are the boundary between linear tensors and planar tensors, and the traceless surfaces are the boundary between tensors of positive traces and those of negative traces. Degenerate curves, neutral surfaces, and traceless surfaces together form a partition of the eigenvalue manifold, which provides a more complete tensor field analysis than degenerate curves alone. We also extract and visualize the isosurfaces of tensor modes, tensor isotropy, and tensor magnitude, which we have found useful for domain applications in fluid and solid mechanics. Extracting neutral and traceless surfaces using the Marching Tetrahedra method can cause the loss of geometric and topological details, which can lead to false physical interpretation. To robustly extract neutral surfaces and traceless surfaces, we develop a polynomial description of them which enables us to borrow techniques from algebraic surface extraction, a topic well-researched by the computer-aided design (CAD) community as well as the algebraic geometry community. In addition, we adapt the surface extraction technique, called A-patches, to improve the speed of finding degenerate curves. Finally, we apply our analysis to data from solid and fluid mechanics as well as scalar field analysis.

Diffusion Tensor Imaging (DTI) is a magnetic resonance imaging modality that enables the in-vivo reconstruction and visualization of fibrous structures. To inspect the local and individual diffusion tensors, glyph-based visualizations are commonly used since they are able to effectively convey full aspects of the diffusion tensor. For several applications it is necessary to compare tensor fields, e.g., to study the effects of acquisition parameters, or to investigate the influence of pathologies on white matter structures. This comparison is commonly done by extracting scalar information out of the tensor fields and then comparing these scalar fields, which leads to a loss of information. If the glyph representation is kept, simple juxtaposition or superposition can be used. However, neither facilitates the identification and interpretation of the differences between the tensor fields. Inspired by the checkerboard style visualization and the superquadric tensor glyph, we design a new glyph to locally visualize differences between two diffusion tensors by combining juxtaposition and explicit encoding. Because tensor scale, anisotropy type, and orientation are related to anatomical information relevant for DTI applications, we focus on visualizing tensor differences in these three aspects. As demonstrated in a user study, our new glyph design allows users to efficiently and effectively identify the tensor differences. We also apply our new glyphs to investigate the differences between DTI datasets of the human brain in two different contexts using different b-values, and to compare datasets from a healthy and HIV-infected subject.

Abstract Complex volume rendering tasks require high‐dimensional transfer functions, which are notoriously difficult to design. One solution to this is to learn transfer functions from scribbles that the user places in the volumetric domain in an intuitive and natural manner. In this paper, we explicitly model and visualize the uncertainty in the resulting classification. To this end, we extend a previous intelligent system approach to volume rendering, and we systematically compare five supervised classification techniques – Gaussian Naive Bayes, k Nearest Neighbor, Support Vector Machines, Neural Networks, and Random Forests – with respect to probabilistic classification, support for multiple materials, interactive performance, robustness to unreliable input, and easy parameter tuning, which we identify as key requirements for the successful use in this application. Based on theoretical considerations, as well as quantitative and visual results on volume datasets from different sources and modalities, we conclude that, while no single classifier can be expected to outperform all others under all circumstances, random forests are a useful off‐the‐shelf technique that provides fast, easy, robust and accurate results in many scenarios.

Gaining insight into anatomic co variation helps the understanding of organismic shape variability in general and is of particular interest for delimiting morphological modules. Generation of hypotheses on structural co variation is undoubtedly a highly creative process, and as such, requires an exploratory approach. In this work we propose a new local anatomic covariance tensor which enables interactive visualizations to explore co variation at different levels of detail, stimulating rapid formation and (qualitative) evaluation of hypotheses. The effectiveness of the presented approach is demonstrated on a μCT dataset of mouse mandibles for which results from the literature are successfully reproduced, while providing a more detailed representation of co variation compared to state-of-the-art methods.

Abstract Obtaining reproducible fiber direction estimates from diffusion MRI is crucial for successful fiber tracking. Modeling and visualizing the probability distribution of the inferred fiber directions is an important step in evaluating and comparing different acquisition schemes and fiber models. However, this distribution is usually strongly dominated by its main direction, which makes it difficult to examine when plotted naively. In this work, we propose a new visualization of the fiber probability distribution. It is based on embedding the probability measure into a particular reproducing kernel Hilbert space. This permits a decomposition into an embedded delta peak, representing the main direction, and a non‐negative residual. They are then combined into a new glyph representation which visually enhances the residual, in order to highlight even subtle differences. Moreover, the magnitude of the delta peak component quantifies precision of the main fiber direction. We demonstrate that our new glyph provides a more detailed impression of the uncertainty than the current standard method, cones that contain 95% of the estimated directions. We use our new method to contribute to the validation of different ways of resampling the data (bootstrapping), and to visualize the differences between alternative acquisition schemes and models for high angular resolution diffusion imaging (HARDI).

Spectral clustering is a powerful and versatile technique, whose broad range of applications includes 3D image analysis. However, its practical use often involves a tedious and time-consuming process of tuning parameters and making application-specific choices. In the absence of training data with labeled clusters, help from a human analyst is required to decide the number of clusters, to determine whether hierarchical clustering is needed, and to define the appropriate distance measures, parameters of the underlying graph, and type of graph Laplacian. We propose to simplify this process via an open-box approach, in which an interactive system visualizes the involved mathematical quantities, suggests parameter values, and provides immediate feedback to support the required decisions. Our framework focuses on applications in 3D image analysis, and links the abstract high-dimensional feature space used in spectral clustering to the three-dimensional data space. This provides a better understanding of the technique, and helps the analyst predict how well specific parameter settings will generalize to similar tasks. In addition, our system supports filtering outliers and labeling the final clusters in such a way that user actions can be recorded and transferred to different data in which the same structures are to be found. Our system supports a wide range of inputs, including triangular meshes, regular grids, and point clouds. We use our system to develop segmentation protocols in chest CT and brain MRI that are then successfully applied to other datasets in an automated manner.

Abstract The topological structure of scalar, vector, and second‐order tensor fields provides an important mathematical basis for data analysis and visualization. In this paper, we extend this framework towards higher‐order tensors. First, we establish formal uniqueness properties for a geometrically constrained tensor decomposition. This allows us to define and visualize topological structures in symmetric tensor fields of orders three and four. We clarify that in 2D, degeneracies occur at isolated points, regardless of tensor order. However, for orders higher than two, they are no longer equivalent to isotropic tensors, and their fractional Poincaré index prevents us from deriving continuous vector fields from the tensor decomposition. Instead, sorting the terms by magnitude leads to a new type of feature, lines along which the resulting vector fields are discontinuous. We propose algorithms to extract these features and present results on higher‐order derivatives and higher‐order structure tensors.

Abstract Glyphs are a fundamental tool in tensor visualization, since they provide an intuitive geometric representation of the full tensor information. The Higher‐Order Maximum Enhancing (HOME) glyph, a generalization of the second‐order tensor ellipsoid, was recently shown to emphasize the orientational information in the tensor through a pointed shape around maxima. This paper states and formally proves several important properties of this novel glyph, presents its first three‐dimensional implementation, and proposes a new coloring scheme that reflects peak direction and sharpness. Application to data from High Angular Resolution Diffusion Imaging (HARDI) shows that the method allows for interactive data exploration and confirms that the HOME glyph conveys fiber spread and crossings more effectively than the conventional polar plot.

Symmetric second-order tensor fields play a central role in scientific and biomedical studies as well as in image analysis and feature-extraction methods. The utility of displaying tensor field samples has driven the development of visualization techniques that encode the tensor shape and orientation into the geometry of a tensor glyph. With some exceptions, these methods work only for positive-definite tensors (i.e. having positive eigenvalues, such as diffusion tensors). We expand the scope of tensor glyphs to all symmetric second-order tensors in two and three dimensions, gracefully and unambiguously depicting any combination of positive and negative eigenvalues. We generalize a previous method of superquadric glyphs for positive-definite tensors by drawing upon a larger portion of the superquadric shape space, supplemented with a coloring that indicates the quadratic form (including eigenvalue sign). We show that encoding arbitrary eigenvalue magnitudes requires design choices that differ fundamentally from those in previous work on traceless tensors that arise in the study of liquid crystals. Our method starts with a design of 2-D tensor glyphs guided by principles of scale-preservation and symmetry, and creates 3-D glyphs that include the 2-D glyphs in their axis-aligned cross-sections. A key ingredient of our method is a novel way of mapping from the shape space of three-dimensional symmetric second-order tensors to the unit square. We apply our new glyphs to stress tensors from mechanics, geometry tensors and Hessians from image analysis, and rate-of-deformation tensors in computational fluid dynamics.

Crease surfaces are two-dimensional manifolds along which a scalar field assumes a local maximum (ridge) or a local minimum (valley) in a constrained space. Unlike isosurfaces, they are able to capture extremal structures in the data. Creases have a long tradition in image processing and computer vision, and have recently become a popular tool for visualization. When extracting crease surfaces, degeneracies of the Hessian (i.e., lines along which two eigenvalues are equal) have so far been ignored. We show that these loci, however, have two important consequences for the topology of crease surfaces: First, creases are bounded not only by a side constraint on eigenvalue sign, but also by Hessian degeneracies. Second, crease surfaces are not, in general, orientable. We describe an efficient algorithm for the extraction of crease surfaces which takes these insights into account and demonstrate that it produces more accurate results than previous approaches. Finally, we show that diffusion tensor magnetic resonance imaging (DT-MRI) stream surfaces, which were previously used for the analysis of planar regions in diffusion tensor MRI data, are mathematically ill-defined. As an example application of our method, creases in a measure of planarity are presented as a viable substitute.

Abstract Fiber tracking is a standard tool to estimate the course of major white matter tracts from diffusion tensor magnetic resonance imaging (DT‐MRI) data. In this work, we aim at supporting the visual analysis of classical streamlines from fiber tracking by integrating context from anatomical data, acquired by aT1‐weighted MRI measurement. To this end, we suggest a novel visualization metaphor, which is based on data‐driven deformation of geometry and has been inspired by a technique for anatomical fiber preparation known as Klingler dissection. We demonstrate that our method conveys the relation between streamlines and surrounding anatomical features more effectively than standard techniques like slice images and direct volume rendering. The method works automatically, but its GPU‐based implementation allows for additional, intuitive interaction.

Diffusion weighted magnetic resonance imaging is a unique tool for non-invasive investigation of major nerve fiber tracts. Since the popular diffusion tensor (DT-MRI) model is limited to voxels with a single fiber direction, a number of high angular resolution techniques have been proposed to provide information about more diverse fiber distributions. Two such approaches are Q-Ball imaging and spherical deconvolution, which produce orientation distribution functions (ODFs) on the sphere. For analysis and visualization, the maxima of these functions have been used as principal directions, even though the results are known to be biased in case of crossing fiber tracts. In this paper, we present a more reliable technique for extracting discrete orientations from continuous ODFs, which is based on decomposing their higher-order tensor representation into an isotropic component, several rank-1 terms, and a small residual. Comparing to ground truth in synthetic data shows that the novel method reduces bias and reliably reconstructs crossing fibers which are not resolved as individual maxima in the ODF. We present results on both Q-Ball and spherical deconvolution data and demonstrate that the estimated directions allow for plausible fiber tracking in a real data set.

While isosurfaces of anisotropy measures for data from diffusion tensor magnetic resonance imaging (DT-MRI) are known to depict major anatomical structures, the anisotropy metric reduces the rich tensor data to a simple scalar field. In this work, we suggest that the part of the data which has been ignored by the metric can be used to segment anisotropy isosurfaces into anatomically meaningful regions. For the implementation, we propose an edge-based watershed method that adapts and extends a method from curvature-based mesh segmentation [MW99]. Finally, we use the segmentation results to enhance visualization of the data.

Topological methods give concise and expressive visual representations of flow fields. The present work suggests a comparable method for the visualization of human brain diffusion MRI data. We explore existing techniques for the topological analysis of generic tensor fields, but find them inappropriate for diffusion MRI data. Thus, we propose a novel approach that considers the asymptotic behavior of a probabilistic fiber tracking method and define analogs of the basic concepts of flow topology, like critical points, basins, and faces, with interpretations in terms of brain anatomy. The resulting features are fuzzy, reflecting the uncertainty inherent in any connectivity estimate from diffusion imaging. We describe an algorithm to extract the new type of features, demonstrate its robustness under noise, and present results for two regions in a diffusion MRI dataset to illustrate that the method allows a meaningful visual analysis of probabilistic fiber tracking results.