Fourier Neural Operators (FNOs) have proven to be an efficient and effective method for resolution-independent operator learning in a broad variety of application areas across scientific machine learning. A key reason for their success is their ability to accurately model long-range dependencies in spatio-temporal data by learning global convolutions in a computationally efficient manner. To this end, FNOs rely on the discrete Fourier transform (DFT), however, DFTs cause visual and spectral artifacts as well as pronounced dissipation when learning operators in spherical coordinates by incorrectly assuming flat geometry. To overcome this limitation, we generalize FNOs on the sphere, introducing Spherical FNOs (SFNOs) for learning operators on spherical geometries. We apply SFNOs to forecasting atmo- spheric dynamics, and demonstrate stable autoregressive rollouts for a year of simulated time (1,460 steps), while retaining physically plausible dynamics. The SFNO has important implications for machine learning-based simulation of climate dynamics that could eventually help accelerate our response to climate change.

We present a framework for real-time 3D reconstruction of non-rigidly moving surfaces captured with a single RGB-D camera. Based on the variational level set method, it warps a given truncated signed distance field (TSDF) to a target TSDF via gradient flow without explicit correspondence search. We optimize an energy that contains a data term which steers towards voxel-wise alignment. To ensure geometrically consistent reconstructions, we develop and compare different strategies, namely an approximately Killing vector field regularizer, gradient flow in Sobolev space and newly devised accelerated optimization. The underlying TSDF evolution makes our approach capable of capturing rapid motions, topological changes and interacting agents, but entails loss of data association. To recover correspondences, we propose to utilize the lowest-frequency Laplacian eigenfunctions of the TSDFs, which encode inherent deformation patterns. For moderate motions we are able to obtain implicit associations via a term that imposes voxel-wise eigenfunction alignment. This is not sufficient for larger motions, so we explicitly estimate voxel correspondences via signature matching of lower-dimensional eigenfunction embeddings. We carry out qualitative and quantitative evaluation of our geometric reconstruction fidelity and voxel correspondence accuracy, demonstrating advantages over related techniques in handling topological changes and fast motions.

Breast cancer is the most common invasive cancer in women, affecting more than 10% of women worldwide. Microscopic analysis of a biopsy remains one of the most important methods to diagnose the type of breast cancer. This requires specialized analysis by pathologists, in a task that i) is highly time- and cost-consuming and ii) often leads to nonconsensual results. The relevance and potential of automatic classification algorithms using hematoxylin-eosin stained histopathological images has already been demonstrated, but the reported results are still sub-optimal for clinical use. With the goal of advancing the state-of-the-art in automatic classification, the Grand Challenge on BreAst Cancer Histology images (BACH) was organized in conjunction with the 15th International Conference on Image Analysis and Recognition (ICIAR 2018). BACH aimed at the classification and localization of clinically relevant histopathological classes in microscopy and whole-slide images from a large annotated dataset, specifically compiled and made publicly available for the challenge. Following a positive response from the scientific community, a total of 64 submissions, out of 677 registrations, effectively entered the competition. The submitted algorithms improved the state-of-the-art in automatic classification of breast cancer with microscopy images to an accuracy of 87%. Convolutional neuronal networks were the most successful methodology in the BACH challenge. Detailed analysis of the collective results allowed the identification of remaining challenges in the field and recommendations for future developments. The BACH dataset remains publicly available as to promote further improvements to the field of automatic classification in digital pathology.

Three-dimensional freehand imaging techniques are gaining wider adoption due to their ?exibility and cost ef?ciency. Typical examples for such a combination of a tracking system with an imaging device are freehand SPECT or freehand 3D ultrasound. However, the quality of the resulting image data is heavily dependent on the skill of the human operator and on the level of noise of the tracking data. The latter aspect can introduce blur or strong artifacts, which can signi?cantly hamper the interpretation of image data. Unfortunately, the most commonly used tracking systems to date, i.e. optical and electromagnetic, present a trade-off between invading the surgeon's workspace (due to line-of-sight requirements) and higher levels of noise and sensitivity due to the interference of surrounding metallic objects. In this work, we propose a novel approach for total variation regularization of data from tracking systems (which we term pose signals) based on a variational formulation in the manifold of Euclidean transformations. The performance of the proposed approach was evaluated using synthetic data as well as real ultrasound sweeps executed on both a Lego phantom and human anatomy, showing signi?cant improvement in terms of tracking data quality and compounded ultrasound images. Source code can be found at https://github.com/IFL-CAMP/pose_regularization.

We present a system that builds 3D models of non-rigidly moving surfaces from scratch in real time using a single RGB-D stream. Our solution is based on the variational level set method, thus it copes with arbitrary geometry, including topological changes. It warps a given truncated signed distance field (TSDF) to a target TSDF via gradient flow. Unlike previous approaches that define the gradient using an L2inner product, our method relies on gradient flow in Sobolev space. Its favourable regularity properties allow for a more straightforward energy formulation that is faster to compute and that achieves higher geometric detail, mitigating the over-smoothing effects introduced by other regularization schemes. In addition, the coarse-to-fine evolution behaviour of the flow is able to handle larger motions, making few frames sufficient for a high-fidelity reconstruction. Last but not least, our pipeline determines voxel correspondences between partial shapes by matching signatures in a low-dimensional embedding of their Laplacian eigenfunctions, and is thus able to reliably colour the output model. A variety of quantitative and qualitative evaluations demonstrate the advantages of our technique.

We introduce a geometry-driven approach for real-time 3D reconstruction of deforming surfaces from a single RGB-D stream without any templates or shape priors. To this end, we tackle the problem of non-rigid registration by level set evolution without explicit correspondence search. Given a pair of signed distance fields (SDFs) representing the shapes of interest, we estimate a dense deformation field that aligns them. It is defined as a displacement vector field of the same resolution as the SDFs and is determined iteratively via variational minimization. To ensure it generates plausible shapes, we propose a novel regularizer that imposes local rigidity by requiring the deformation to be a smooth and approximately Killing vector field, i.e. generating nearly isometric motions. Moreover, we enforce that the level set property of unity gradient magnitude is preserved over iterations. As a result, KillingFusion reliably reconstructs objects that are undergoing topological changes and fast inter-frame motion. In addition to incrementally building a model from scratch, our system can also deform complete surfaces. We demonstrate these capabilities on several public datasets and introduce our own sequences that permit both qualitative and quantitative comparison to related approaches.

Many prediction tasks contain uncertainty. In some cases, uncertainty is inherent in the task itself. In future prediction, for example, many distinct outcomes are equally valid. In other cases, uncertainty arises from the way data is labeled. For example, in object detection, many objects of interest often go unlabeled, and in human pose estimation, occluded joints are often labeled with ambiguous values. In this work we focus on a principled approach for handling such scenarios. In particular, we propose a frame-work for reformulating existing single-prediction models as multiple hypothesis prediction (MHP) models and an associated meta loss and optimization procedure to train them. To demonstrate our approach, we consider four diverse applications: human pose estimation, future prediction, image classification and segmentation. We find that MHP models outperform their single-hypothesis counterparts in all cases, and that MHP models simultaneously expose valuable insights into the variability of predictions.

Registration of vascular structures is crucial for preoperative planning, intraoperative navigation, and follow-up assessment. Typical applications include, but are not limited to, Trans-catheter Aortic Valve Implantation and monitoring of tumor vasculature or aneurysm growth. In order to achieve the aforementioned goals, a large number of various registration algorithms has been developed. With this review paper we provide a comprehensive overview over the plethora of existing techniques with a particular focus on the suitable classification criteria such as the involved modalities of the employed optimization methods. However, we wish to go beyond a static literature review which is naturally doomed to be outdated after a certain period of time due to the research progress. We augment this review paper with an extendable and interactive database in order to obtain a living review whose currency goes beyond the one of a printed paper. All papers in this database are labeled with one or multiple tags according to 13 carefully defined categories. The classification of all entries can then be visualized as one or multiple trees which are presented via a web-based interactive app (http://livingreview.in.tum.de) allowing the user to choose a unique perspective for literature review. In addition, the user can search the underlying database for specific tags or publications related to vessel registration. Many applications of this framework are conceivable, including the use for getting a general overview on the topic or the utilization by physicians for deciding about the best-suited algorithm for a specific application.

This paper presents a method for the simultaneous segmentation and regularization of a series of shapes from a corresponding sequence of images. Such series arise as time series of 2D images when considering video data, or as stacks of 2D images obtained by slicewise tomographic reconstruction. We first derive a model where the regularization of the shape signal is achieved by a total variation prior on the shape manifold. The method employs a modified Kendall shape space to facilitate explicit computations together with the concept of Sobolev gradients. For the proposed model, we derive an efficient and computationally accessible splitting scheme. Using a generalized forward-backward approach, our algorithm treats the total variation atoms of the splitting via proximal mappings, whereas the data terms are dealt with by gradient descent. The potential of the proposed method is demonstrated on various application examples dealing with 3D data. We explain how to extend the proposed combined approach to shape fields which, for instance, arise in the context of 3D+t imaging modalities, and show an application in this setup as well.

In this paper, we consider combined TV denoising and diffusion tensor fitting in DTI using the affine-invariant Riemannian metric on the space of diffusion tensors. Instead of first fitting the diffusion tensors, and then denoising them, we define a suitable TV type energy functional which incorporates the measured DWIs (using an inverse problem setup) and which measures the nearness of neighboring tensors in the manifold. To approach this functional, we propose generalized forward- backward splitting algorithms which combine an explicit and several implicit steps performed on a decomposition of the functional. We validate the performance of the derived algorithms on synthetic and real DTI data. In particular, we work on real 3D data. To our knowledge, the present paper describes the first approach to TV regularization in a combined manifold and inverse problem setup.

Staining and scanning of tissue samples for microscopic examination is fraught with undesirable color variations arising from differences in raw materials and manufacturing techniques of stain vendors, staining protocols of labs, and color responses of digital scanners. When comparing tissue samples, color normalization and stain separation of the tissue images can be helpful for both pathologists and software. Techniques that are used for natural images fail to utilize structural properties of stained tissue samples and produce undesirable color distortions. The stain concentration cannot be negative. Tissue samples are stained with only a few stains and most tissue regions are characterized by at most one effective stain. We model these physical phenomena that define the tissue structure by first decomposing images in an unsupervised manner into stain density maps that are sparse and non-negative. For a given image, we combine its stain density maps with stain color basis of a pathologist-preferred target image, thus altering only its color while preserving its structure described by the maps. Stain density correlation with ground truth and preference by pathologists were higher for images normalized using our method when compared to other alternatives. We also propose a computationally faster extension of this technique for large whole-slide images that selects an appropriate patch sample instead of using the entire image to compute the stain color basis.

Cryo-Electron Tomography is a leading imaging technique in structural biology, which is capable of acquiring two-dimensional projections of cellular structures at high resolution and close-to-native state. Due to the limited electron dose the resulting projections exhibit extremely low SNR and contrast. The 3D structure is then reconstructed and passed through a number of post-processing steps including de-noising and sub-tomogram averaging to provide a better understanding and interpretation. As CET is mainly used for imaging fine scale structures, any denoising method applied to CET images should be scale selective and in particular be able to preserve such fine scale structures. In this context, we propose a new denoising framework based on regularized graph spectral filtering with a full control of scale-space and global consistency. Using the gold-standard metrics, we show that our denoising algorithm significantly outperforms the state-of-the-art methods such as NAD, NLM and RGF in terms of noise removal and structure preservation.

This paper introduces the concept of shape signals, i.e., series of shapes which have a natural temporal or spatial ordering, as well as a variational formulation for the regularization of these signals. The proposed formulation can be seen as the shape-valued generalization of the Rudin-Osher-Fatemi (ROF) functional for intensity images. We derive a variant of the classical finite-dimensional representation of Kendall, but our framework is generic in the sense that it can be combined with any shape space. This representation allows for the explicit computation of geodesics and thus facilitates the efficient numerical treatment of the variational formulation by means of the cyclic proximal point algorithm. Similar to the ROF-functional, we demonstrate experimentally that ℓ1-type penalties both for data fidelity term and regularizer perform best in regularizing shape signals. Finally, we show applications of our method to shape signals obtained from synthetic, photometric, and medical data sets.

We propose a novel, physics-based method for detecting multi-scale tubular features in ultrasound images. The detector is based on a Hessian-matrix eigenvalue method, but unlike previous work, our detector is guided by an optimal model of vessel-like structures with respect to the ultrasound-image formation process. Our method provides a voxel-wise probability map, along with estimates of the radii and orientations of the detected tubes. These results can then be used for further processing, including segmentation and enhanced volume visualization. Most Hessian-based algorithms, including the well-known Frangi filter, were developed for CTA or MRA; they implicitly assume symmetry about the vessel centerline. This is not consistent with ultrasound data. We overcome this limitation by introducing a novel filter that allows multi-scale estimation both with respect to the vessel's centerline and with respect to the vessel's border. We use manually-segmented ultrasound imagery from 35 patients to show that our method is superior to standard Hessian-based methods. We evaluate the performance of the proposed methods based on the sensitivity and specificity like measures, and finally demonstrate further applicability of our method to vascular ultrasound images of the carotid artery, as well as ultrasound data for abdominal aortic aneurysms.

With the need for adequate analysis of blood flow dynamics, different maging modalities have been developed to measure varying blood velocities over time. Due to its numerous advantages, Doppler ultrasound sonography remains one of the most widely used techniques in clinical routine, but requires additional preprocessing to recover 3D velocity information. Despite great progress in the last years, recent approaches do not jointly consider spatial and temporal variation in blood flow. In this work, we present a novel gating- and compounding-free method to simultaneously reconstruct a 3D velocity field and a temporal flow profile from arbitrarily sampled Doppler ultrasound measurements obtained from multiple directions. Based on a laminar flow assumption, a patch-wise B-spline formulation of blood velocity is coupled for the first time with a global waveform model acting as temporal regularization. We evaluated our method on three virtual phantom datasets, demonstrating robustness in terms of noise, angle between measurements and data sparsity, and applied it successfully to five real case datasets of carotid artery examination.

Direct volume visualization techniques offer powerful insight into volumetric medical images and are part of the clinical routine for many applications. Up to now, however, their use is mostly limited to tomographic imaging modalities such as CT or MRI. With very few exceptions, such as fetal ultrasound, classic volume rendering using one-dimensional intensity-based transfer functions fails to yield satisfying results in case of ultrasound volumes. This is particularly due its gradient-like nature, a high amount of noise and speckle, and the fact that individual tissue types are rather characterized by a similar texture than by similar intensity values. Therefore, clinicians still prefer to look at 2D slices extracted from the ultrasound volume. In this work, we present an entirely novel approach to the classification and compositing stage of the volume rendering pipeline, specifically designed for use with ultrasonic images. We introduce point predicates as a generic formulation for integrating the evaluation of not only low-level information like local intensity or gradient, but also of high-level information, such as non-local image features or even anatomical models. Thus, we can successfully filter clinically relevant from non-relevant information. In order to effectively reduce the potentially high dimensionality of the predicate configuration space, we propose the predicate histogram as an intuitive user interface. This is augmented by a scribble technique to provide a comfortable metaphor for selecting predicates of interest. Assigning importance factors to the predicates allows for focus-and-context visualization that ensures to always show important (focus) regions of the data while maintaining as much context information as possible. Our method naturally integrates into standard ray casting algorithms and yields superior results in comparison to traditional methods in terms of visualizing a specific target anatomy in ultrasound volumes.

Polar object representations have proven to be a powerful shape model for many medical as well as other computer vision applications, such as interactive image segmentation or tracking. Inspired by recent work on Sobolev active contours we derive a Sobolev-type function space for polar curves. This so-called polar space is endowed with a metric that allows us to favor origin translations and scale changes over smooth deformations of the curve. Moreover, the resulting curve flow inherits the coarse-to-fine behavior of Sobolev active contours and is thus very robust to local minima. These properties make the resulting polar active contours a powerful segmentation tool for many medical applications, such as cross-sectional vessel segmentation, aneurysm analysis, or cell tracking.

We present a preconditioning scheme for improving the efficiency of optimization of arbitrary difference measures in deformable registration problems. This is of particular interest for high-dimensional registration problems with statistical difference measures such as MI, and the demons method, since in these cases the range of applicable optimization methods is limited. The proposed scheme is simple and computationally efficient: It performs an approximate normalization of the point-wise vectors of the difference gradient to unit length. The major contribution of this work is a theoretical analysis which demonstrates the improvement of the condition by our approach, which is furthermore shown to be an approximation to the optimal case for the analyzed model. Our scheme improves the convergence speed while adding only negligible computational cost, thus resulting in shorter effective runtimes. The theoretical findings are confirmed by experiments on 3D brain data.

Variational level set methods are formulated as energy minimisation problems, which are often solved by gradient-based optimisation methods, such as gradient descent. Unfortunately, the gradient obtained by applying the calculus of variations is not suitable, because it is only an element of the function space L 2 making it prone to lead into wrong local minima. Consequently, some regularisation strategy - be it the restriction to signed distance functions or the choice of smooth function spaces - is necessary. In this paper we propose diffusion-based regularisation strategies and compare them to the recently proposed ones of Charpiat et al. and Sundaramoorthi et al. From this comparison we derive two general regularisation paradigms at the level of update equations and show that the diffusion-based paradigm enjoys both theoretical and practical advantages, such as an improved convergence rate, while being of the same computational complexity as the other paradigm.