Optical coherence tomography angiography (OCTA) is a non-invasive imaging modality that can acquire high-resolution volumes of the retinal vasculature and aid the diagnosis of ocular, neurological and cardiac diseases. Segmenting the visible blood vessels is a common first step when extracting quantitative biomarkers from these images. Classical segmentation algorithms based on thresholding are strongly affected by image artifacts and limited signal-to-noise ratio. The use of modern, deep learning-based segmentation methods has been inhibited by a lack of large datasets with detailed annotations of the blood vessels. To address this issue, recent work has employed transfer learning, where a segmentation network is trained on synthetic OCTA images and is then applied to real data. However, the previously proposed simulations fail to faithfully model the retinal vasculature and do not provide effective domain adaptation. Because of this, current methods are unable to fully segment the retinal vasculature, in particular the smallest capillaries. In this work, we present a lightweight simulation of the retinal vascular network based on space colonization for faster and more realistic OCTA synthesis. We then introduce three contrast adaptation pipelines to decrease the domain gap between real and artificial images. We demonstrate the superior segmentation performance of our approach in extensive quantitative and qualitative experiments on three public datasets that compare our method to traditional computer vision algorithms and supervised training using human annotations. Finally, we make our entire pipeline publicly available, including the source code, pretrained models, and a large dataset of synthetic OCTA images.

Point-based geometry representations have become widely used in numerous contexts, ranging from particle-based simulations, over stereo image matching, to depth sensing via light detection and ranging. Our application focus is on the reconstruction of curved line structures in noisy 3D point cloud data. Respective algorithms operating on such point clouds often rely on the notion of a local neighborhood. Regarding the latter, our approach employs multi-scale neighborhoods, for which weighted covariance measures of local points are determined. Curved line structures are reconstructed via vector field tracing, using a bidirectional piecewise streamline integration. We also introduce an automatic selection of optimal starting points via multi-scale geometric measures. The pipeline development and choice of parameters was driven by an extensive, automated initial analysis process on over a million prototype test cases. The behavior of our approach is controlled by several parameters — the majority being set automatically, leaving only three to be controlled by a user. In an extensive, automated final evaluation, we cover over one hundred thousand parameter sets, including 3D test geometries with varying curvature, sharp corners, intersections, data holes, and systematically applied varying types of noise. Further, we analyzed different choices for the point of reference in the co-variance computation; using a weighted mean performed best in most cases. In addition, we compared our method to current, publicly available line reconstruction frameworks. Up to thirty times faster execution times were achieved in some cases, at comparable error measures. Finally, we also demonstrate an exemplary application on four real-world 3D light detection and ranging datasets, extracting power line cables.

We propose a method for adding small-scale details to surfaces of 3D geometries in the context of interactive deformation computation of elastic objects. This is relevant in real-time applications, for instance, in surgical simulation or interactive animation. The key idea is the procedural generation of surface details via a weighted sum of periodic functions, applied as an on-surface displacement field. We first calculate local deformation strains of a low-resolution 3D input mesh, which are then employed to estimate amplitudes, orientations, and positions of high-resolution details. The shapes and spatial frequencies of the periodic details are obtained from mechanical parameters, assuming the physical model of a film-substrate aggregate. Finally, our approach creates the highly-detailed output mesh fully on the GPU. The performance is independent of the spatial frequency of the inserted details as well as, within certain limits, of the resolution of the output mesh. We can reproduce numerous commonly observed, characteristic surface deformation patterns, such as wrinkles or buckles, allowing for the representation of a wide variety of simulated materials and interaction processes. We highlight the performance of our method with several examples.

Data-driven methods have received increasing attention in recent years in order to meet real-time requirements in computationally intensive tasks. In our current work we examine the application of such approaches in soft-tissue simulation. The core idea is to split deformations into a coarse approximation and a differential part that contains the details. We employ the data-driven stamping approach to enrich a fast simulation surface with details that have been extracted from a set of example deformations obtained in offline computations. In this paper we detail our technique, and suggest further extensions over our previous work. First, we propose an improved method for correlating the current coarse approximation to the examples in the database. The new correlation metric combines Euclidean distances with cosine similarity. It allows for better example discrimination, resulting in a well-conditioned linear system. This also enables us to use a non-negative least squares solver that leads to a better regression and guarantees positive stamp blending weights. Second, we suggest a frequency-space stamp compression scheme that saves memory and, in most instances, is faster, since many operations can be done in the compressed space. Third, cutting is included by employing a physically-inspired influence map that allows for proper handling of material discontinuities that were not present in the original examples. We thoroughly evaluate our method and demonstrate its practical application in a surgical simulator prototype.

Interaction with virtual deformable models is common in several haptic contexts, such as in medical training simulators. This paper presents a methodological procedure for the creation of such virtual models from their real-life counterparts. Both the surface geometry and the elastic parametrization of an object are reconstructed from position/force readings during an operator-assisted exploration of the object. A 3D mesh model is then generated from the surface contact points. The internal elastic modulus is found using the 3D finite element method. This modeling method is compared with two common 1D elastic models, namely Kelvin-Voigt and Hunt-Crossley. Results using three deformable homogeneous silicone samples show successful geometry reconstruction. 1D model parameterizations exhibit high variation dependent on geometry and contact location. In contrast, elastic modulus reconstruction yields a global model parameterization independent of geometry. Elastic moduli estimated in experiments correlated with their known values, and were shown to be reproducible among samples with different geometries.

In this paper we suggest two extensions to our previous work in data-driven haptic rendering. First, we propose a two-staged process to handle inhomogeneous material behavior. In an initial broad phase, representative locations of similar force responses are identified, based on a pattern clustering algorithm. Thereafter, separate detail-rich radial basis function approximations are generated for each detected region. These are employed in the rendering to provide varying feedback depending on the contact location. Secondly, we suggest a new computational formulation for obtaining the radial basis function reconstructions of the reaction force signal. Inspired by Compressive Sensing, we employ an ℓ1-minimization with a random selection strategy. The improved performance of both extensions is illustrated on different test cases.

Passive haptics, also known as tactile augmentation, denotes the use of a physical counterpart to a virtual environment to provide tactile feedback. Employing passive haptics can result in more realistic touch sensations than those from active force feedback, especially for rigid contacts. However, changes in the virtual environment would necessitate modifications of the physical counterparts. In recent work space warping has been proposed as one solution to overcome this limitation. In this technique virtual space is distorted such that a variety of virtual models can be mapped onto one single physical object. In this paper, we propose as an extension adaptive space warping; we show how this technique can be employed in a mixed-reality surgical training simulator in order to map different virtual patients onto one physical anatomical model. We developed methods to warp different organ geometries onto one physical mock-up, to handle different mechanical behaviors of the virtual patients, and to allow interactive modifications of the virtual structures, while the physical counterparts remain unchanged. Various practical examples underline the wide applicability of our approach. To the best of our knowledge this is the first practical usage of such a technique in the specific context of interactive medical training.

ABSTRACT We propose a novel mesh optimization approach that is useful for speeding up a simulation of finite elements‐based deformable objects. The approach is based on a quality metric derived from the critical simulation time step of explicit time integration schemes (i.e., the stability limit for the integration of dynamic equations). Our mesh smoothing approach consists of a set of small and independent spring systems. These are made up of a reference mesh node connected to a set of fixed endpoints, which represent the positions that maximize the time step of the elements adjacent to that node. The reference node is displaced to an equilibrium position through a few local iterations. Each spring's stiffness is weighted depending on the quality of its corresponding element. All spring systems can be computed in parallel. Global iterations update the mesh and spring systems. In addition, we combine our smoothing algorithm with topological transformations. With this approach, the simulation performance could be increased by more than 30% depending on the mesh. This approach is suitable for the generation of finite element method meshes, particularly those requiring interactive applications and haptic rendering. Copyright © 2012 John Wiley & Sons, Ltd.

We present a novel hybrid method to allow large time steps in explicit integrations for the simulation of deformable objects. In explicit integration schemes, the time step is typically limited by the size and the shape of the discretization elements as well as by the material parameters. We propose a two-step strategy to enable large time steps for meshes with elements potentially destabilizing the integration. First, the necessary time step for a stable computation is identified per element using modal analysis. This allows determining which elements have to be handled specially given a desired simulation time step. The identified critical elements are treated by a geometric deformation model, while the remaining ones are simulated with a standard deformation model (in our case, a corotational linear Finite Element Method). In order to achieve a valid deformation behavior, we propose a strategy to determine appropriate parameters for the geometric model. Our hybrid method allows taking much larger time steps than using an explicit Finite Element Method alone. The total computational costs per second are significantly lowered. The proposed scheme is especially useful for simulations requiring interactive mesh updates, such as for instance cutting in surgical simulations.

Treating the interactions of soft tissue with rigid user-guided tools is a difficult problem. This is particularly true if the soft tissue has a slender shape, i.e., resembling a thin shell, and if the underlying numerical time-integration scheme employs large time steps. In this case, large mutual displacements of both the tool and the soft tissue occur frequently, resulting in deep interpenetrations or breakthroughs. As a consequence, the computation of spatially and temporally coherent contact spaces turns out to be very challenging. In this paper, an approach is proposed that is tailored to these kinds of interactions. To solve this problem, a novel spatially reduced representation of the soft tissue geometry is employed where the dominant dimensions of the object are approximated by a 2D triangle surface, while the third dimension is given in terms of nodal radii. To construct a feasible, nonpenetrating configuration, a novel manifold projection scheme is presented where the colliding triangles are rasterized into a distance field in order to robustly estimate the contact spaces, even for large intersections. The method produces physically plausible results, albeit it is purely geometric, and the material parameters are neglected at the collision response stage. Various examples, including an interactive prototype arthroscopy simulator, underline the wide applicability of the approach.

Haptic augmented reality merges real and virtual feedback, allowing a user to touch the real environment augmented with virtual haptic stimuli. A key functionality of this technology is altering the haptic properties of real objects by means of virtual haptic feedback. Previously, we have developed a haptic augmentation system in which object stiffness was modulated. In this paper, we extend our framework to cover further haptic properties: friction and weight. Simple but effective algorithms for estimating and altering these properties have been developed. The first approach allows us to change the inherent friction between a tool tip and a surface to a desired one identified in an offline process. The second technique enables a user to perceive an altered weight when lifting an object at two interaction points. The performance of the proposed algorithms has been evaluated on real objects. The errors remain within reasonable bounds and compare well to the relevant perceptual Weber fractions. Limitations of our technique are identified, and possible extensions are proposed.

In our current work we explore the feasibility of using visuo-haptic augmented reality techniques for training and remote collaboration. Accurate calibration of system components is a key factor in this context. In this paper we deal with the calibration of the kinematic parameters of the haptic device. We compare a previously developed technique using an external optical tracker to a new approach not relying on additional devices. Simulated and real experiments are carried out to investigate the performance of the new technique. As will be shown, both methods yield reasonable results with average errors in the workspace below 2mm. It is found that the trackerless technique is an acceptable alternative to using a tracker for calibration, with only slightly reduced accuracy. It is straightforward to apply and allows to improve rendering fidelity.

Haptic augmented reality (AR) allows to modulate the haptic properties of a real object by providing virtual haptic feedback. We previously developed a haptic AR system wherein the stiffness of a real object can be augmented with the aid of a haptic interface. To demonstrate its potential, this paper presents a case study for medical training of breast cancer palpation. A real breast model made of soft silicone is augmented with a virtual tumor rendered inside. Haptic stimuli for the virtual tumor are generated based on a contact dynamics model identified via real measurements, without the need of geometric information on the breast. A subjective evaluation confirmed the realism and fidelity of our palpation system.

In haptic augmented reality (AR), a stimulus from a real object is augmented by a synthetic haptic signal. We previously developed a haptic AR system wherein the stiffness of a real object can be augmented with the aid of a haptic interface. This demonstration presents a case study of this technology for medical training of breast cancer palpation. A real breast model made of soft silicone is augmented with a virtual tumor rendered inside. Haptic stimuli for the virtual tumor are generated based on a contact dynamics model identified via real measurements. With our palpation system, a user can experience excellent realism comparable to that of a real breast mock-up containing a real tumor.

Abstract Elastic rods are thin flexible objects typically undergoing large non‐linear deformations that cannot be modeled with linear methods. They are used in a number of research fields, e.g., to represent hair or ropes in animations, or catheters or needles in medical simulations. In this paper, we propose a deformation model for inextensible elastic rods. The method of Lagrange multipliers is employed to enforce the inextensibility of the rod, and to couple the material frames with the centerline. The resulting system is banded, allowing for an efficient linear time solution. We also propose a manifold projection method to incorporate the non‐penetration constraints resulting from contact handling into our constrained Lagrangian mechanics (CLM) problem. We further augment the contact model by treating torsional friction. This allows to reproduce friction effects such as dynamic rolling and twisting of rods. Various examples underline the benefits and applicability of our model. Copyright © 2010 John Wiley & Sons, Ltd.

In this article, the data-driven haptic rendering approach presented in our earlier work is assessed. The approach relies on recordings from real objects from which a data-driven model is derived that captures the haptic properties of the object. We conducted two studies. In the first study, the Just Noticeable Difference (JND) for small forces, as encountered in our set-up, was determined. JNDs were obtained both for active and passive user interaction. A conservative threshold curve was derived that was then used to guide the model generation in the second study. The second study examined the achievable rendering fidelity for two objects with different stiffnesses. Subjects directly compared data-driven virtual feedback with the real objects. Results indicated that it is crucial to include dynamic material effects to achieve haptic feedback that cannot be distinguished from real objects. Results also showed that the fidelity is considerably decreased for stiffer objects due to limits of the display hardware.

We present a setup for simple and fast experimental estimation of drag coefficients. Our system can accurately determine the parameters of fluid-object interaction for complicated geometry and boundary conditions in the realm of classical Stokes equations. Obtained results are compared with theoretical and numerical solutions. Good agreement with those references is achieved in both cases. An advantage of our method is the prompt and easy parameter retrieval that still maintains appropriate accuracy. Comparable detailed numerical estimations run on the order of hours or days.

Visual delays are unavoidable in augmented reality setups and occur in different steps of the rendering pipeline. In the context of haptic interaction with virtual objects, it has been shown that delayed force feedback can alter the perception of object stiffness. We hypothesize that delays in augmented reality systems can have similar consequences. To test this, we carried out a user study to investigate the effect of visual and haptic delays on the perception of stiffness. The experiment has been performed in an optimized visuo-haptic augmented reality setup, which allows to artificially manipulate delays during visual and haptic rendering. In line with previous results, delays for haptic feedback resulted in decreased perceived stiffness. In contrast, visual delays caused an increase in perceived stiffness. However, the simultaneous occurrence of delays in both sensory channels led to a partial compensation of these effects. This could potentially help to correct stiffness perception of virtual objects in visuo-haptic augmented reality systems.

In our current research we examine the application of visuo-haptic augmented reality setups in medical training. To this end, highly accurate calibration, system stability, and low latency are indispensable prerequisites. These are necessary to maintain user immersion and avoid breaks in presence which potentially diminish the training outcome. In this paper we describe the developed calibration methods for visuo-haptic integration, the hybrid tracking technique for stable alignment of the augmentation, and the distributed framework ensuring low latency and component synchronization. Finally, we outline an early prototype system based on the multimodal augmented reality framework. The latter allows colocated visuo-haptic interaction with real and virtual scene components in a simplified open surgery setting.

Mass spring models are frequently used to simulate deformable objects because of their conceptual simplicity and computational speed. Unfortunately, the model parameters are not related to elastic material constitutive laws in an obvious way. Several methods to set optimal parameters have been proposed, but so far only with limited success. We analyze the parameter identification problem and show the difficulties, which have prevented previous work from reaching wide usage. Our main contribution is a new method to derive analytical expressions for the spring parameters from an isotropic linear elastic reference model. The method is described and expressions for several mesh topologies are derived. These include triangle, rectangle and tetrahedron meshes. The formulae are validated by comparing the static deformation of the MSM with reference deformations simulated with the finite element method.

The driving force of our research is the precise combination of real and - possibly indistinguishable - virtual objects in an interactive augmented reality environment. This requires real-time, multimodal simulation, as well as stable and accurate overlay of the computer-generated objects. This paper describes several methods to improve accuracy and stability of our hybrid augmented reality system. In a comparison of two approaches to hybrid head pose refinement, we show that Quasi-Newton method enables high performance optimization for image space error minimization. Moreover, a 3D landmark refinement step is proposed, which significantly improves quality and robustness of the overlay process. The enhanced system is demonstrated in an interactive AR environment, which provides accurate haptic feedback from real and virtual deformable objects. Finally, the effect of landmark occlusion on tracking stability during user interaction is also analyzed.

A central training objective of virtual reality based surgical simulation is the removal of pathologic tissue. This necessitates stable, real-time updates of the underlying mesh representation. Within the framework of a hysteroscopy simulator, we have developed a hybrid cutting approach for tetrahedral meshes. It combines the topological update by subdivision with adjustments of the existing topology. Moreover, the mechanical and the visual model are decoupled, thus allowing different resolutions for the underlying mesh representations. With our method, we can closely approximate an arbitrary, user-defined cut surface while avoiding the creation of small or badly shaped elements, thus strongly reducing stability problems in the subsequent deformation computation. The presented approach has been integrated into a virtual reality training system for hysteroscopic interventions. The performance of the algorithm is demonstrated by examples of intra-uterine tumor ablations.

An integral part of every augmented reality system is the calibration between camera and camera-mounted tracking markers. Accuracy and robustness of the AR overlay process is greatly influenced by the quality of this step. In order to meet the very high precision requirements of medical skill training applications, we have set up a calibration environment based on direct sensing of LED markers. A simulation framework has been developed to predict and study the achievable accuracy of the backprojection needed for the scene augmentation process. We demonstrate that the simulation is in good agreement with experimental results. Even if a slight improvement of the precision has been observed compared to well-known hand-eye calibration methods, the subpixel accuracy required by our application cannot be achieved even when using commercial tracking systems providing marker positions within very low error limits.

No abstract available.

Medical image acquisition devices are becoming increasingly multidimensional, and predictions assume 5000 images per patient study within the next decade. Therefore, new paradigms for computerized medical image analysis and visualization are of fundamental importance to make possibly full use of the information buried in the enormous flood of image data. The aim of the presented project is the optimal cooperation between computer-based image analysis algorithms and human operators using new closed-loop segmentation systems for improved information flow. This paper describes an enhanced, visuo-haptic interaction tool we have developed for medical segmentation. Evaluation studies with the system, which confirm the value of adding haptic feedback, are also presented.

We present a new virtual reality-based interaction metaphor for semi-automatic segmentation of medical 3D volume data. The mouse-based, manual initialization of deformable surfaces in 3D represents a major bottleneck in interactive segmentation. In our multi-modal system we enhance this process with additional sensory feedback. A 3D haptic device is used to extract the centerline of a tubular structure. Based on the obtained path a cylinder with varying diameter is generated which in turn is used as the initial guess for a deformable surface.

Abstract We present a new virtual reality‐based interaction metaphor for semi‐automatic segmentation of medical 3D volume data. The mouse‐based, manual initialization of deformable surfaces in 3D represents a major bottleneck in interactive segmentation. In our multi‐modal system we enhance this process with additional sensory feedback. A 3D haptic device is used to extract the centreline of a tubular structure. Based on the obtained path a cylinder with varying diameter is generated, which in turn is used as the initial guess for a deformable surface. Copyright © 2002 John Wiley & Sons, Ltd.