In contemporary radiotherapy planning (RTP), a key module leaf sequencing is predominantly addressed by optimization-based approaches. In this paper, we propose a novel deep reinforcement learning (DRL) model termed as Reinforced Leaf Sequencer (RLS) in a multi-agent framework for leaf sequencing. The RLS model offers improvements to time-consuming iterative optimization steps via large-scale training and can control movement patterns through the design of reward mechanisms. We have conducted experiments on four datasets with four metrics and compared our model with a leading optimization sequencer. Our findings reveal that the proposed RLS model can achieve reduced fluence reconstruction errors, and potential faster convergence when integrated in an optimization planner. Additionally, RLS has shown promising results in a full artificial intelligence RTP pipeline. We hope this pioneer multi-agent RL leaf sequencer can foster future research on machine learning for RTP.

Deep learning has been utilized in knowledge-based radiotherapy planning in which a system trained with a set of clinically approved plans is employed to infer a three-dimensional dose map for a given new patient. However, previous deep methods are primarily limited to simple scenarios, e.g., a fixed planning type or a consistent beam angle configuration. This in fact limits the usability of such approaches and makes them not generalizable over a larger set of clinical scenarios. Herein, we propose a novel conditional generative model, Flexible-Cm GAN, utilizing additional information regarding planning types and various beam geometries. A miss-consistency loss is proposed to deal with the challenge of having a limited set of conditions on the input data, e.g., incomplete training samples. To address the challenges of including clinical preferences, we derive a differentiable shift-dose-volume loss to incorporate the well-known dose-volume histogram constraints. During inference, users can flexibly choose a specific planning type and a set of beam angles to meet the clinical requirements. We conduct experiments on an illustrative face dataset to show the motivation of Flexible-Cm GAN and further validate our model's potential clinical values with two radiotherapy datasets. The results demonstrate the superior performance of the proposed method in a practical heterogeneous radiotherapy planning application compared to existing deep learning-based approaches.

3-D image registration, which involves aligning two or more images, is a critical step in a variety of medical applications from diagnosis to therapy. Image registration is commonly performed by optimizing an image matching metric as a cost function. However this task is challenging due to the non-convex nature of the matching metric over the plausible registration parameter space and insufficient approches for a robust optimization. As a result, current approaches are often customized to a specific problem and sensitive to image quality and artifacts. In this paper, we propose a completely different approach to image registration, inspired by how experts perform the task. We first cast the image registration problem as a "strategic learning" process, where the goal is to find the best sequence of motion actions (e.g. up, down, etc) that yields image alignment. Within this approach, an artificial agent is learned, modeled using deep convolutional neural networks, with 3D raw image data as the input, and the next optimal action as the output. To copy with the dimensionality of the problem, we propose a greedy supervised approach for an end-to-end training, coupled with attention-driven hierarchical strategy. The resulting registration approach inherently encodes both a data-driven matching metric and an optimal registration strategy (policy). We demonstrate on two 3-D/3-D medical image registration examples with drastically different nature of challenges, that the artificial agent outperforms several state-of-the-art registration methods by a large margin in terms of both accuracy and robustness.

Radiofrequency ablation (RFA) is an established treatment for liver cancer when resection is not possible. Yet, its optimal delivery is challenged by the presence of large blood vessels and the time-varying thermal conductivity of biological tissue. Incomplete treatment and an increased risk of recurrence are therefore common. A tool that would enable the accurate planning of RFA is hence necessary. This manuscript describes a new method to compute the extent of ablation required based on the Lattice Boltzmann Method (LBM) and patient-specific, pre-operative images. A detailed anatomical model of the liver is obtained from volumetric images. Then a computational model of heat diffusion, cellular necrosis, and blood flow through the vessels and liver is employed to compute the extent of ablated tissue given the probe location, ablation duration and biological parameters. The model was verified against an analytical solution, showing good fidelity. We also evaluated the predictive power of the proposed framework on ten patients who underwent RFA, for whom pre- and post-operative images were available. Comparisons between the computed ablation extent and ground truth, as observed in postoperative images, were promising (DICE index: 42%, sensitivity: 67%, positive predictive value: 38%). The importance of considering liver perfusion while simulating electrical-heating ablation was also highlighted. Implemented on graphics processing units (GPU), our method simulates 1 minute of ablation in 1.14 minutes, allowing near real-time computation.

This paper presents an approach to predict the deformation of the lungs and surrounding organs during respiration. The framework incorporates a computational model of the respiratory system, which comprises an anatomical model extracted from computed tomography (CT) images at end-expiration (EE), and a biomechanical model of the respiratory physiology, including the material behavior and interactions between organs. A personalization step is performed to automatically estimate patient-specific thoracic pressure, which drives the biomechanical model. The zone-wise pressure values are obtained by using a trust-region optimizer, where the estimated motion is compared to CT images at end-inspiration (EI). A detailed convergence analysis in terms of mesh resolution, time stepping and number of pressure zones on the surface of the thoracic cavity is carried out. The method is then tested on five public datasets. Results show that the model is able to predict the respiratory motion with an average landmark error of 3.40 ±1.0 mm over the entire respiratory cycle. The estimated 3-D lung motion may constitute as an advanced 3-D surrogate for more accurate medical image reconstruction and patient respiratory analysis.

We present a preconditioning scheme for improving the efficiency of optimization of arbitrary difference measures in deformable registration problems. This is of particular interest for high-dimensional registration problems with statistical difference measures such as MI, and the demons method, since in these cases the range of applicable optimization methods is limited. The proposed scheme is simple and computationally efficient: It performs an approximate normalization of the point-wise vectors of the difference gradient to unit length. The major contribution of this work is a theoretical analysis which demonstrates the improvement of the condition by our approach, which is furthermore shown to be an approximation to the optimal case for the analyzed model. Our scheme improves the convergence speed while adding only negligible computational cost, thus resulting in shorter effective runtimes. The theoretical findings are confirmed by experiments on 3D brain data.

We revisit the classic Horn and Schunck optical flow method, with focus on its interpretation as a Gauss-Newton (GN) minimisation of a non-linear energy. This is in contrast to the traditional derivation by linearisation of the brightness constancy assumption. The proposed interpretation provides a much simpler derivation and a better theoretical understanding of the method, and allows for its variations, by casting it in the least-squares optimisation framework. An important resulting implication is that ‐ in contrast to popular belief ‐ the incremental version of Horn and Schunck actually minimises a non-linear energy. We emphasise this finding by demonstrating the equivalence of incremental Horn and Schunck to several methods committed to minimising the nonlinearised energy. Furthermore, we analyse the effect of GN for motion estimation in comparison to methods based on gradient descent. Several examples demonstrate the practical applications of the proposed interpretation. We specify the class of difference measures with sparse Jacobians of the error term as the one which can be efficiently treated in the Horn and Schunck framework. For extension to arbitrary difference measures, we propose a modification based on the analysis of GN for motion estimation, and the concept of preconditioning. We further discuss a modification resulting in decoupled linear systems and the use of compositional updates.

We apply the concept of natural gradients to deformable registration. The motivation stems from the lack of physical interpretation for gradients of image-based difference measures. The main idea is to endow the space of deformations with a distance metric which reflects the variation of the difference measure between two deformations. This is in contrast to standard approaches which assume the Euclidean frame. The modification of the distance metric is realized by treating the deformations as a Riemannian manifold. In our case, the manifold is induced by the Riemannian metric tensor based on the approximation of the Fisher Information matrix, which takes into account the information about the chosen difference measure and the input images. Thus, the resulting natural gradient defined on this manifold inherently takes into account this information. The practical advantages of the proposed approach are the improvement of registration error and faster convergence for low-gradient regions. The proposed scheme is applicable to arbitrary difference measures and can be readily integrated into standard variational deformable registration methods with practically no computational overhead.

Nonlinear registration is mostly performed after initialization by a global, linear transformation (in this work, we focus on similarity transformations), computed by a linear registration method. For the further processing of the results, it is mostly assumed that this preregistration step completely removes the respective linear transformation. However, we show that in deformable settings, this is not the case. As a consequence, a significant linear component is still existent in the deformation computed by the nonlinear registration algorithm. For construction of statistical shape models (SSM) from deformations, this is an unwanted property: SSMs should not contain similarity transformations, since these do not capture information about shape. We propose a method which performs an a posteriori extraction of a similarity transformation from a given nonlinear deformation field, and we use the processed fields as input for SSM construction. For computation of minimal displacements, a closed-form solution minimizing the squared Euclidean norm of the displacement field subject to similarity parameters is used. Experiments on real inter-subject data and on a synthetic example show that the theoretically justified removal of the similarity component by the proposed method has a large influence on the shape model and significantly improves the results.

We propose a novel numerical approach for solving the free-form deformable registration problem. The central idea is to utilize the well understood techniques from variational deformable registration problems. We demonstrate that it is possible to formulate the free-form deformable registration problem as the optimization of an energy functional as in the dense deformation case. This energy functional possesses image distance and regularization terms, which are both functions of the free-form deformation control points. We then setup a semi-backward (implicit) partial differential equation that optimizes the established energy functional. In addition to being mathematically justified, this approach provides both accuracy and speed. Our evaluation on synthetic, real, two dimensional, and three dimensional data demonstrates accuracy and computational effectiveness.

Precise localization of moving targets is essential to increase local control of the cancer via dose escalation while reducing the severity of normal tissue complication. Localization of targets in real time with radio-opaque marker is less favorable considering the excess radiation dose to the patient and potential complications of implantation. Various external surrogates could provide indications of the targets' positions during the breathing process. However, there is a great deal of uncertainty in the correlation between external surrogates and internal target positions/trajectory during respiratory cycles. In order to address this problem, we have developed an algorithm that automatically establishes correspondences between the fluoroscopic sequence frames taken from the patient on the day of treatment and the various phases of a 4DCT planning data set. Image based mapping/synchronization procedure is performed using an underlying Markov model established for the breathing process. The mapping procedure is formulated as an optimization process and is solved efficiently using a dynamic programming technique. Results on the phantom, synthetic, and real patient data demonstrate the effectiveness of the proposed method in coping with respiratory correlation variations. The approach could primarily be used for automatic gating interval adaptation in the gated radiotherapy.

Latency is a key property of video see through AR systems since users' performance is strongly related to it. However, there is no standard way of latency measurement of an AR system in the literature. We have created a stable and comparable way of estimating the latency in a video see through AR system. The latency is estimated by encoding the time in the image and decoding the time after camera feedback. We have encoded the time as a translation of a circle in the image. The cross ratio has been used as an image feature that is preserved in a projective transformation. The encoding allows for a simple but accurate way of decoding. We show that this way of encoding has an adequate accuracy for latency measurements. As the method allows for a series of automatic measurements we propose to visualize the measurements in a histogram. This histogram reveals meaningful information about the system other than the mean value and standard deviation of the latency. The method has been tested on four different AR systems that use different camera technology, resolution and frame rates.

We report on a new single PC based stereoscopic video-see-through AR system which we developed for medical applications. Recent advances in graphics hardware, memory bandwidth, and computing power of standard PCs made it possible that this system outperforms an earlier version which included 3 networked SGI workstations. We designed and implemented a new AR software platform. It is component based and - in conjunction with XML configuration files - provides efficiency, modularity, and extensibility for fast and robust prototyping of AR applications. The system has a compelling real-time performance with 30 frames/second, displaying stereoscopic augmented video views with XGA resolution.

We have built a system for augmented reality visualization based on a single head mounted tracking camera. The camera includes an infrared illuminator and works in conjunction with a set of retro-reflective markers that are placed around the workspace. This marker frame configuration delivers excellent pose information, which translates to stable, jitter-free augmentation. In this article, we describe using the same single camera system for tracking relatively small marker clusters, which can be used for tool or instrument tracking. Tracking of such a marker cluster is more susceptible to noise compared to tracking of a marker frame, mainly due to its small image coverage. The sensitivity to noise is studied using Monte Carlo simulations and verified in an experimental setup. We achieved jitter-free augmentation with an optimized cluster design.

Unmanned vehicles are employed more and more frequently for a range of scientific and commercial undersea applications. However, the critical dependency on a tether link, mainly for the transmission of live images to the surface for command and control, is a significant technological obstacle limiting vehicle maneuverability. The elimination of the tether requires the capability to compress a massive amount of live video data to meet the bandwidth limitations of acoustic telemetry. This paper addresses motion-compensated compression of underwater video imagery and the extraction of the sought-after transformations by the application of the brightness constancy assumption and a generalized dynamic image model for the analysis of time-varying imagery. Two approaches, suitable for automatic vision-based navigation or operator-assisted missions of unmanned submersible vehicles, are considered. In the former case, the 3D motion and position information extracted from the raw data by the vision-based navigation system is used directly to perform the compression. In the latter case, the compression is achieved using the motion and radiometric information extracted from the live and reconstructed images at the surface station. To evaluate the performance of proposed methods, results from experiments with synthetic and real data are presented and compared to compression methods that are based on the brightness constancy model.

A vision system has been developed based on the application of a 3D direct motion estimation algorithm to facilitate autonomous or operated-assisted missions of AUVS and ROVs near the ocean floor. The main capabilities of interest are vision-based vehicle positioning, navigation and trajectory following, as well as mosaicking of sea bed images. The system performance in enabling these functions has been demonstrated on a three-thruster surface vehicle that operates in a 6'/spl times/12'/spl times/6' water tank, with the bottom surface set up to simulate a sea floor environment. Images from a down-look camera installed on the vehicle are digitized and processed on a Windows NT Dual Pentium-200 to estimate the vehicle's motion, which is transmitted via the serial link to the vehicle control system running on a 386 processor. This information is employed to maintain or move to a desired position, follow a specified trajectory, and to construct in (near) real-time a composite image of the scene surface.

We examined the relationship between stress wave velocities and the lung density. The discrete cubic spline wavelet was used to calculate time delay and frequency characteristics of the wave that is propagated on the surface of 3 canine lungs. The analysis verified previous results ascertaining the linear elastic model. As the vascular volume, extra vascular lung water and lung density increased, the wave velocities were decreased; that is the wave velocities were inversely proportional to the square root of the lung density. The analysis verified the exact time-frequency relationship for the dominant stress wave velocity (i.e., shear or Rayleigh-type surface wave velocity) and showed other transit times besides the previously observed short and long ones are not valid.