One of the major challenges in the use of opaque, complex AI models is the need or desire to provide an explanation to the end-user (and other stakeholders) as to how the system arrived at the answer it did. While there is significant research in the development of explainability techniques for AI, the question remains as to who needs an explanation, what an explanation consists of, and how to communicate this to a lay user who lacks direct expertise in the area. In this position paper, an interdisciplinary team of researchers argue that the example of clinical communications offers lessons to those interested in improving the transparency and interpretability of AI systems. We identify five lessons from clinical communications: (1) offering explanations for AI systems and disclosure of their use recognizes the dignity of those using and impacted by it; (2) AI explanations can be productively targeted rather than totally comprehensive; (3) AI explanations can be enforced through codified rules but also norms, guided by core values; (4) what constitutes a “good” AI explanation will require repeated updating due to changes in technology and social expectations; 5) AI explanations will have impacts beyond defining any one AI system, shaping and being shaped by broader perceptions of AI. We review the history, debates and consequences surrounding the institutionalization of one type of clinical communication, informed consent, in order to illustrate the challenges and opportunities that may await attempts to offer explanations of opaque AI models. We highlight takeaways and implications for computer scientists and policymakers in the context of growing concerns and moves toward AI governance.

The empirical performance of neural ordinary differential equations (NODEs) is significantly inferior to discrete-layer models on benchmark tasks (e.g. image classification). We demonstrate an explanation is the inaccuracy of existing gradient estimation methods: the adjoint method has numerical errors in reverse-mode integration; the naive method suffers from a redundantly deep computation graph. We propose the Adaptive Checkpoint Adjoint (ACA) method: ACA applies a trajectory checkpoint strategy which records the forward- mode trajectory as the reverse-mode trajectory to guarantee accuracy; ACA deletes redundant components for shallow computation graphs; and ACA supports adaptive solvers. On image classification tasks, compared with the adjoint and naive method, ACA achieves half the error rate in half the training time; NODE trained with ACA outperforms ResNet in both accuracy and test-retest reliability. On time-series modeling, ACA outperforms competing methods. Furthermore, NODE with ACA can incorporate physical knowledge to achieve better accuracy.

Most popular optimizers for deep learning can be broadly categorized as adaptive methods (e.g.~Adam) and accelerated schemes (e.g.~stochastic gradient descent (SGD) with momentum). For many models such as convolutional neural networks (CNNs), adaptive methods typically converge faster but generalize worse compared to SGD; for complex settings such as generative adversarial networks (GANs), adaptive methods are typically the default because of their stability. We propose AdaBelief to simultaneously achieve three goals: fast convergence as in adaptive methods, good generalization as in SGD, and training stability. The intuition for AdaBelief is to adapt the stepsize according to the "belief" in the current gradient direction. Viewing the exponential moving average (EMA) of the noisy gradient as the prediction of the gradient at the next time step, if the observed gradient greatly deviates from the prediction, we distrust the current observation and take a small step; if the observed gradient is close to the prediction, we trust it and take a large step. We validate AdaBelief in extensive experiments, showing that it outperforms other methods with fast convergence and high accuracy on image classification and language modeling. Specifically, on ImageNet, AdaBelief achieves comparable accuracy to SGD. Furthermore, in the training of a GAN on Cifar10, AdaBelief demonstrates high stability and improves the quality of generated samples compared to a well-tuned Adam optimizer.

The accurate segmentation of the brain surface in post-surgical computed tomography (CT) images is critical for image-guided neurosurgical procedures in epilepsy patients. Following surgical implantation of intracranial electrodes, surgeons require accurate registration of the post-implantation CT images to the pre-implantation functional and structural magnetic resonance imaging to guide surgical resection of epileptic tissue. One way to perform the registration is via surface matching. The key challenge in this setup is the CT segmentation, where the extraction of the cortical surface is difficult due to the missing parts of the skull and artifacts introduced from the electrodes. In this paper, we present a dictionary learning-based method to segment the brain surface in post-surgical CT images of epilepsy patients following surgical implantation of electrodes. We propose learning a model of locally oriented appearance that captures both the normal tissue and the artifacts found along this brain surface boundary. Utilizing a database of clinical epilepsy imaging data to train and test our approach, we demonstrate that our method using locally oriented image appearance both more accurately extracts the brain surface and better localizes electrodes on the post-operative brain surface compared to standard, non-oriented appearance modeling. In addition, we compare our method to a standard atlas-based segmentation approach and to a U-Net-based deep convolutional neural network segmentation method.

Respiratory motion during positron emission tomography (PET)/computed tomography (CT) imaging can cause significant image blurring and underestimation of tracer concentration for both static and dynamic studies. In this paper, with the aim to eliminate both intra-cycle and inter-cycle motions, and apply to dynamic imaging, we developed a non-rigid event-by-event (NR-EBE) respiratory motion-compensated list-mode reconstruction algorithm. The proposed method consists of two components: the first component estimates a continuous non-rigid motion field of the internal organs using the internal-external motion correlation. This continuous motion field is then incorporated into the second component, non-rigid MOLAR (NR-MOLAR) reconstruction algorithm to deform the system matrix to the reference location where the attenuation CT is acquired. The point spread function (PSF) and time-of-flight (TOF) kernels in NR-MOLAR are incorporated in the system matrix calculation, and therefore are also deformed according to motion. We first validated NR-MOLAR using a XCAT phantom with a simulated respiratory motion. NR-EBE motion-compensated image reconstruction using both the components was then validated on three human studies injected with18F-FPDTBZ and one with18F-fluorodeoxyglucose (FDG) tracers. The human results were compared with conventional non-rigid motion correction using discrete motion field (NR-discrete, one motion field per gate) and a previously proposed rigid EBE motion-compensated image reconstruction (R-EBE) that was designed to correct for rigid motion on a target lesion/organ. The XCAT results demonstrated that NR-MOLAR incorporating both PSF and TOF kernels effectively corrected for non-rigid motion. The18F-FPDTBZ studies showed that NR-EBE out-performed NR-Discrete, and yielded comparable results with R-EBE on target organs while yielding superior image quality in other regions. The FDG study showed that NR-EBE clearly improved the visibility of multiple moving lesions in the liver where some of them could not be discerned in other reconstructions, in addition to improving quantification. These results show that NR-EBE motion-compensated image reconstruction appears to be a promising tool for lesion detection and quantification when imaging thoracic and abdominal regions using PET.

Vasko et al., Comput Oper Res 29 (2002), 441–458 defined the cable‐trench problem (CTP) as a combination of the Shortest Path and Minimum Spanning Tree Problems. Specifically, let be a connected weighted graph with specified vertex (referred to as the root), length for each , and positive parameters and . The CTP is the problem of finding a spanning tree of such that is minimized, where is the total length of the spanning tree and is the total path length in from to all other vertices of . Recently, Jiang et al., Proceedings of MICCAI 6893 (2011), 528–536 modeled the vascular network connectivity problem in medical image analysis as an extraordinarily large‐scale application of the generalized cable‐trench problem (GCTP). They proposed an efficient solution based on a modification of Prim's algorithm (MOD_PRIM), but did not elaborate on it. In this article, we formally define the GCTP, describe MOD_PRIM in detail, and describe two linearly parallelizable metaheuristics which significantly improve the performance of MOD_PRIM. These metaheuristics are capable of finding near‐optimal solutions of very large GCTPs in quadratic time in . We also give empirical results for graphs with up to 25,001 vertices. © 2015 Wiley Periodicals, Inc. NETWORKS, Vol. 67(3), 199–208 2016

Kernel methods are powerful tools for nonlinear feature representation. Incorporated with structured LASSO, the kernelized structured LASSO is an effective feature selection approach that can preserve the nonlinear input-output relationships as well as the structured sparseness. But as the data dimension increases, the method can quickly become computationally prohibitive. In this paper we propose a stochastic optimization algorithm that can efficiently address this computational problem on account of the redundant kernel representations of the given data. Experiments on simulation data and PET 3D brain image data show that our method can achieve superior accuracy with less computational cost than existing methods.

In the above-named document [ibid, vol. 33, no. 6, pp. 1275–1289, Jun. 2014], the funding information should have appeared as follows: "This work was supported in part by the National Institutes of Health (The National Heart, Lung, and Blood Institute) through these awards: R01HL082640, 5T32HL098069, and R01HL121226. The work of B. A. Lin was supported by an ASE Foundation Career Development Award."

Accurate and robust image registration is a fundamental task in medical image analysis applications, and requires non-rigid transformations with a large number of degrees of freedom. Statistical deformation models (SDMs) attempt to learn the distribution of non-rigid deformations, and can be used both to reduce the transformation dimensionality and to constrain the registration process. However, high-dimensional SDMs are difficult to train given orders of magnitude fewer training samples. In this paper, we utilize both a small set of annotated imaging data and a large set of unlabeled data to effectively learn an SDM of non-rigid transformations in a semi-supervised training (SST) framework. We demonstrate results applying this framework towards inter-subject registration of skull-stripped, magnetic resonance (MR) brain images. Our approach makes use of 39 labeled MR datasets to create a set of supervised registrations, which we augment with a set of over 1200 unsupervised registrations using unlabeled MRIs. Through leave-one-out cross validation, we show that SST of a non-rigid SDM results in a robust registration algorithm with significantly improved accuracy compared to standard, intensity-based registration, and does so with a 99% reduction in transformation dimensionality.

Quantitative analysis of left ventricular deformation can provide valuable information about the extent of disease as well as the efficacy of treatment. In this work, we develop an adaptive multi-level compactly supported radial basis approach for deformation analysis in 3D+time echocardiography. Our method combines displacement information from shape tracking of myocardial boundaries (derived from B-mode data) with mid-wall displacements from radio-frequency-based ultrasound speckle tracking. We evaluate our methods on open-chest canines (N=8) and show that our combined approach is better correlated to magnetic resonance tagging-derived strains than either individual method. We also are able to identify regions of myocardial infarction (confirmed by postmortem analysis) using radial strain values obtained with our approach.

During neurosurgery, nonrigid brain deformation may affect the reliability of tissue localization based on preoperative images. To provide accurate surgical guidance in these cases, preoperative images must be updated to reflect the intraoperative brain. This can be accomplished by warping these preoperative images using a biomechanical model. Due to the possible complexity of this deformation, intraoperative information is often required to guide the model solution. In this paper, a linear elastic model of the brain is developed to infer volumetric brain deformation associated with measured intraoperative cortical surface displacement. The developed model relies on known material properties of brain tissue, and does not require further knowledge about intraoperative conditions. To provide an initial estimation of volumetric model accuracy, as well as determine the model's sensitivity to the specified material parameters and surface displacements, a realistic brain phantom was developed. Phantom results indicate that the linear elastic model significantly reduced localization error due to brain shift, from > 16 mm to under 5 mm, on average. In addition, though in vivo quantitative validation is necessary, preliminary application of this approach to images acquired during neocortical epilepsy cases confirms the feasibility of applying the developed model to in vivo data.

Fiber tracking in diffusion tensor magnetic resonance images (DTIs) reveals 3-D structural connectivity of the brain conveniently and thus is a viable tool for investigating neural differences. Unfortunately, local noise, image artifacts and numerical tracking errors during integration-based techniques are cumulative. Prematurely terminated fibers and under-sampled fiber bundles result in incomplete reconstruction of white matter fiber tracts and hence incorrect anatomical measurements. Quantitative cross-subject tract analysis, which is critical for abnormality detection, is complicated by inefficient and inaccurate tract reconstruction and normalization from fiber bundles. Because of the above problems, we propose a parcellation method that aims for lower sensitivity to initialization and local orientation error by directly segmenting full white matter tracts (Fasciculography), rather than reconstructing individual curves, from diffusion tensor fields. A fast, robust volumetric, and intrinsically normalized solution is achieved by noise-filtering using a generic parametrized tract model to prevent premature tract termination. At the same time, orientation information reduces the search space, significantly speeding up the tract parcellation process with less human intervention. Detailed comparisons against streamline tracking, shortest-path tracking, and nonrigid registration using synthetic and real DTIs confirmed the superior properties of Fasciculography. Since a normalized tract can be delineated interactively in a just few seconds using the proposed method, accurate high volume tract comparisons become feasible.

During neurosurgery, nonrigid brain deformation prevents preoperatively-acquired images from accurately depicting the intraoperative brain. Stereo vision systems can be used to track intraoperative cortical surface deformation and update preoperative brain images in conjunction with a biomechanical model. However, these stereo systems are often plagued with calibration error, which can corrupt the deformation estimation. In order to decouple the effects of camera calibration from the surface deformation estimation, a framework that can solve for disparate and often competing variables is needed. Game theory, which was developed to handle decision making in this type of competitive environment, has been applied to various fields from economics to biology. In this paper, game theory is applied to cortical surface tracking during neocortical epilepsy surgery and used to infer information about the physical processes of brain surface deformation and image acquisition. The method is successfully applied to eight in vivo cases, resulting in an 81% decrease in mean surface displacement error. This includes a case in which some of the initial camera calibration parameters had errors of 70%. Additionally, the advantages of using a game theoretic approach in neocortical epilepsy surgery are clearly demonstrated in its robustness to initial conditions.

Statistical models have shown considerable promise as a basis for segmenting and interpreting cardiac images. While a variety of statistical models have been proposed to improve the segmentation results, most of them are either static models (SMs), which neglect the temporal dynamics of a cardiac sequence, or generic dynamical models (GDMs), which are homogeneous in time and neglect the intersubject variability in cardiac shape and deformation. In this paper, we develop a subject-specific dynamical model (SSDM) that simultaneously handles temporal dynamics (intrasubject variability) and intersubject variability. We also propose a dynamic prediction algorithm that can progressively identify the specific motion patterns of a new cardiac sequence based on the shapes observed in past frames. The incorporation of this SSDM into the segmentation framework is formulated in a recursive Bayesian framework. It starts with a manual segmentation of the first frame, and then segments each frame according to intensity information from the current frame as well as the prediction from past frames. In addition, to reduce error propagation in sequential segmentation, we take into account the periodic nature of cardiac motion and perform segmentation in both forward and backward directions. We perform "leave-one-out" test on 32 canine sequences and 22 human sequences, and compare the experimental results with those from SM, GDM, and active appearance motion model (AAMM). Quantitative analysis of the experimental results shows that SSDM outperforms SM, GDM, and AAMM by having better global and local consistencies with manual segmentation. Moreover, we compare the segmentation results from forward and forward-backward segmentation. Quantitative evaluation shows that forward-backward segmentation suppresses the propagation of segmentation errors.

Shape registration is often involved in computing statistical differences between groups of shapes, which is a key aspect of morphometric study. The results of shape difference are found to be sensitive to registration, i.e., different registration methods lead to varied results. This raises the question of how to improve the reliability of registration procedures. This paper proposes a perturbation scheme, which perturbs registrations by feeding them with different resampled shape groups, and then aggregates the resulting shape differences. Experiments are conducted using three typical registration algorithms on both synthetic and biomedical shapes, where more reliable inter-group shape differences are found under the proposed scheme.

Statistical model-based segmentation of the left ventricle from cardiac images has received considerable attention in recent years. While a variety of statistical models have been shown to improve segmentation results, most of them are either static models (SM) which neglect the temporal coherence of a cardiac sequence or generic dynamical models (GDM) which neglect the inter-subject variability of cardiac shapes and deformations. In this paper, we use a subject-specific dynamical model (SSDM) that handles inter-subject variability and temporal dynamics (intra-subject variability) simultaneously. It can progressively identify the specific motion patterns of a new cardiac sequence based on the segmentations observed in the past frames. We formulate the integration of the SSDM into the segmentation process in a recursive Bayesian framework in order to segment each frame based on the intensity information from the current frame and the prediction from the past frames. We perform "Leave-one-out" test on 32 sequences to validate our approach. Quantitative analysis of experimental results shows that the segmentation with the SSDM outperforms those with the SM and GDM by having better global and local consistencies with the manual segmentation.

The effective visualization of vascular structures is critical for diagnosis, surgical planning as well as treatment evaluation. In recent work, we have developed an algorithm for vessel detection that examines the intensity profile around each voxel in an angiographic image and determines the likelihood that any given voxel belongs to a vessel; we term this the "vesselness coefficient" of the voxel. Our results show that our algorithm works particularly well for visualizing branch points in vessels. Compared to standard Hessian based techniques, which are fine-tuned to identify long cylindrical structures, our technique identifies branches and connections with other vessels. Using our computed vesselness coefficient, we explore a set of techniques for visualizing vasculature. Visualizing vessels is particularly challenging because not only is their position in space important for clinicians but it is also important to be able to resolve their spatial relationship. We applied visualization techniques that provide shape cues as well as depth cues to allow the viewer to differentiate between vessels that are closer from those that are farther. We use our computed vesselness coefficient to effectively visualize vasculature in both clinical neurovascular x-ray computed tomography based angiography images, as well as images from three different animal studies. We conducted a formal user evaluation of our visualization techniques with the help of radiologists, surgeons, and other expert users. Results indicate that experts preferred distance color blending and tone shading for conveying depth over standard visualization techniques.

Neurosurgical planning and image guided neurosurgery require the visualization of multimodal data obtained from various functional and structural image modalities, such as Magnetic Resonance Imaging (MRI), Computed Tomography (CT), functional MRI, Single photon emission computed tomography (SPECT) and so on. In the case of epilepsy neurosurgery for example, these images are used to identify brain regions to guide intracranial electrode implantation and resection. Generally, such data is visualized using 2D slices and in some cases using a 3D volume rendering along with the functional imaging results. Visualizing the activation region effectively by still preserving sufficient surrounding brain regions for context is exceedingly important to neurologists and surgeons. We present novel interaction techniques for visualization of multimodal data to facilitate improved exploration and planning for neurosurgery. We extended the line widget from VTK to allow surgeons to control the shape of the region of the brain that they can visually crop away during exploration and surgery. We allow simple spherical, cubical, ellipsoidal and cylindrical (probe aligned cuts) for exploration purposes. In addition we integrate the cropping tool with the image-guided navigation system used for epilepsy neurosurgery. We are currently investigating the use of these new tools in surgical planning and based on further feedback from our neurosurgeons we will integrate them into the setup used for image-guided neurosurgery.

This paper presents a new shape registration algorithm that establishes "meaningful correspondence " between objects, in that it preserves the local shape correspondence between the source and target objects. By observing that an object's skeleton corresponds to its local shape peaks, we use skeleton to characterize the local shape of the source and target objects. Unlike traditional graph-based skeleton matching algorithms that focus on matching skeletons alone and ignore the overall alignment of the boundaries, our algorithm is formulated in a variational framework which aligns local shape by registering two potential fields that are associated with skeletons. Also, we add a boundary constraint term to the energy functional, such that our algorithm can be applied to match bulky objects where skeleton and boundary are far away to each other. To increase the robustness of our algorithm, we incorporate M-estimator and dynamic pruning algorithm to form a feedback system that eliminates local shape outliers caused by nonrigid deformation, occlusion, and missing parts. Experiments on 2D binary shapes and 3D cardiac sequences validate the accuracy and robustness of this algorithm.

During neurosurgery, nonrigid brain deformation prevents preoperatively acquired images from accurately depicting the intraoperative brain. Stereo vision systems can be used to track cortical surface deformation and update preoperative brain images in conjunction with a biomechanical model. However, these stereo systems are often plagued with calibration error, which can corrupt the deformation estimation. In order to decouple the effects of camera calibration and surface deformation, a framework is needed which can solve for disparate and often competing variables. Game theory, which was developed specifically to handle decision making in this type of competitive environment, has been applied to various fields from economics to biology. In this paper, we apply game theory to cortical surface tracking and use it to infer information about the physical processes of brain deformation and image acquisition.

The quantitative estimation of regional cardiac deformation from three-dimensional (3-D) image sequences has important clinical implications for the assessment of viability in the heart wall. We present here a generic methodology for estimating soft tissue deformation which integrates image-derived information with biomechanical models, and apply it to the problem of cardiac deformation estimation. The method is image modality independent. The images are segmented interactively and then initial correspondence is established using a shape-tracking approach. A dense motion field is then estimated using a transversely isotropic, linear-elastic model, which accounts for the muscle fiber directions in the left ventricle. The dense motion field is in turn used to calculate the deformation of the heart wall in terms of strain in cardiac specific directions. The strains obtained using this approach in open-chest dogs before and after coronary occlusion, exhibit a high correlation with strains produced in the same animals using implanted markers. Further, they show good agreement with previously published results in the literature. This proposed method provides quantitative regional 3-D estimates of heart deformation.

We present a new method for the estimation of optical flow which uses a dynamic programming based algorithm to simultaneously detect the presence of motion boundaries and to estimate optical flow. This allows for a more accurate estimation of the motion field near discontinuities. The results compare favorably with those produced by other methods.