Aortic stenosis (AS), a prevalent and serious heart valve disorder, requires early detection but remains difficult to diagnose in routine practice. Although echocardiography with Doppler imaging is the clinical standard, these assessments are typically limited to trained specialists. Point-of-care ultrasound (POCUS) offers an accessible alternative for AS screening but is restricted to basic 2D B-mode imaging, often lacking the analysis Doppler provides. Our project introduces MultiASNet, a multimodal machine learning framework designed to enhance AS screening with POCUS by combining 2D B-mode videos with structured data from echocardiography reports, including Doppler parameters. Using contrastive learning, MultiASNet aligns video features with report features in tabular form from the same patient to improve interpretive quality. To address misalignment where a single report corresponds to multiple video views, some irrelevant to AS diagnosis, we use cross-attention in a transformer-based video and tabular network to assign less importance to irrelevant report data. The model integrates structured data only during training, enabling independent use with B-mode videos during inference for broader accessibility. MultiASNet also incorporates sample selection to counteract label noise from observer variability, yielding improved accuracy on two datasets. We achieved balanced accuracy scores of 93.0% on a private dataset and 83.9% on the public TMED-2 dataset for AS detection. For severity classification, balanced accuracy scores were 80.4% and 59.4% on the private and public datasets, respectively. This model facilitates reliable AS screening in non-specialist settings, bridging the gap left by Doppler data while reducing noise-related errors. Our code is publicly available at github.com/DeepRCL/MultiASNet.

Subpopulation shift, characterized by a disparity in subpopulation distribution between the training and target datasets, can significantly degrade the performance of machine learning models. Current solutions to subpopulation shift involve modifying empirical risk minimization with re-weighting strategies to improve generalization. This strategy relies on assumptions about the number and nature of subpopulations and annotations on group membership, which are unavailable for many real-world datasets. Instead, we propose using an ensemble of diverse classifiers to adaptively capture risk associated with subpopulations. Given a feature extractor network, we replace its standard linear classification layer with a mixture of prototypical classifiers, where each member is trained to classify the data while focusing on different features and samples from other members. In empirical evaluation on nine real-world datasets, covering diverse domains and kinds of subpopulation shift, our method of Diverse Prototypical Ensembles (DPEs) often outperforms the prior state-of-the-art in worst-group accuracy. The code is available at https://github.com/minhto2802/dpe4subpop.

Aortic stenosis (AS) is a prevalent heart valve disease that requires accurate and timely diagnosis for effective treatment. Current methods for automated AS severity classification rely on black-box deep learning techniques, which suffer from a low level of trustworthiness and hinder clinical adoption. To tackle this challenge, we propose ProtoASNet, a prototype-based neural network designed to classify the severity of AS from B-mode echocardiography videos. ProtoASNet bases its predictions exclusively on the similarity scores between the input and a set of learned spatio-temporal prototypes, ensuring inherent interpretability. Users can directly visualize the similarity between the input and each prototype, as well as the weighted sum of similarities. This approach provides clinically relevant evidence for each prediction, as the prototypes typically highlight markers such as calcification and restricted movement of aortic valve leaflets. Moreover, ProtoASNet utilizes abstention loss to estimate aleatoric uncertainty by defining a set of prototypes that capture ambiguity and insufficient information in the observed data. This feature augments prototype-based models with the ability to explain when they may fail. We evaluate ProtoASNet on a private dataset and the publicly available TMED-2 dataset. It surpasses existing state-of-the-art methods, achieving a balanced accuracy of 80.0% on our private dataset and 79.7% on the TMED-2 dataset, respectively. By discarding cases flagged as uncertain, ProtoASNet achieves an improved balanced accuracy of 82.4% on our private dataset. Furthermore, by offering interpretability and an uncertainty measure for each prediction, ProtoASNet improves transparency and facilitates the interactive usage of deep networks in aiding clinical decision-making. Our source code is available at: https://github.com/hooman007/ProtoASNet.

Aortic stenosis (AS) is characterized by restricted motion and calcification of the aortic valve and is the deadliest valvular cardiac disease. Assessment of AS severity is typically done by expert cardiologists using Doppler measurements of valvular flow from echocardiography. However, this limits the assessment of AS to hospitals staffed with experts to provide comprehensive echocardiography service. As accurate Doppler acquisition requires significant clinical training, in this paper, we present a deep learning framework to determine the feasibility of AS detection and severity classification based only on two-dimensional echocardiographic data. We demonstrate that our proposed spatio-temporal architecture effectively and efficiently combines both anatomical features and motion of the aortic valve for AS severity classification. Our model can process cardiac echo cine series of varying length and can identify, without explicit supervision, the frames that are most informative towards the AS diagnosis. We present an empirical study on how the model learns phases of the heart cycle without any supervision and frame-level annotations. Our architecture outperforms state-of-the-art results on a private and a public dataset, achieving 95.2% and 91.5% in AS detection, and 78.1% and 83.8% in AS severity classification on the private and public datasets, respectively. Notably, due to the lack of a large public video dataset for AS, we made slight adjustments to our architecture for the public dataset. Furthermore, our method addresses common problems in training deep networks with clinical ultrasound data, such as a low signal-to-noise ratio and frequently uninformative frames. Our source code is available at: https://github.com/neda77aa/FTC.git.

Lung ultrasound (LUS) is an important imaging modality used by emergency physicians to assess pulmonary congestion at the patient bedside. B-line artifacts in LUS videos are key findings associated with pulmonary congestion. Not only can the interpretation of LUS be challenging for novice operators, but visual quantification of B-lines remains subject to observer variability. In this work, we investigate the strengths and weaknesses of multiple deep learning approaches for automated B-line detection and localization in LUS videos. We curate and publish,BEDLUS, a new ultrasound dataset comprising 1,419 videos from 113 patients with a total of 15,755 expert-annotated B-lines. Based on this dataset, we present a benchmark of established deep learning methods applied to the task of B-line detection. To pave the way for interpretable quantification of B-lines, we propose a novel “single-point” approach to B-line localization using only the point of origin. Our results show that (a) the area under the receiver operating characteristic curve ranges from 0.864 to 0.955 for the benchmarked detection methods, (b) within this range, the best performance is achieved by models that leverage multiple successive frames as input, and (c) the proposed single-point approach for B-line localization reaches an F$_{1}$-score of 0.65, performing on par with the inter-observer agreement. The dataset and developed methods can facilitate further biomedical research on automated interpretation of lung ultrasound with the potential to expand the clinical utility.

This paper presents U-LanD, a framework for automatic detection of landmarks on key frames of the video by leveraging the uncertainty of landmark prediction. We tackle a specifically challenging problem, where training labels are noisy and highly sparse. U-LanD builds upon a pivotal observation: a deep Bayesian landmark detector solely trained on key video frames, has significantly lower predictive uncertainty on those frames vs. other frames in videos. We use this observation as an unsupervised signal to automatically recognize key frames on which we detect landmarks. As a test-bed for our framework, we use ultrasound imaging videos of the heart, where sparse and noisy clinical labels are only available for a single frame in each video. Using data from 4,493 patients, we demonstrate that U-LanD can exceedingly outperform the state-of-the-art non-Bayesian counterpart by a noticeable absolute margin of 42% in${R}^{{2}}$score, with almost no overhead imposed on the model size.

Localization of anatomical landmarks to perform two-dimensional measurements in echocardiography is part of routine clinical workflow in cardiac disease diagnosis. Automatic localization of those landmarks is highly desirable to improve workflow and reduce interobserver variability. Training a machine learning framework to perform such localization is hindered given the sparse nature of gold standard labels; only few percent of cardiac cine series frames are normally manually labeled for clinical use. In this paper, we propose a new end-to-end reciprocal detection and tracking model that is specifically designed to handle the sparse nature of echocardiography labels. The model is trained using few annotated frames across the entire cardiac cine sequence to generate consistent detection and tracking of landmarks, and an adversarial training for the model is proposed to take advantage of these annotated frames. The superiority of the proposed reciprocal model is demonstrated using a series of experiments.

In echocardiography (echo), an electrocardiogram (ECG) is conventionally used to temporally align different cardiac views for assessing critical measurements. However, in emergencies or point-of-care situations, acquiring an ECG is often not an option, hence motivating the need for alternative temporal synchronization methods. Here, we propose Echo-SyncNet, a self-supervised learning framework to synchronize various cross-sectional 2D echo series without any human supervision or external inputs. The proposed framework takes advantage of two types of supervisory signals derived from the input data: spatiotemporal patterns found between the frames of a single cine (intra-view self-supervision) and interdependencies between multiple cines (inter-view self-supervision). The combined supervisory signals are used to learn a feature-rich and low dimensional embedding space where multiple echo cines can be temporally synchronized. Two intra-view self-supervisions are used, the first is based on the information encoded by the temporal ordering of a cine (temporal intra-view) and the second on the spatial similarities between nearby frames (spatial intra-view). The inter-view self-supervision is used to promote the learning of similar embeddings for frames captured from the same cardiac phase in different echo views. We evaluate the framework with multiple experiments: 1) Using data from 998 patients, Echo-SyncNet shows promising results for synchronizing Apical 2 chamber and Apical 4 chamber cardiac views, which are acquired spatially perpendicular to each other; 2) Using data from 3070 patients, our experiments reveal that the learned representations of Echo-SyncNet outperform a supervised deep learning method that is optimized for automatic detection of fine-grained cardiac cycle phase; 3) We go one step further and show the usefulness of the learned representations in a one-shot learning scenario of cardiac key-frame detection. Without any fine-tuning, key frames in 1188 validation patient studies are identified by synchronizing them with only one labeled reference cine. We do not make any prior assumption about what specific cardiac views are used for training, and hence we show that Echo-SyncNet can accurately generalize to views not present in its training set. Project repository: github.com/fatemehtd/Echo-SyncNet>.

Ensembling is now recognized as an effective approach for increasing the predictive performance and calibration of deep networks. We introduce a new approach, Parameter Ensembling by Perturbation (PEP), that constructs an ensemble of parameter values as random perturbations of the optimal parameter set from training by a Gaussian with a single variance parameter. The variance is chosen to maximize the log-likelihood of the ensemble average (𝕃) on the validation data set. Empirically, and perhaps surprisingly, 𝕃 has a well-defined maximum as the variance grows from zero (which corresponds to the baseline model). Conveniently, calibration level of predictions also tends to grow favorably until the peak of 𝕃 is reached. In most experiments, PEP provides a small improvement in performance, and, in some cases, a substantial improvement in empirical calibration. We show that this "PEP effect'' (the gain in log-likelihood) is related to the mean curvature of the likelihood function and the empirical Fisher information. Experiments on ImageNet pre-trained networks including ResNet, DenseNet, and Inception showed improved calibration and likelihood. We further observed a mild improvement in classification accuracy on these networks. Experiments on classification benchmarks such as MNIST and CIFAR-10 showed improved calibration and likelihood, as well as the relationship between the PEP effect and overfitting; this demonstrates that PEP can be used to probe the level of overfitting that occurred during training. In general, no special training procedure or network architecture is needed, and in the case of pre-trained networks, no additional training is needed.

Uncertainty of labels in clinical data resulting from intra-observer variability can have direct impact on the reliability of assessments made by deep neural networks. In this paper, we propose a method for modelling such uncertainty in the context of 2D echocardiography (echo), which is a routine procedure for detecting cardiovascular disease at point-of-care. Echo imaging quality and acquisition time is highly dependent on the operator's experience level. Recent developments have shown the possibility of automating echo image quality quantification by mapping an expert's assessment of quality to the echo image via deep learning techniques. Nevertheless, the observer variability in the expert's assessment can impact the quality quantification accuracy. Here, we aim to model the intra-observer variability in echo quality assessment as an aleatoric uncertainty modelling regression problem with the introduction of a novel method that handles the regression problem with categorical labels. A key feature of our design is that only a single forward pass is sufficient to estimate the level of uncertainty for the network output. Compared to the 0.11 ± 0.09 absolute error (in a scale from 0 to 1) archived by the conventional regression method, the proposed method brings the error down to 0.09 ± 0.08, where the improvement is statistically significant and equivalents to 5.7% test accuracy improvement. The simplicity of the proposed approach means that it could be generalized to other applications of deep learning in medical imaging, where there is often uncertainty in clinical labels.

Fully convolutional neural networks (FCNs), and in particular U-Nets, have achieved state-of-the-art results in semantic segmentation for numerous medical imaging applications. Moreover, batch normalization and Dice loss have been used successfully to stabilize and accelerate training. However, these networks are poorly calibrated i.e. they tend to produce overconfident predictions for both correct and erroneous classifications, making them unreliable and hard to interpret. In this paper, we study predictive uncertainty estimation in FCNs for medical image segmentation. We make the following contributions: 1) We systematically compare cross-entropy loss with Dice loss in terms of segmentation quality and uncertainty estimation of FCNs; 2) We propose model ensembling for confidence calibration of the FCNs trained with batch normalization and Dice loss; 3) We assess the ability of calibrated FCNs to predict segmentation quality of structures and detect out-of-distribution test examples. We conduct extensive experiments across three medical image segmentation applications of the brain, the heart, and the prostate to evaluate our contributions. The results of this study offer considerable insight into the predictive uncertainty estimation and out-of-distribution detection in medical image segmentation and provide practical recipes for confidence calibration. Moreover, we consistently demonstrate that model ensembling improves confidence calibration.

Accurate detection of end-systolic (ES) and end-diastolic (ED) frames in an echocardiographic cine series can be difficult but necessary pre-processing step for the development of automatic systems to measure cardiac parameters. The detection task is challenging due to variations in cardiac anatomy and heart rate often associated with pathological conditions. We formulate this problem as a regression problem and propose several deep learning-based architectures that minimize a novel global extrema structured loss function to localize the ED and ES frames. The proposed architectures integrate convolution neural networks (CNNs)-based image feature extraction model and recurrent neural networks (RNNs) to model temporal dependencies between each frame in a sequence. We explore two CNN architectures: DenseNet and ResNet, and four RNN architectures: long short-term memory, bi-directional LSTM, gated recurrent unit (GRU), and Bi-GRU, and compare the performance of these models. The optimal deep learning model consists of a DenseNet and GRU trained with the proposed loss function. On average, we achieved 0.20 and 1.43 frame mismatch for the ED and ES frames, respectively, which are within reported inter-observer variability for the manual detection of these frames.

Image guidance improves tissue sampling during biopsy by allowing the physician to visualize the tip and trajectory of the biopsy needle relative to the target in MRI, CT, ultrasound, or other relevant imagery. This paper reports a system for fast automatic needle tip and trajectory localization and visualization in MRI that has been developed and tested in the context of an active clinical research program in prostate biopsy. To the best of our knowledge, this is the first reported system for this clinical application and also the first reported system that leverages deep neural networks for segmentation and localization of needles in MRI across biomedical applications. Needle tip and trajectory were annotated on 583 T2-weighted intra-procedural MRI scans acquired after needle insertion for 71 patients who underwent transperineal MRI-targeted biopsy procedure at our institution. The images were divided into two independent training-validation and test sets at the patient level. A deep 3-D fully convolutional neural network model was developed, trained, and deployed on these samples. The accuracy of the proposed method, as tested on previously unseen data, was 2.80-mm average in needle tip detection and 0.98° in needle trajectory angle. An observer study was designed in which independent annotations by a second observer, blinded to the original observer, were compared with the output of the proposed method. The resultant error was comparable to the measured inter-observer concordance, reinforcing the clinical acceptability of the proposed method. The proposed system has the potential for deployment in clinical routine.

Current deep supervised learning methods typically require large amounts of labeled data for training. Since there is a significant cost associated with clinical data acquisition and labeling, medical datasets used for training these models are relatively small in size. In this paper, we aim to alleviate this limitation by proposing a variational generative model along with an effective data augmentation approach that utilizes the generative model to synthesize data. In our approach, the model learns the probability distribution of image data conditioned on a latent variable and the corresponding labels. The trained model can then be used to synthesize new images for data augmentation. We demonstrate the effectiveness of the approach on two independent clinical datasets consisting of ultrasound images of the spine and magnetic resonance images of the brain. For the spine dataset, a baseline and a residual model achieve an accuracy of 85% and 92%, respectively, using our method compared to 78% and 83% using a conventional training approach for image classification task. For the brain dataset, a baseline and a U-net network achieve an accuracy of 84% and 88%, respectively, in Dice coefficient in tumor segmentation compared to 80% and 83% for the convention training approach.

Temporal enhanced ultrasound (TeUS), comprising the analysis of variations in backscattered signals from a tissue over a sequence of ultrasound frames, has been previously proposed as a new paradigm for tissue characterization. In this paper, we propose to use deep recurrent neural networks (RNN) to explicitly model the temporal information in TeUS. By investigating several RNN models, we demonstrate that long short-term memory (LSTM) networks achieve the highest accuracy in separating cancer from benign tissue in the prostate. We also present algorithms for in-depth analysis of LSTM networks. Our in vivo study includes data from 255 prostate biopsy cores of 157 patients. We achieve area under the curve, sensitivity, specificity, and accuracy of 0.96, 0.76, 0.98, and 0.93, respectively. Our result suggests that temporal modeling of TeUS using RNN can significantly improve cancer detection accuracy over previously presented works.

Accurate identification of the needle target is crucial for effective epidural anesthesia. Currently, epidural needle placement is administered by a manual technique, relying on the sense of feel, which has a significant failure rate. Moreover, misleading the needle may lead to inadequate anesthesia, post dural puncture headaches, and other potential complications. Ultrasound offers guidance to the physician for identification of the needle target, but accurate interpretation and localization remain challenges. A hybrid machine learning system is proposed to automatically localize the needle target for epidural needle placement in ultrasound images of the spine. In particular, a deep network architecture along with a feature augmentation technique is proposed for automatic identification of the anatomical landmarks of the epidural space in ultrasound images. Experimental results of the target localization on planes of 3-D as well as 2-D images have been compared against an expert sonographer. When compared with the expert annotations, the average lateral and vertical errors on the planes of 3-D test data were 1 and 0.4 mm, respectively. On 2-D test data set, an average lateral error of 1.7 mm and vertical error of 0.8 mm were acquired.

Echocardiography (echo) is a skilled technical procedure that depends on the experience of the operator. The aim of this paper is to reduce user variability in data acquisition by automatically computing a score of echo quality for operator feedback. To do this, a deep convolutional neural network model, trained on a large set of samples, was developed for scoring apical four-chamber (A4C) echo. In this paper, 6,916 end-systolic echo images were manually studied by an expert cardiologist and were assigned a score between 0 (not acceptable) and 5 (excellent). The images were divided into two independent training-validation and test sets. The network architecture and its parameters were based on the stochastic approach of the particle swarm optimization on the training-validation data. The mean absolute error between the scores from the ultimately trained model and the expert's manual scores was 0.71 ± 0.58. The reported error was comparable to the measured intra-rater reliability. The learned features of the network were visually interpretable and could be mapped to the anatomy of the heart in the A4C echo, giving confidence in the training result. The computation time for the proposed network architecture, running on a graphics processing unit, was less than 10 ms per frame, sufficient for real-time deployment. The proposed approach has the potential to facilitate the widespread use of echo at the point-of-care and enable early and timely diagnosis and treatment. Finally, the approach did not use any specific assumptions about the A4C echo, so it could be generalizable to other standard echo views.

In the above-title paper [ibid., vol. 36, no. 6, pp. 1221-1230, Jun. 2017], the first footnote should have indicated the following information: A. H. Abdi and C. Luong are joint first authors.

We propose a joint information approach for automatic analysis of 2D echocardiography (echo) data. The approach combines a priori images, their segmentations and patient diagnostic information within a unified framework to determine various clinical parameters, such as cardiac chamber volumes, and cardiac disease labels. The main idea behind the approach is to employ joint Independent Component Analysis of both echo image intensity information and corresponding segmentation labels to generate models that jointly describe the image and label space of echo patients on multiple apical views, instead of independently. These models are then both used for segmentation and volume estimation of cardiac chambers such as the left atrium and for detecting pathological abnormalities such as mitral regurgitation. We validate the approach on a large cohort of echoes obtained from 6,993 studies. We report performance of the proposed approach in estimation of the left-atrium volume and detection of mitral-regurgitation severity. A correlation coefficient of 0.87 was achieved for volume estimation of the left atrium when compared to the clinical report. Moreover, we classified patients that suffer from moderate or severe mitral regurgitation with an average accuracy of 82%.

Segmentation of the wrist bones in CT images has been frequently used in different clinical applications including arthritis evaluation, bone age assessment and image-guided interventions. The major challenges include non-uniformity and spongy textures of the bone tissue as well as narrow inter-bone spaces. In this work, we propose an automatic wrist bone segmentation technique for CT images based on a statistical model that captures the shape and pose variations of the wrist joint across 60 example wrists at nine different wrist positions. To establish the correspondences across the training shapes at neutral positions, the wrist bone surfaces are jointly aligned using a group-wise registration framework based on a Gaussian Mixture Model. Principal component analysis is then used to determine the major modes of shape variations. The variations in poses not only across the population but also across different wrist positions are incorporated in two pose models. An intra-subject pose model is developed by utilizing the similarity transforms at all wrist positions across the population. Further, an inter-subject pose model is used to model the pose variations across different wrist positions. For segmentation of the wrist bones in CT images, the developed model is registered to the edge point cloud extracted from the CT volume through an expectation maximization based probabilistic approach. Residual registration errors are corrected by application of a non-rigid registration technique. We validate the proposed segmentation method by registering the wrist model to a total of 66 unseen CT volumes of average voxel size of 0.38 mm. We report a mean surface distance error of 0.33 mm and a mean Jaccard index of 0.86.

Low-dose-rate prostate brachytherapy treatment takes place by implantation of small radioactive seeds in and sometimes adjacent to the prostate gland. A patient specific target anatomy for seed placement is usually determined by contouring a set of collected transrectal ultrasound images prior to implantation. Standard-of-care in prostate brachytherapy is to delineate the clinical target anatomy, which closely follows the real prostate boundary. Subsequently, the boundary is dilated with respect to the clinical guidelines to determine a planning target volume. Manual contouring of these two anatomical targets is a tedious task with relatively high observer variability. In this work, we aim to reduce the segmentation variability and planning time by proposing an efficient learning-based multi-label segmentation algorithm. We incorporate a sparse representation approach in our methodology to learn a dictionary of sparse joint elements consisting of images, and clinical and planning target volume segmentation. The generated dictionary inherently captures the relationships among elements, which also incorporates the institutional clinical guidelines. The proposed multi-label segmentation method is evaluated on a dataset of 590 brachytherapy treatment records by 5-fold cross validation. We show clinically acceptable instantaneous segmentation results for both target volumes.

Recent studies highlight temporal ultrasound data as highly promising in differentiating between malignant and benign tissues in prostate cancer patients. Since Hidden Markov Models can be used for capturing order and patterns in time varying signals, we employ them to model temporal aspects of ultrasound data that are typically not incorporated in existing models. By comparing order-preserving and order-altering models, we demonstrate that the order encoded in the series is necessary to model the variability in ultrasound data of prostate tissues. In future studies, we will investigate the influence of order on the differentiation between malignant and benign tissues.

UNLABELLED: This paper presents the results of a computer-aided intervention solution to demonstrate the application of RF time series for characterization of prostate cancer, in vivo. METHODS: We pre-process RF time series features extracted from 14 patients using hierarchical clustering to remove possible outliers. Then, we demonstrate that the mean central frequency and wavelet features extracted from a group of patients can be used to build a nonlinear classifier which can be applied successfully to differentiate between cancerous and normal tissue regions of an unseen patient. RESULTS: In a cross-validation strategy, we show an average area under receiver operating characteristic curve (AUC) of 0.93 and classification accuracy of 80%. To validate our results, we present a detailed ultrasound to histology registration framework. CONCLUSION: Ultrasound RF time series results in differentiation of cancerous and normal tissue with high AUC.

A common challenge when performing surface-based registration of images is ensuring that the surfaces accurately represent consistent anatomical boundaries. Image segmentation may be difficult in some regions due to either poor contrast, low slice resolution, or tissue ambiguities. To address this, we present a novel non-rigid surface registration method designed to register two partial surfaces, capable of ignoring regions where the anatomical boundary is unclear. Our probabilistic approach incorporates prior geometric information in the form of a statistical shape model (SSM), and physical knowledge in the form of a finite element model (FEM). We validate results in the context of prostate interventions by registering pre-operative magnetic resonance imaging (MRI) to 3D transrectal ultrasound (TRUS). We show that both the geometric and physical priors significantly decrease net target registration error (TRE), leading to TREs of 2.35 ± 0.81 mm and 2.81 ± 0.66 mm when applied to full and partial surfaces, respectively. We investigate robustness in response to errors in segmentation, varying levels of missing data, and adjusting the tunable parameters. Results demonstrate that the proposed surface registration method is an efficient, robust, and effective solution for fusing data from multiple modalities.

In surface-based registration for image-guided interventions, the presence of missing data can be a significant issue. This often arises with real-time imaging modalities such as ultrasound, where poor contrast can make tissue boundaries difficult to distinguish from surrounding tissue. Missing data poses two challenges: ambiguity in establishing correspondences; and extrapolation of the deformation field to those missing regions. To address these, we present a novel non-rigid registration method. For establishing correspondences, we use a probabilistic framework based on a Gaussian mixture model (GMM) that treats one surface as a potentially partial observation. To extrapolate and constrain the deformation field, we incorporate biomechanical prior knowledge in the form of a finite element model (FEM). We validate the algorithm, referred to as GMM-FEM, in the context of prostate interventions. Our method leads to a significant reduction in target registration error (TRE) compared to similar state-of-the-art registration algorithms in the case of missing data up to 30%, with a mean TRE of 2.6 mm. The method also performs well when full segmentations are available, leading to TREs that are comparable to or better than other surface-based techniques. We also analyze robustness of our approach, showing that GMM-FEM is a practical and reliable solution for surface-based registration.

Low-dose-rate brachytherapy is a radiation treatment method for localized prostate cancer. The standard of care for this treatment procedure is to acquire transrectal ultrasound images of the prostate in order to devise a plan to deliver sufficient radiation dose to the cancerous tissue. Brachytherapy planning involves delineation of contours in these images, which closely follow the prostate boundary, i.e., clinical target volume. This process is currently performed either manually or semi-automatically, which requires user interaction for landmark initialization. In this paper, we propose a multi-atlas fusion framework to automatically delineate the clinical target volume in ultrasound images. A dataset of a priori segmented ultrasound images, i.e., atlases, is registered to a target image. We introduce a pairwise atlas agreement factor that combines an image-similarity metric and similarity between a priori segmented contours. This factor is used in an atlas selection algorithm to prune the dataset before combining the atlas contours to produce a consensus segmentation. We evaluate the proposed segmentation approach on a set of 280 transrectal prostate volume studies. The proposed method produces segmentation results that are within the range of observer variability when compared to a semi-automatic segmentation technique that is routinely used in our cancer clinic.

A single-task functional magnetic resonance imaging (fMRI) experiment may only partially highlight alterations to functional brain networks affected by a particular disorder. Multivariate analysis across multiple fMRI tasks may increase the sensitivity of fMRI-based diagnosis. Prior research using multi-task analysis in fMRI, such as those that use joint independent component analysis (jICA), has mainly assumed that brain activity patterns evoked by different tasks are independent. This may not be valid in practice. Here, we use sparsity, which is a natural characteristic of fMRI data in the spatial domain, and propose a joint sparse representation analysis (jSRA) method to identify common information across different functional subtraction (contrast) images in data from a multi-task fMRI experiment. Sparse representation methods do not require independence, or that the brain activity patterns be nonoverlapping. We use functional subtraction images within the joint sparse representation analysis to generate joint activation sources and their corresponding sparse modulation profiles. We evaluate the use of sparse representation analysis to capture individual differences with simulated fMRI data and with experimental fMRI data. The experimental fMRI data was acquired from 16 young (age: 19-26) and 16 older (age: 57-73) adults obtained from multiple speech comprehension tasks within subjects, where an independent measure (namely, age in years) can be used to differentiate between groups. Simulation results show that this method yields greater sensitivity, precision, and higher Jaccard indexes (which measures similarity and diversity of the true and estimated brain activation sources) than does the jICA method. Moreover, superiority of the jSRA method in capturing individual differences was successfully demonstrated using experimental fMRI data.

This work reports the use of ultrasound radio frequency (RF) time series analysis as a method for ultrasound-based classification of malignant breast lesions. The RF time series method is versatile and requires only a few seconds of raw ultrasound data with no need for additional instrumentation. Using the RF time series features, and a machine learning framework, we have generated malignancy maps, from the estimated cancer likelihood, for decision support in biopsy recommendation. These maps depict the likelihood of malignancy for regions of size 1 mm(2) within the suspicious lesions. We report an area under receiver operating characteristics curve of 0.86 (95% confidence interval [CI]: 0.84%-0.90%) using support vector machines and 0.81 (95% CI: 0.78-0.85) using Random Forests classification algorithms, on 22 subjects with leave-one-subject-out cross-validation. Changing the classification method yielded consistent results which indicates the robustness of this tissue typing method. The findings of this report suggest that ultrasound RF time series, along with the developed machine learning framework, can help in differentiating malignant from benign breast lesions, subsequently reducing the number of unnecessary biopsies after mammography screening.

A big challenge in sensorless image-based ultrasound tracking is in the out-of-plane motion estimation. The correlation value of a specific model of speckle known as fully developed speckle (FDS) can be used to estimate the out-of-plane displacement. In real tissue, this kind of pattern is rare and the deviation of speckle pattern from the ideal FDS model diminishes the accuracy of the out-of-plane motion estimation. In this paper a new method for estimation of the out-of-plane motion is proposed. Firstly a closed-form mathematical derivation is provided for the correlation of two RF echo signal patches at different positions. A linear regression model of the ultrasound beam profile is proposed to account for the spatial variability of the ultrasound beam and enhance the accuracy of out-of-plane motion estimation in real tissue. The statistical model of speckle used here is based on the Rician-Inverse Gaussian (RiIG) stochastic process of the speckle formation, which can be considered as a generalized form of the K-distribution with richer parametrization. In this work, for the first time the second-order statistics of the RIG model is used for speckle tracking. This statistical model allows for derivation of a closed-form formulation for the correlation coefficient based on the statistical parameters of every patch. Since the effect of coherency is considered in the RiIG model, it increases the reliability of the out-of-plane motion estimation. The flexibility of the proposed method enables almost any patch through the whole image to be used for the purpose of displacement estimation. The method has been evaluated both on ex vivo and in vivo tissues in various experiments including out-of-plane rotation (tilt, yaw) and free-hand imaging. The overall outcome demonstrates the potential of the proposed method for in vivo tissues.

Most conventional spine interventions are performed under X-ray fluoroscopy guidance. In recent years, there has been a growing interest to develop nonionizing imaging alternatives to guide these procedures. Ultrasound guidance has emerged as a leading alternative. However, a challenging problem is automatic identification of the spinal anatomy in ultrasound data. In this paper, we propose a local phase-based bone feature enhancement technique that can robustly identify the spine surface in ultrasound images. The local phase information is obtained using a gradient energy tensor filter. This information is used to construct local phase tensors in ultrasound images, which highlight the spine surface. We show that our proposed approach results in a more distinct enhancement of the bone surfaces compared to recently proposed techniques based on monogenic scale-space filters and logarithmic Gabor filters. We also demonstrate that registration accuracy of a statistical shape+pose model of the spine to 3-D ultrasound images can be significantly improved, using the proposed method, compared to those obtained using monogenic scale-space filters and logarithmic Gabor filters.

We propose a method for the analysis of brain structural data to simultaneously identify differences in position, orientation and size (i.e. pose), and in shape of multiple brain regions between young people with, and without, a depressive disorder. Different structures in both hemispheres of the brain of depressed and control participants were segmented and corresponding points on the surface of each structure were extracted. Coordinates of these surface points were used to obtain shape variations, and parameters of similarity transformations between brain structures across subjects were used to generate pose variations. Since these surface points and similarity transformations form Lie groups, a logarithmic mapping of members of the Lie groups was performed to transform them to a linear tangent space. Then, Independent Component Analysis (ICA) was used to obtain the independent sources of pose and shape variations on Lie group members, and their corresponding modulation profiles. A method for ordering the independent sources is proposed. The top ordered sources were used to detect pose and shape differences between the two groups, and confirm that even in their first depressive episode, the brains of depressed adolescents differ structurally from the brains of their nondepressed age- and sex-matched peers.

Segmentation of the spinal column from computed tomography (CT) images is a preprocessing step for a range of image-guided interventions. One intervention that would benefit from accurate segmentation is spinal needle injection. Previous spinal segmentation techniques have primarily focused on identification and separate segmentation of each vertebra. Recently, statistical multi-object shape models have been introduced to extract common statistical characteristics between several anatomies. These models can be used for segmentation purposes because they are robust, accurate, and computationally tractable. In this paper, we develop a statistical multi-vertebrae shape+pose model and propose a novel registration-based technique to segment the CT images of spine. The multi-vertebrae statistical model captures the variations in shape and pose simultaneously, which reduces the number of registration parameters. We validate our technique in terms of accuracy and robustness of multi-vertebrae segmentation of CT images acquired from lumbar vertebrae of 32 subjects. The mean error of the proposed technique is below 2 mm, which is sufficient for many spinal needle injection procedures, such as facet joint injections.

This paper presents a novel, fast, group-wise registration technique based on establishing soft correspondences between groups of point sets. The registration approach is used to create a statistical shape model, capable of learning the shape variations within a training set. The shape model consists of a mean shape and its transformations to all training shapes. We decouple the procedure into two steps: updating the mean shape and registering it to the training shapes. The algorithm alternates between these two steps until convergence. Following the generation of the statistical shape model, we use the soft correspondence approach to register the model to a new observation. We perform extensive experiments on two data sets: lumbar spine and hippocampi. We compare our algorithm to available state-of- the-art group-wise registration algorithms including feature-based and volume-based approaches. We demonstrate improved generalization, specificity and compactness compared to these algorithms.

We study the effect of point dependent (heteroscedastic) and identically distributed anisotropic fiducial localization noise on fiducial target registration error (TRE). We derive an analytic expression, based on the concept of mechanism spatial stiffness, for predicting TRE. The accuracy of the predicted TRE is compared to simulated values where the optimal registration transformation is computed using the heteroscedastic errors in variables algorithm. The predicted values are shown to be contained by the 95% confidence intervals of the root mean square TRE obtained from the simulations.

Image registration is a single point of failure in the image-guided computer-assisted surgery. Registration is primarily used to align and fuse the data sets taken from patient's anatomy before and during surgeries. Point-based rigid-body registration is usually performed by identifying corresponding fiducials (either natural landmarks or implanted ones) in the data sets. Since the localization of fiducials is imprecise and is generally perturbed by random noise, the performed registration is imperfect and has some error. Previous work has extensively analyzed the behavior of this error when the fiducial localization error has zero-mean over the entire set of fiducials. However, if noise has a nonzero-mean or a bias, no formulation yet exists to determine the effect of noise on the overall registration accuracy. In this work, we derive novel formulations that relate the bias in the localized fiducials to the accuracy of the performed registration. We analytically and numerically demonstrate that by eliminating the estimated bias from the measured fiducial locations, one can effectively increase the accuracy of the performed registration.

In point-based rigid-body registration, target registration error (TRE) is an important measure of the accuracy of the performed registration. The registration's accuracy depends on the fiducial localization error (FLE) which, in turn, is due to the measurement errors in the points (fiducials) used to perform the registration. FLE may have different characteristics and distributions at each point of the registering data sets, and along each orthogonal axis. Previously, the distribution of TRE was estimated based on the assumption that FLE has an independent, identical, and isotropic or anisotropic distribution for each point in the registering data sets. In this article, we present a general solution based on the Maximum Likelihood (ML) algorithm that estimates the distribution of TRE for the cases where FLE has an independent, identical or inhomogeneous, isotropic or anisotropic, distribution at each point in the registering data sets, and when an algorithm is available that is capable of calculating the optimum registration to first order. Mathematically, we show that the proposed algorithm simplifies to the one proposed by Fitzpatrick and West when FLE has an independent, identical, and isotropic distribution in the registering data sets. Furthermore, we use numerical simulations to show that the proposed algorithm accurately estimates the distribution of TRE when FLE has an independent, inhomogeneous, and anisotropic distribution in the registering data sets.

Rigid-body point-based registration is frequently used in computer assisted surgery to align corresponding points, or fiducials, in preoperative and intraoperative data. This alignment is mostly achieved by assuming the same homogeneous error distribution for all the points; however, due to the properties of the medical instruments used in measuring the point coordinates, the error distribution might be inhomogeneous and different for each point. In this paper, in an effort to understand the effect of error distribution in the localized points on the performed registration, we derive a closed-form solution relating the error distribution of each point with the performed registration accuracy. The solution uses maximum likelihood estimation to calculate the probability density function of registration error at each fiducial point. Extensive numerical simulations are performed to validated the proposed solution.

This paper presents a framework for the design of a haptic-based medical ultrasound training simulator. The proposed simulator is composed of a PHANToM haptic device and a modular software package that allows for visual feedback and kinesthetic interactions between an operator and multimodality image databases. The system provides real-time ultrasound images in the same fashion as a typical ultrasound machine, enhanced with corresponding augmented computerized tomographic (CT) and/or MRI images. The proposed training system allows trainees to develop radiology techniques and knowledge of the patient's anatomy with minimum practice on live patients, or in places or at times when radiology devices or patients with rare cases may not be available. Low-level details of the software structure that can be migrated to other similar medical simulators are described. A preliminary human factors study, conducted on the prototype of the developed simulator, demonstrates the potential usage of the system for clinical training.

This work presents a haptic-based medical ultrasound diagnostic simulator that can be used as an ultrasound training tool for radiology residents as well as an examination system for remote applications. The proposed system allows to develop radiology expertise with minimum practice on live patients, or in places or at times when radiology devices or patients with rare cases may not be available. The proposed simulator consists of a PC workstation with dual monitors, a PHANToMtrade haptic device and a modular software package that allows for visual feedback and kinesthetic interactions between the operator and multi-modality image databases. The system helps emulate a real ultrasound examination condition at hospital, which is enhanced with augmented CT and/or MRI images. The haptic interface creates position correspondence between the operator's hand and a virtual probe. Preliminary human factors studies have demonstrated significant potential of the developed system for scientific and commercial applications

We present and validate a novel registration algorithm mapping two data sets, generated from a rigid object, in the presence of Gaussian noise. The proposed method is based on the Unscented Kalman Filter (UKF) algorithm that is generally employed for analyzing nonlinear systems corrupted by additive Gaussian noise. First, we employ our proposed registration algorithm to fit two randomly generated data sets in the presence of isotropic Gaussian noise, when the corresponding points between the two data sets are assumed to be known. Then, we extend the registration method to the case where the data (with known correspondences) is stimulated by anisotropic Gaussian noise. The new registration method not only reliably converges to the correct registration solution, but it also estimates the variance, as a confidence measure, for each of the estimated registration transformation parameters. Furthermore, we employ the proposed registration algorithm for rigid-body, point-based registration where corresponding points between two registering data sets are unknown. The algorithm is tested on point data sets which are garnered from a pelvic cadaver and a scaphoid bone phantom by means of computed tomography (CT) and tracked free-hand ultrasound imaging. The collected 3-D points in the ultrasound frame are registered to the 3-D meshes in the CT frame by using the proposed and the standard Iterative Closest Points (ICP) registration algorithms. Experimental results demonstrate that our proposed method significantly outperforms the ICP registration algorithm in the presence of additive Gaussian noise. It is also shown that the proposed registration algorithm is more robust than the ICP registration algorithm in terms of outliers in data sets and initial misalignment between the two data sets.

With the rapid expansion of Internet in recent years, computer systems are facing increased number of security threats. Despite numerous technological innovations for information assurance, it is still very difficult to protect computer systems. Therefore, unwanted intrusions take place when the actual software systems are running. Different soft computing based approaches have been proposed to detect computer network attacks. This paper presents a genetic algorithm (GA) based approach to network intrusion detection, and the software implementation of the approach. The genetic algorithm is employed to derive a set of classification rules from network audit data, and the support-confidence framework is utilized as fitness function to judge the quality of each rule. The generated rules are then used to detect or classify network intrusions in a real-time environment. Unlike most existing GA-based approaches, because of the simple representation of rules and the effective fitness function, the proposed method is easier to implement while providing the flexibility to either generally detect network intrusions or precisely classify the types of attacks. Experimental results show the achievement of acceptable detection rates based on benchmark DARPA data sets on intrusions, while no other complementary techniques or relevant heuristics are applied.

The normal direction of the bone contour in computed tomography (CT) images provides important anatomical information and can guide segmentation algorithms. Since various bones in CT images have different sizes, and the intensity values of bone pixels are generally nonuniform and noisy, estimation of the normal direction using a single scale is not reliable. We propose a multiscale approach to estimate the normal direction of bone edges. The reliability of the estimation is calculated from the estimated results and, after re-scaling, the reliability is used to further correct the normal direction. The optimal scale at each point is obtained while estimating the normal direction; this scale is then used in a simple edge detector. Our experimental results have shown that use of this estimated/corrected normal direction improves the segmentation quality by decreasing the number of unexpected edges and discontinuities (gaps) of real contours. The corrected normal direction could also be used in postprocessing to delete false edges. Our segmentation algorithm is automatic, and its performance is evaluated on CT images of the human pelvis, leg, and wrist.

The paper presents a novel segmentation technique to extract prostate contours from transrectal ultrasound (TRUS) images. A sticks filter is first used to reduce the speckle and enhance the image contrast. The problem is then discretized by projecting equispaced radii from an arbitrary seed point inside the prostate cavity towards its boundary. The distance of the prostate boundary from the seed point is modeled by the trajectory of a moving object. The motion of this moving object is assumed to be governed by a finite set of dynamical models subject to uncertainty. Candidate edge points obtained along each radius include the measurement of the object position and some false returns. This modeling approach enables us to employ the interacting multiple model (IMM) estimator along with a probabilistic data association filter (PDAF) for prostate contour extraction. Since the method does not employ any numerical optimization, convergence is very fast. The robustness and accuracy of the method is demonstrated by segmenting contours from a series of prostate ultrasound images.

This paper presents a novel segmentation technique for extracting cavity contours from ultrasound images. The problem is first discretized by projecting equispaced radii from an arbitrary seed point inside the cavity toward its boundary. The distance of the cavity boundary from the seed point is modeled by the trajectory of a moving object. The motion of this moving object is assumed to be governed by a finite set of dynamical models subject to uncertainty. Candidate edge points obtained along each radius include the measurement of the object position and some false returns. The modeling approach enables us to use the interacting multiple model estimator along with a probabilistic data association filter, for contour extraction. The convergence rate of the method is very fast because it does not employ any numerical optimization. The robustness and accuracy of the method are demonstrated by segmenting contours from a series of ultrasound images. The results are validated through comparison with manual segmentations performed by an expert. An application of the method in segmenting bone contours from computed tomography images is also presented.

A robot-assisted system for medical diagnostic ultrasound has been developed by the authors. The paper presents the visual servo controller used in this system. While the ultrasound transducer is positioned by a robot, the operator, the robot controller, and an ultrasound image processor have shared control over its motion. Ultrasound image features that can be selected by the operator are recognized and tracked by a variety of techniques. Based on feature tracking, ultrasound image servoing in three axes has been incorporated in the interface and can be enabled to automatically compensate, through robot motions, unwanted motions in the plane of the ultrasound beam. The accuracy of the system is illustrated through a 3-D reconstruction of an ultrasound phantom. An Internet-based robot-assisted teleultrasound system has also been demonstrated.

A robot-assisted system for medical diagnostic ultrasound has been developed by the authors. This paper presents key features of the user interface used in this system. While the ultrasound transducer is positioned by a robot, the operator, the robot controller, and an ultrasound image processor have shared control over its motion. Ultrasound image features that can be selected by the operator are recognized and tracked by a variety of techniques. Based on feature tracking, ultrasound image servoing in three axes has been incorporated in the interface and can be enabled to automatically compensate, through robot motions, unwanted motions in the plane of the ultrasound beam. The stability and accuracy of the system is illustrated through a 3D reconstruction of an ultrasound phantom.

This paper presents the development of a novel, fully-automatic tracking and segmentation system to extract the boundary of the carotid artery from ultrasound images in real-time. The center of the carotid artery is tracked using the Star algorithm. The stability of the Star algorithm has been improved by using a temporal Kalman filter. A spatial Kalman filter is used to estimate the carotid artery boundary. Since the method does not employ any numerical optimization, convergence is very fast. The stability and accuracy of the method is demonstrated by tracking the carotid artery over a 30 second sequence of ultrasound images taken during a carotid artery examination. An application of the tracking method to ultrasound image servoing is also presented.

Describes the haptic rendering of a virtual environment by drawing upon concepts developed in the area of teleoperation. A four-channel teleoperation architecture is shown to be an effective means of coordinating the control of a 3-DOF haptic interface with the simulation of a virtual dynamic environment.

The authors previously (1999) presented a fully counter-balanced 6-DOF robot for ultrasound carotid artery diagnosis. This paper presents the design of its controller. The controller is a velocity controller capable of incorporating position control, force control, and image based control, using shared control. Safety is a significant issue in the design of the control system. The counterbalanced mechanical design enables the use of small motor torques which result in 10N maximum force at the probe, and allows the probe to remain in its position in case of power failure. The robot can be positioned by an input device via motion/force control and by image feature based control. It can also be positioned by directly pushing its linkage in case that either the patient or the sonographer shows the slightest sign of discomfort. Feasibility experiments are reported.