Auto-segmentation quality or accuracy influences their clinical usefulness. However, currently widely utilized segmentation metrics (e.g., Dice Coefficient (DC) and Hausdorff Distance (HD)) cannot effectively express the manual mending effort required when utilizing auto-segmentation results in clinical practice. In this article, we explore ways of evaluating auto-segmentations with clinical efficiency considerations in mind. The time required for correcting auto-segmentations by experts is recorded to indicate ground-truth mending effort. Extended from our previous work, five explicitly-defined metrics are studied in detail for their ability to predict mending effort. More importantly, we explore the use of deep learning networks to provide an implicit metric, which predict mending effort using auto-segmentation masks and original images as input. A 3-institution evaluation is conducted with 7 different anatomic organs in the setting of auto-contouring for radiation therapy planning. Among the five explicit metrics, one form of the proposed Mendability Index (MIhd) shows the best performance to indicate the mending effort for sparse objects with 6.2-14.4% error, while one form of HD (sHD) performs best when assessing large non-sparse objects. Interestingly, while the explicit metrics all require ground truth segmentations for estimating mending effort, the implicit models obtained via deep learning are effective in predicting mending efforts (with 2.9-12.9% error) without the need for ground-truth segmentations and directly from the given image plus the auto-segmentations. We conclude that once effort-predicting deep models are created, it is feasible to assess the clinical usability of new segmentation models, going beyond bench technical evaluation commonly done via explicit metrics.

Eye tracking research is important in computer vision because it can help us understand how humans interact with the visual world. Specifically for high-risk applications, such as in medical imaging, eye tracking can help us to comprehend how radiologists and other medical professionals search, analyze, and interpret images for diagnostic and clinical purposes. Hence, the application of eye tracking techniques in disease classification has become increasingly popular in recent years. Contemporary works usually transform gaze information collected by eye tracking devices into visual attention maps (VAMs) to supervise the learning process. However, this is a time-consuming preprocessing step, which stops us from applying eye tracking to radiologists’ daily work. To solve this problem, we propose a novel gaze-guided graph neural network (GNN), GazeGNN, to leverage raw eye-gaze data without being converted into VAMs. In GazeGNN, to directly integrate eye gaze into image classification, we create a unified representation graph that models both images and gaze pattern information. With this benefit, we develop a real-time, real-world, end-to-end disease classification algorithm for the first time in the literature. This achievement demonstrates the practicality and feasibility of integrating real-time eye tracking techniques into the daily work of radiologists. To our best knowledge, GazeGNN is the first work that adopts GNN to integrate image and eye-gaze data. Our experiments on the public chest X-ray dataset show that our proposed method exhibits the best classification performance compared to existing methods. The code is available at https://github.com/ukaukaaaa/GazeGNN.

OBJECTIVE: The objective was to develop a fully automated algorithm for abdominal fat segmentation and to deploy this method at scale in an academic biobank. MATERIALS AND METHODS: We built a fully automated image curation and labeling technique using deep learning and distributive computing to identify subcutaneous and visceral abdominal fat compartments from 52,844 computed tomography scans in 13,502 patients in the Penn Medicine Biobank (PMBB). A classification network identified the inferior and superior borders of the abdomen, and a segmentation network differentiated visceral and subcutaneous fat. Following technical evaluation of our method, we conducted studies to validate known relationships with visceral and subcutaneous fat. RESULTS: When compared with 100 manually annotated cases, the classification network was on average within one 5-mm slice for both the superior (0.4 ± 1.1 slice) and inferior (0.4 ± 0.6 slice) borders. The segmentation network also demonstrated excellent performance with intraclass correlation coefficients of 1.00 (P < 2 × 10-16) for subcutaneous and 1.00 (P < 2 × 10-16) for visceral fat on 100 testing cases. We performed integrative analyses of abdominal fat with the phenome extracted from the electronic health record and found highly significant associations with diabetes mellitus, hypertension, and renal failure, among other phenotypes. CONCLUSIONS: This work presents a fully automated and highly accurate method for the quantification of abdominal fat that can be applied to routine clinical imaging studies to fuel translational scientific discovery.