Significant and rapid advancements in cancer research have been attributed to Artificial Intelligence (AI). However, AI’s role and impact on the clinical side has been limited. This discrepancy manifests due to the overlooked, yet profound, differences in the clinical and research practices in oncology. Our contribution seeks to scrutinize physicians’ engagement with AI by interviewing 7 medical-imaging experts and disentangle its future alignment across the clinical and research workflows, diverging from the existing “one-size-fits-all” paradigm within Human-Centered AI discourses. Our analysis revealed that physicians’ trust in AI is less dependent on their general acceptance of AI, but more on their contestable experiences with AI. Contestability, in clinical workflows, underpins the need for personal supervision of AI outcomes and processes, i.e., clinician-in-the-loop. Finally, we discuss tensions in the desired attributes of AI, such as explainability and control, contextualizing them within the divergent intentionality and scope of clinical and research workflows.

International benchmarking competitions have become fundamental for the comparative performance assessment of image analysis methods. However, little attention has been given to investigating what can be learnt from these competitions. Do they really generate scientific progress? What are common and successful participation strategies? What makes a solution superior to a competing method? To address this gap in the literature, we performed a multicenter study with all 80 competitions that were conducted in the scope of IEEE ISBI 2021 and MICCAI 2021. Statistical analyses performed based on comprehensive descriptions of the submitted algorithms linked to their rank as well as the underlying participation strategies revealed common characteristics of winning solutions. These typically include the use of multi-task learning (63%) and/or multi-stage pipelines (61%), and a focus on augmentation (100%), image preprocessing (97%), data curation (79%), and post-processing (66%). The “typical” lead of a winning team is a computer scientist with a doctoral degree, five years of experience in biomedical image analysis, and four years of experience in deep learning. Two core general development strategies stood out for highly-ranked teams: the reflection of the metrics in the method design and the focus on analyzing and handling failure cases. According to the organizers, 43% of the winning algorithms exceeded the state of the art but only 11% completely solved the respective domain problem. The insights of our study could help researchers (1) improve algorithm development strategies when approaching new problems, and (2) focus on open research questions revealed by this work.

By focusing on metabolic and morphological tissue properties respectively, FluoroDeoxyGlucose (FDG)-Positron Emission Tomography (PET) and Computed Tomography (CT) modalities include complementary and synergistic information for cancerous lesion delineation and characterization (e.g. for outcome prediction), in addition to usual clinical variables. This is especially true in Head and Neck Cancer (HNC). The goal of the HEad and neCK TumOR segmentation and outcome prediction (HECKTOR) challenge was to develop and compare modern image analysis methods to best extract and leverage this information automatically. We present here the post-analysis of HECKTOR 2nd edition, at the 24th International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI) 2021. The scope of the challenge was substantially expanded compared to the first edition, by providing a larger population (adding patients from a new clinical center) and proposing an additional task to the challengers, namely the prediction of Progression-Free Survival (PFS). To this end, the participants were given access to a training set of 224 cases from 5 different centers, each with a pre-treatment FDG-PET/CT scan and clinical variables. Their methods were subsequently evaluated on a held-out test set of 101 cases from two centers. For the segmentation task (Task 1), the ranking was based on a Borda counting of their ranks according to two metrics: mean Dice Similarity Coefficient (DSC) and median Hausdorff Distance at 95th percentile (HD95). For the PFS prediction task, challengers could use the tumor contours provided by experts (Task 3) or rely on their own (Task 2). The ranking was obtained according to the Concordance index (C-index) calculated on the predicted risk scores. A total of 103 teams registered for the challenge, for a total of 448 submissions and 29 papers. The best method in the segmentation task obtained an average DSC of 0.759, and the best predictions of PFS obtained a C-index of 0.717 (without relying on the provided contours) and 0.698 (using the expert contours). An interesting finding was that best PFS predictions were reached by relying on DL approaches (with or without explicit tumor segmentation, 4 out of the 5 best ranked) compared to standard radiomics methods using handcrafted features extracted from delineated tumors, and by exploiting alternative tumor contours (automated and/or larger volumes encompassing surrounding tissues) rather than relying on the expert contours. This second edition of the challenge confirmed the promising performance of fully automated primary tumor delineation in PET/CT images of HNC patients, although there is still a margin for improvement in some difficult cases. For the first time, the prediction of outcome was also addressed and the best methods reached relatively good performance (C-index above 0.7). Both results constitute another step forward toward large-scale outcome prediction studies in HNC.

MOTIVATION: Rotated template matching is an efficient and versatile algorithm to analyze microscopy images, as it automates the detection of stereotypical structures, such as organelles that can appear at any orientation. Its performance however quickly degrades in noisy image data. RESULTS: We introduce Steer'n'Detect, an ImageJ plugin implementing a recently published algorithm to detect patterns of interest at any orientation with high accuracy from a single template in 2D images. Steer'n'Detect provides a faster and more robust substitute to template matching. By adapting to the statistics of the image background, it guarantees accurate results even in the presence of noise. The plugin comes with an intuitive user interface facilitating results analysis and further post-processing. AVAILABILITY AND IMPLEMENTATION: https://github.com/Biomedical-Imaging-Group/Steer-n-Detect. SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.

This paper relates the post-analysis of the first edition of the HEad and neCK TumOR (HECKTOR) challenge. This challenge was held as a satellite event of the 23rd International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI) 2020, and was the first of its kind focusing on lesion segmentation in combined FDG-PET and CT image modalities. The challenge's task is the automatic segmentation of the Gross Tumor Volume (GTV) of Head and Neck (H&N) oropharyngeal primary tumors in FDG-PET/CT images. To this end, the participants were given a training set of 201 cases from four different centers and their methods were tested on a held-out set of 53 cases from a fifth center. The methods were ranked according to the Dice Score Coefficient (DSC) averaged across all test cases. An additional inter-observer agreement study was organized to assess the difficulty of the task from a human perspective. 64 teams registered to the challenge, among which 10 provided a paper detailing their approach. The best method obtained an average DSC of 0.7591, showing a large improvement over our proposed baseline method and the inter-observer agreement, associated with DSCs of 0.6610 and 0.61, respectively. The automatic methods proved to successfully leverage the wealth of metabolic and structural properties of combined PET and CT modalities, significantly outperforming human inter-observer agreement level, semi-automatic thresholding based on PET images as well as other single modality-based methods. This promising performance is one step forward towards large-scale radiomics studies in H&N cancer, obviating the need for error-prone and time-consuming manual delineation of GTVs.

We provide a complete pipeline for the detection of patterns of interest in an image. In our approach, the patterns are assumed to be adequately modeled by a known template, and are located at unknown positions and orientations that we aim at retrieving. We propose a continuous-domain additive image model, where the analyzed image is the sum of the patterns to localize and a background with self-similar isotropic power-spectrum. We are then able to compute the optimal filter fulfilling the SNR criterion based on one single template and background pair: it strongly responds to the template while being optimally decoupled from the background model. In addition, we constrain our filter to be steerable, which allows for a fast template detection together with orientation estimation. In practice, the implementation requires to discretize a continuous-domain formulation on polar grids, which is performed using quadratic radial B-splines. We demonstrate the practical usefulness of our method on a variety of template approximation and pattern detection experiments. We show that the detection performance drastically improves when we exploit the statistics of the background via its power-spectrum decay, which we refer to as spectral-shaping. The proposed scheme outperforms state-of-the-art steerable methods by up to 50% of absolute detection performance.

Many image acquisition techniques used in biomedical imaging, material analysis, and structural geology are capable of acquiring 3-D solid images. Computational analysis of these images is complex but necessary since it is difficult for humans to visualize and quantify their detailed 3-D content. One of the most common methods to analyze 3-D data is to characterize the volumetric texture patterns. Texture analysis generally consists of encoding the local organization of image scales and directions, which can be extremely diverse in 3-D. Current state-of-the- art techniques face many challenges when working with 3-D solid texture, where most approaches are not able to consistently characterize both scale and directional information. 3-D Riesz- wavelets can deal with both properties. One key property of Riesz filterbanks is steerability, which can be used to locally align the filters and compare textures with arbitrary (local) orientations. This paper proposes and compares three novel local alignment criteria for higher-order 3-D Riesz-wavelet transforms. The estimations of local texture orientations are based on higher- order extensions of regularized structure tensors. An experimental evaluation of the proposed methods for the classification of synthetic 3-D solid textures with alterations (such as rotations and noise) demonstrated the importance of local directional information for robust and accurate solid texture recognition. These alignment methods improved the accuracy of the unaligned Riesz descriptors up to 0.63, from 0.32 to 0.95 over 1 in the rotated data, which is better than all other techniques that are published and tested on the same database.

We present texture operators encoding class-specific local organizations of image directions (LOIDs) in a rotation-invariant fashion. The LOIDs are key for visual understanding, and are at the origin of the success of the popular approaches, such as local binary patterns (LBPs) and the scale-invariant feature transform (SIFT). Whereas, LBPs and SIFT yield hand-crafted image representations, we propose to learn data-specific representations of the LOIDs in a rotation-invariant fashion. The image operators are based on steerable circular harmonic wavelets (CHWs), offering a rich and yet compact initial representation for characterizing natural textures. The joint location and orientation required to encode the LOIDs is preserved by using moving frames (MFs) texture representations built from locally-steered image gradients that are invariant to rigid motions. In a second step, we use support vector machines to learn a multi-class shaping matrix for the initial CHW representation, yielding data-driven MFs called steerable wavelet machines (SWMs). The SWM forward function is composed of linear operations (i.e., convolution and weighted combinations) interleaved with non-linear steermax operations. We experimentally demonstrate the effectiveness of the proposed operators for classifying natural textures. Our scheme outperforms recent approaches on several test suites of the Outex and the CUReT databases.

Visual pattern recognition is a key research topic in the field of image processing and computer vision. Texture analysis based on steerable Riesz wavelets is powerful, but requires computing pixel-wise operations resulting in a run time in the order of days when large volumes of data are processed. To overcome this limitation we propose a Graphics Processing Unit (GPU) based solution. A standard CPU version is used as starting point for the development of baseline GPU versions. To further increase the performance, and to overcome compute and memory limitations we apply a series of optimization techniques, leading to five versions in total. The best performing GPU solution ensures a speed-up of 93× for the parallelized section of the application and of 29.6× for the entire application. Furthermore, we show that a higher Riesz order and/or a higher image resolution further increases the speed-up.

This paper proposes a novel imaging biomarker of lung cancer relapse from 3-D texture analysis of CT images. Three-dimensional morphological nodular tissue properties are described in terms of 3-D Riesz-wavelets. The responses of the latter are aggregated within nodular regions by means of feature covariances, which leverage rich intra- and inter-variations of the feature space dimensions. When compared to the classical use of the average for feature aggregation, feature covariances preserve spatial co-variations between features. The obtained Riesz-covariance descriptors lie on a manifold governed by Riemannian geometry allowing geodesic measurements and differentiations. The latter property is incorporated both into a kernel for support vector machines (SVM) and a manifold-aware sparse regularized classifier. The effectiveness of the presented models is evaluated on a dataset of 110 patients with non-small cell lung carcinoma (NSCLC) and cancer recurrence information. Disease recurrence within a timeframe of 12 months could be predicted with an accuracy of 81.3-82.7%. The anatomical location of recurrence could be discriminated between local, regional and distant failure with an accuracy of 78.3-93.3%. The obtained results open novel research perspectives by revealing the importance of the nodular regions used to build the predictive models.

Image retrieval approaches can assist radiologists by finding similar images in databases as a means to providing decision support. In general, images are indexed using low-level imaging features, and a distance function is used to find the best matches in the feature space. However, using low-level features to capture the appearance of diseases in images is challenging and the semantic gap between these features and the high-level visual concepts in radiology may impair the system performance. We present a semantic framework that enables retrieving similar images based on high-level semantic image annotations. This framework relies on (1) an automatic approach to predict the annotations as semantic terms from Riesz texture image features and (2) a distance function to compare images considering both texture-based and radiodensity-based similarities among image annotations. Experiments performed on CT images emphasize the relevance of this framework.

We propose a texture learning approach that exploits local organizations of scales and directions. First, linear combinations of Riesz wavelets are learned using kernel support vector machines. The resulting texture signatures are modeling optimal class-wise discriminatory properties. The visualization of the obtained signatures allows verifying the visual relevance of the learned concepts. Second, the local orientations of the signatures are optimized to maximize their responses, which is carried out analytically and can still be expressed as a linear combination of the initial steerable Riesz templates. The global process is iteratively repeated to obtain final rotation-covariant texture signatures. Rapid convergence of class-wise signatures is observed, which demonstrates that the instances are projected into a feature space that leverages the local organizations of scales and directions. Experimental evaluation reveals average classification accuracies in the range of 97% to 98% for the Outex_TC_00010, the Outex_TC_00012, and the Contrib_TC_00000 suites for even orders of the Riesz transform, and suggests high robustness to changes in images orientation and illumination. The proposed framework requires no arbitrary choices of scales and directions and is expected to perform well in a large range of computer vision applications.

We describe a framework to model visual semantics of liver lesions in CT images in order to predict the visual semantic terms (VST) reported by radiologists in describing these lesions. Computational models of VST are learned from image data using linear combinations of high-order steerable Riesz wavelets and support vector machines (SVM). In a first step, these models are used to predict the presence of each semantic term that describes liver lesions. In a second step, the distances between all VST models are calculated to establish a nonhierarchical computationally-derived ontology of VST containing inter-term synonymy and complementarity. A preliminary evaluation of the proposed framework was carried out using 74 liver lesions annotated with a set of 18 VSTs from the RadLex ontology. A leave-one-patient-out cross-validation resulted in an average area under the ROC curve of 0.853 for predicting the presence of each VST. The proposed framework is expected to foster human-computer synergies for the interpretation of radiological images while using rotation-covariant computational models of VSTs to 1) quantify their local likelihood and 2) explicitly link them with pixel-based image content in the context of a given imaging domain.

In this paper, we propose mobile access to peer-reviewed medical information based on textual search and content-based visual image retrieval. Web-based interfaces designed for limited screen space were developed to query via web services a medical information retrieval engine optimizing the amount of data to be transferred in wireless form. Visual and textual retrieval engines with state-of-the-art performance were integrated. Results obtained show a good usability of the software. Future use in clinical environments has the potential of increasing quality of patient care through bedside access to the medical literature in context.

We propose near-affine-invariant texture descriptors derived from isotropic wavelet frames for the characterization of lung tissue patterns in high-resolution computed tomography (HRCT) imaging. Affine invariance is desirable to enable learning of nondeterministic textures without a priori localizations, orientations, or sizes. When combined with complementary gray-level histograms, the proposed method allows a global classification accuracy of 76.9% with balanced precision among five classes of lung tissue using a leave-one-patient-out cross validation, in accordance with clinical practice.

In this paper, classical approaches such as maximum combinations (combMAX), sum combinations (comb-SUM) and the product of the maximum and a non-zero number (combMNZ) were employed and the trade-off between two fusion effects (chorus and dark horse effects) was studied based on the sum of n maximums. Various normalization strategies were tried out. The fusion algorithms are evaluated using the best four visual and textual runs of the ImageCLEF medical image retrieval task 2008 and 2009. The results show that fused runs outperform the best original runs and multi-modality fusion statistically outperforms single modality fusion. The logarithmic rank penalization shows to be the most stable normalization. The dark horse effect is in competition with the chorus effect and each of them can produce best fusion performance depending on the nature of the input data.

This paper concerns lung tissue classification using asymmetric-margin support vector machine (ASVM) to handle the imbalance of the positive and negative classes in a one-against-all multiclass classification problem. The hyperparameters of the algorithm are obtained using an optimization of the upper bound of the leave-one-out error of the ASVM. The ASVM is applied on the dataset with its original distribution and oversampled so that the ratio of the examples is equal to the prevalence of patients having the tissue in the database. The two versions of the ASVM models were compared with a model build with a conventional SVM. The ASVM improved the results obtained with a conventional SVM. The incorporation of prior knowledge concerning the prevalence of the patients improved the results obtained with ASVM.

In this paper, we investigate the influence of the clinical context of high–resolution computed tomography (HRCT) images of the chest on tissue classification. Evaluation of the classification performance is based on high–quality visual data extracted from clinical routine. The clinical attributes with highest information gain ratio show to be relevant and consistent for the classification of lung tissue patterns. A combination of visual and clinical attributes allowed a mean of 93% correct predictions of testing instances among the five classes of lung tissue with optimized support vector machines (SVM), which represents a significant benefit of 8% compared to a pure visually–based classification.

Today's medical institutions produce enormous amounts of data on patients, including multimedia data, which is increasingly produced in digital form. These data in their clinical context contain much information and experience that is currently not being used up to its full potential. Through the digital form the data has become accessible for automatic analysis and treatment for a variety of applications. At the same time, the variety of images produced can be confusing even for trained specialists causing an information overload exists for many medical doctors. This suggests that content-based image retrieval can be a valuable tool for helping manage these data and access the right information at the right time. This article gives a short state of the art of content-based medical image retrieval followed by a description of the medGIFT project on image retrieval with its main components. Then, several challenges are used to illustrate areas where much more work is currently needed to advance biomedical image retrieval. This shows that we have now progresses beyond the phase, where medical doctors transfer a database to computer scientists to only evaluate their algorithms. We conclude that visual information retrieval can have a real impact in the medical field if the techniques can adapt to this rapidly changing field and get integrated into the workflow in radiology and other medical fields.