Determining the type of kidney stones is crucial for prescribing appropriate treatments to prevent recurrence. Currently, various approaches exist to identify the type of kidney stones. However, obtaining results through the reference ex vivo identification procedure can take several weeks, while in vivo visual recognition requires highly trained specialists. For this reason, deep learning models have been developed to provide urologists with an automated classification of kidney stones during ureteroscopies. Nevertheless, a common issue with these models is the lack of training data. This contribution presents a deep learning method based on few-shot learning, aimed at producing sufficiently discriminative features for identifying kidney stone types in endoscopic images, even with a very limited number of samples. This approach was specifically designed for scenarios where endoscopic images are scarce or where uncommon classes are present, enabling classification even with a limited training dataset. The results demonstrate that Prototypical Networks, using up to 25 % of the training data, can achieve performance equal to or better than traditional deep learning models trained with the complete dataset.

This contribution demonstrates the efficacy of targeted image pre-processing techniques in enhancing deep-learning-based 3D reconstruction of colon surfaces. It challenges the conventional approach of applying global image illumination corrections or only specular reflection removal in colonoscopy by advocating for the correction of local under-and over-exposures. Initially, an overview of the pipeline, encompassing image exposure correction coupled with a Recurrent Neural Network Simultaneous Localization and Mapping (RNN-SLAM) system is provided. Subsequently, this paper quantifies the reconstruction accuracy of endoscope trajectories within the colon, comparing results obtained with and without appropriate illumination correction. Notably, the results underscore the significant impact of the Endo-LMSPEC method on trajectory accuracy. Through targeted exposure correction, the average pose error (APE) is notably reduced, accompanied by a decrease in the root mean square error (RMSE). Moreover, the median pose error experiences a substantial improvement, highlighting the robustness of the Endo-LMSPEC method in mitigating local illumination artifacts. These findings underscore the critical role of tailored image pre-processing techniques in achieving more accurate and reliable 3D reconstructions of colon surfaces during endoscopic procedures.

Endoscopy is a widely used imaging technique for diagnosing diseases in hollow organs. However, endoscopic images often suffer from limited visibility and can be affected by many imaging artifacts, such as those related to under or overexposure, which can hamper the performance of AI-based diagnostic tools. Addressing this issue is challenging; thus, most previous work has focused on enhancing underexposed images. In this contribution, we propose an extension to the objective function and deep neural network layers of the IAT Vision Transformer model (Illumination Adaptive Transformer), designed initially for enhancing lowlight or ill-exposed images from natural scenes. Our approach specifically targets exposure correction in endoscopic imaging to preserve color and fine-scale details. First, an extra color normalization layer inside the IAT model has been integrated into the model, and secondly, the objective function has been extended with a Laplacian Pyramid Loss to evaluate different image patches, whereas a histogram-aware loss (HistoLoss) has been used to preserve the color quality of the enhanced endoscopic image, both of these combined allow for the output images not only to improve quantitatively but also qualitative wise. We evaluate our method on the Endo4IE dataset and demonstrate significant improvements over a previous method (Endo-LMSPEC) tailored specifically to endoscopic imaging. Compared with the state-of-the-art (Endo-LMSPEC), our approach achieves an SSIM increase of 2.3% and PSNR increase of 0.523 dB for overexposed images, along with an increase of 2.7% and 1.458 dB improvement in PSNR for underexposure, all while employing only ≈ 90k parameters and running at an inference time of ≈ 74 FPS, outperforming existing state-of-the-art methods on the same dataset and objective.

Contemporary Deep Learning (DL) image classifier approaches typically harness training set correlations to discern meaningful associations between inputs and outputs, often without differentiating causal connections from mere correlations. This practice can lead to Explainable Artificial Intelligence (XAI) techniques that, while identifying key input features, may base explanations on these correlations, thus risking confounded interpretations. This issue is particularly critical in medical imaging, where precise model explanations are vital. To tackle this, we build upon previous efforts for estimating causal links between model features and outputs, introducing the Explainable and Causal Feature Analysis (ECFA) method. Employing ECFA in a medical classification case study, we aim to empower medical professionals to differentiate between causally relevant model-extracted features and correlated features. Our experiments show that ECFA reliably pinpoints the top 1% of causal and anti-causal features to the output labels of a CNN-based classifier, aiding in the assessment of whether model predictions are causally grounded or correlation-based. This facilitates a more informed evaluation of whether a model’s predictions derive from distinguishable causal links or not, marking a notable stride toward enhancing the reliability and interpretability of DL models in medical diagnostics.

Currently, the Morpho-Constitutional Analysis (MCA) is the de facto approach for the etiological diagnosis of kidney stone formation, and it is an important step for establishing personalized treatment to avoid relapses. More recently, research has focused on performing such tasks intra-operatively, an approach known as Endoscopic Stone Recognition (ESR). Both methods rely on features observed in the surface and the section of kidney stones to separate the analyzed samples into several sub-groups. However, given the high intra-observer variability and the complex operating conditions found in ESR, there is a lot of interest in using AI for computer-aided diagnosis. However, current AI models require large datasets to attain a good performance and for generalizing to unseen distributions. This is a major problem as large labeled datasets are very difficult to acquire, and some classes of kidney stones are very rare. Thus, in this paper, we present a method based on diffusion as a way of augmenting pre-existing ex-vivo kidney stone datasets. Our aim is to create plausible diverse kidney stone images that can be used for pre-training models using ex-vivo data. We show that by mixing natural and synthetic images of CCD images, it is possible to train models capable of performing very well on unseen intra-operative data. Our results show that is possible to attain an improvement of 10% in terms of accuracy compared to a baseline model pre-trained only on ImageNet. Moreover, our results show an improvement of 6% for surface images and 10% for section images compared to a model train on CCD images only, which demonstrates the effectiveness of using synthetic images.

The Endoscopy Computer Vision Challenge (EndoCV) is a crowd-sourcing initiative to address eminent problems in developing reliable computer aided detection and diagnosis endoscopy systems and suggest a pathway for clinical translation of technologies. Whilst endoscopy is a widely used diagnostic and treatment tool for hollow-organs, there are several core challenges often faced by endoscopists, mainly: 1) presence of multi-class artefacts that hinder their visual interpretation, and 2) difficulty in identifying subtle precancerous precursors and cancer abnormalities. Artefacts often affect the robustness of deep learning methods applied to the gastrointestinal tract organs as they can be confused with tissue of interest. EndoCV2020 challenges are designed to address research questions in these remits. In this paper, we present a summary of methods developed by the top 17 teams and provide an objective comparison of state-of-the-art methods and methods designed by the participants for two sub-challenges: i) artefact detection and segmentation (EAD2020), and ii) disease detection and segmentation (EDD2020). Multi-center, multi-organ, multi-class, and multi-modal clinical endoscopy datasets were compiled for both EAD2020 and EDD2020 sub-challenges. The out-of-sample generalization ability of detection algorithms was also evaluated. Whilst most teams focused on accuracy improvements, only a few methods hold credibility for clinical usability. The best performing teams provided solutions to tackle class imbalance, and variabilities in size, origin, modality and occurrences by exploring data augmentation, data fusion, and optimal class thresholding techniques.

This paper introduces a structure from motion (SfM)-based surface reconstruction method for images including weak textures and structures. In SfM, the quality of the determination of homologous points between images plays a key role in terms of reconstruction performances. However, classical feature matching-based methods as integrated in the state-of-the-art SfM-algorithms are often inoperative for images with weak structures and textures. This contribution describes a dense optical flow-based solution enabling the point correspondence determination in such scenes. The accuracy and robustness of the proposed method were validated using results obtained for a phantom with known dimensions and with real medical data, respectively. Complex internal stomach wall surfaces were constructed using gastroscopic images.

This paper introduces a robust image mosaicing method. A variational optical flow (OF) method is first proposed to deal with scenes exhibiting strong specular reflections and few texture information. Then, a general form of descriptors invariant to complex illumination variations is given from which a novel descriptor is obtained. Non-linear transformations computed with the OF fields between the images are used to construct the mosaics. Experimental results demonstrate that the proposed method leads to coherent mosaics, even for complex gastroscopic image sequences. Furthermore, it can be potentially used to build mosaics for non-medical scenes.

This paper introduces a generalized descriptor formulation facilitating the design of illumination invariant data-terms in variational optical flow. This contribution also proposes a criterion to check whether a patch-based descriptor is illumination invariant or not. To do so, a local model is used to simulate complex illumination changes between images. As an example, it is shown how a novel patch-based descriptor can be derived from the generalized form. The performances of this descriptor are compared to those of reference descriptors in the literature using data sets with and without strong illumination changes. The accuracy and robustness of the proposed descriptor is also demonstrated through tests on gastroscopic image sequences including complex illumination changes.

Curvature and torsion of discrete curves are important quantities in numerous applications in 3D image processing. Classical algorithms based on high order derivatives lead to high errors when computing torsion of 3D curves with discrete data of low resolution. To face this challenge we present a discrete parameter free approach to calculate the torsion values without fitting continuous curves on the discrete data. The proposed approach does not require prior knowledge and is of complexity O(nlog(n)). Preliminary results obtained with the proposed algorithm are compared to reference data (ground truth) of analytically known test curves. Results are also given for tomographic 3D images.

In white light cystoscopy, bladder images are characterized by a strong texture and scene illumination variability which complicates image mosaicing. State-of-art methods exhibit high image registration accuracy at the expense of computational time. We propose an algorithm which selects either a feature based method or an optical flow method according to the image texture; for fast and accurate bladder wall mosaicing. Total variation (TV) optical flow method (deduced by duality) guarantees robust registration of poorly textured images. Realistic phantom images are registered with subpixel accuracy with a processing speed-up by a factor of 8 and 16 for two reference methods. Patient data results also illustrate the performance of the algorithm.

When image primitives cannot be robustly extracted, the estimation of a perspective transformation between overlapping images can be formulated as a markov random field (MRF) and minimized efficiently using graph cuts. For well contrasted images with low noise level, a first order MRF leads to an accurate and robust registration. With increasing noise however, the registration quality decreases rapidly. This contribution presents a novel algorithm that enforces planarity (as required for perspective transformations) as a soft constraint by adding higher-order cliques to the energy formulation. Results show that for low levels of Gaussian noise (standard deviation σnϵ [0,4]), the algorithm performs comparably to the standard first order formulation. For increasing levels of noise (σnϵ [5,12]), the found solution is roughly twice as accurate (deviation of ≈2 pixels on average compared to ≈4 pixels for σn= 10).

Video-endoscopy is the standard clinical procedure for visual exploration of internal walls of hollow organs. For the bladder, the lesion diagnosis is complex because the endoscopic images are bi-dimensional and cover only small bladder areas. 3D endoscopes, based on stereoscopic active vision principles, were recently proposed and validated. This paper presents a 3D reconstruction algorithm using 2D texture images and a few 3D points located on the internal wall surfaces provided by such endoscopes. The algorithm constructs a 3D panoramic surface using the 3D reconstruction method guided by 2D image registration. We show on realistic bladder phantoms that the algorithm is able to reconstruct 3D points and surfaces with a sub-millimetre accuracy.

This contribution deals with the calibration of active stereoscopic systems consisting of a camera and a structured light projector. Usually, expensive equipments like precise calibration pieces or dedicated positioning devices are used for the projector calibration. We developed a flexible calibration method using only a planar target as calibration board without other equipment. The method requires only two image acquisitions from which the projector parameters are estimated using an optimization method. A quantitative evaluation demonstrates the accuracy and flexibility of the proposed calibration method.

Video endoscopy is one of the standard clinical procedures for visually detecting lesions on the internal wall of human bladders. In order to facilitate the diagnosis, it is helpful to build panoramic maps by registering consecutive images from the video sequence. We show how to efficiently reduce the computation time of graph cut based image registration by an order of magnitude. The number of nodes in a graph is greatly reduced using spatial image properties in order to minimize the loss of information. The set of edges in this sparse graph is obtained by applying a watershed transform on the set of nodes. This graph reduction has negligible negative effects on image registration quality compared to a dense graph cut, so that visually coherent panoramic maps of bladder walls can be built. Results demonstrate that the method improves the registration accuracy and reduces the computation time of other endoscopic bladder image registration methods. This work is an important step towards real time map construction.

We present a new method of endoscopic camera calibration for non-linear radial distortion correction. The algorithm implemented computes both projective (camera) and polynomial (distortion) transformations. The optimization process registrates the corrected distorted pattern image with the non-distorted one. Mutual information was used as measure of similarity and stochastic gradient descent method for optimization. The algorithm was tested with two b/w (chessboard, concentric circles) and one grey level patterns, for 3 angular positions of the endoscope (0/spl deg/, 5/spl deg/ and 10/spl deg/ to perpendicular). Convergence time increased with the angle. Maximal mean correction error was less than 0.45 % with optimized distortion parameters calculated for the grey level pattern. Tested inclinations did not have significant effects on errors. Results obtained show the interest of the method proposed that requires only approximative perpendicular positioning of the endoscope and simple grey level calibration patterns without precise geometrical characteristics.