Determining the type of kidney stones is crucial for prescribing appropriate treatments to prevent recurrence. Currently, various approaches exist to identify the type of kidney stones. However, obtaining results through the reference ex vivo identification procedure can take several weeks, while in vivo visual recognition requires highly trained specialists. For this reason, deep learning models have been developed to provide urologists with an automated classification of kidney stones during ureteroscopies. Nevertheless, a common issue with these models is the lack of training data. This contribution presents a deep learning method based on few-shot learning, aimed at producing sufficiently discriminative features for identifying kidney stone types in endoscopic images, even with a very limited number of samples. This approach was specifically designed for scenarios where endoscopic images are scarce or where uncommon classes are present, enabling classification even with a limited training dataset. The results demonstrate that Prototypical Networks, using up to 25 % of the training data, can achieve performance equal to or better than traditional deep learning models trained with the complete dataset.

This contribution demonstrates the efficacy of targeted image pre-processing techniques in enhancing deep-learning-based 3D reconstruction of colon surfaces. It challenges the conventional approach of applying global image illumination corrections or only specular reflection removal in colonoscopy by advocating for the correction of local under-and over-exposures. Initially, an overview of the pipeline, encompassing image exposure correction coupled with a Recurrent Neural Network Simultaneous Localization and Mapping (RNN-SLAM) system is provided. Subsequently, this paper quantifies the reconstruction accuracy of endoscope trajectories within the colon, comparing results obtained with and without appropriate illumination correction. Notably, the results underscore the significant impact of the Endo-LMSPEC method on trajectory accuracy. Through targeted exposure correction, the average pose error (APE) is notably reduced, accompanied by a decrease in the root mean square error (RMSE). Moreover, the median pose error experiences a substantial improvement, highlighting the robustness of the Endo-LMSPEC method in mitigating local illumination artifacts. These findings underscore the critical role of tailored image pre-processing techniques in achieving more accurate and reliable 3D reconstructions of colon surfaces during endoscopic procedures.

Endoscopy is a widely used imaging technique for diagnosing diseases in hollow organs. However, endoscopic images often suffer from limited visibility and can be affected by many imaging artifacts, such as those related to under or overexposure, which can hamper the performance of AI-based diagnostic tools. Addressing this issue is challenging; thus, most previous work has focused on enhancing underexposed images. In this contribution, we propose an extension to the objective function and deep neural network layers of the IAT Vision Transformer model (Illumination Adaptive Transformer), designed initially for enhancing lowlight or ill-exposed images from natural scenes. Our approach specifically targets exposure correction in endoscopic imaging to preserve color and fine-scale details. First, an extra color normalization layer inside the IAT model has been integrated into the model, and secondly, the objective function has been extended with a Laplacian Pyramid Loss to evaluate different image patches, whereas a histogram-aware loss (HistoLoss) has been used to preserve the color quality of the enhanced endoscopic image, both of these combined allow for the output images not only to improve quantitatively but also qualitative wise. We evaluate our method on the Endo4IE dataset and demonstrate significant improvements over a previous method (Endo-LMSPEC) tailored specifically to endoscopic imaging. Compared with the state-of-the-art (Endo-LMSPEC), our approach achieves an SSIM increase of 2.3% and PSNR increase of 0.523 dB for overexposed images, along with an increase of 2.7% and 1.458 dB improvement in PSNR for underexposure, all while employing only ≈ 90k parameters and running at an inference time of ≈ 74 FPS, outperforming existing state-of-the-art methods on the same dataset and objective.

Contemporary Deep Learning (DL) image classifier approaches typically harness training set correlations to discern meaningful associations between inputs and outputs, often without differentiating causal connections from mere correlations. This practice can lead to Explainable Artificial Intelligence (XAI) techniques that, while identifying key input features, may base explanations on these correlations, thus risking confounded interpretations. This issue is particularly critical in medical imaging, where precise model explanations are vital. To tackle this, we build upon previous efforts for estimating causal links between model features and outputs, introducing the Explainable and Causal Feature Analysis (ECFA) method. Employing ECFA in a medical classification case study, we aim to empower medical professionals to differentiate between causally relevant model-extracted features and correlated features. Our experiments show that ECFA reliably pinpoints the top 1% of causal and anti-causal features to the output labels of a CNN-based classifier, aiding in the assessment of whether model predictions are causally grounded or correlation-based. This facilitates a more informed evaluation of whether a model’s predictions derive from distinguishable causal links or not, marking a notable stride toward enhancing the reliability and interpretability of DL models in medical diagnostics.

Improving the overall quality of exercise is crucial for achieving effective and safe techniques during gym workouts. Moreover, identifying errors during workouts can optimize training benefits and minimize the risk of injury. In this paper, we propose a cross-domain method to exploit angle information in human pose skeletons, aiming to detect fine-grained posture problems in complex real-world environments. Specifically, we integrate the Geometric Representation Extraction (GRE) module along with transformer-based pose estimation. Our approach demonstrates efficacy on the Fitness-AQA dataset, which comprises authentic exercise samples captured in real-world gym settings. This performance is achieved after pose estimation with approximately 164k parameters in its base configuration. The experimental results highlight that our method is a competitive approach compared to self-supervised video/image approaches in complex environments. In the Back Squat exercise, our method outperforms Motion Disentangling (MD) in detecting Knee Inward Error (KIE) with an F1-score of 0.4398. For Static Shallow Squat Error, it achieved the second-best F1-score of 0.8677, just 0.0017 below Cross-View Cross-Subject Pose Contrastive Learning (CVCSPC). In the Overhead Press exercise, the method significantly improved the detection of Knee error, achieving an F1-score of 0.8160, surpassing CVCSPC and other methods. Overall, these results demonstrate that the proposed method provides competitive performance compared to the state-of-the-art models while using ≈ 187x fewer parameters than the model with the highest performance in the AQA dataset, the Motion Disentangling (MD) approach. Code will be available at: https://github.com/CaroFernando/GymPose

Currently, the Morpho-Constitutional Analysis (MCA) is the de facto approach for the etiological diagnosis of kidney stone formation, and it is an important step for establishing personalized treatment to avoid relapses. More recently, research has focused on performing such tasks intra-operatively, an approach known as Endoscopic Stone Recognition (ESR). Both methods rely on features observed in the surface and the section of kidney stones to separate the analyzed samples into several sub-groups. However, given the high intra-observer variability and the complex operating conditions found in ESR, there is a lot of interest in using AI for computer-aided diagnosis. However, current AI models require large datasets to attain a good performance and for generalizing to unseen distributions. This is a major problem as large labeled datasets are very difficult to acquire, and some classes of kidney stones are very rare. Thus, in this paper, we present a method based on diffusion as a way of augmenting pre-existing ex-vivo kidney stone datasets. Our aim is to create plausible diverse kidney stone images that can be used for pre-training models using ex-vivo data. We show that by mixing natural and synthetic images of CCD images, it is possible to train models capable of performing very well on unseen intra-operative data. Our results show that is possible to attain an improvement of 10% in terms of accuracy compared to a baseline model pre-trained only on ImageNet. Moreover, our results show an improvement of 6% for surface images and 10% for section images compared to a model train on CCD images only, which demonstrates the effectiveness of using synthetic images.

The study of brain vascular patterns in preterm infants is relevant for identifying pathologies associated with brain irrigation. However, several drawbacks arise while using these types of images for diagnosis, such as noisy images and difficulties in the quantification of the vessel patterns. The goal of this research is to enhance the images for a subsequent segmentation stage. Thus, as a result of this research, an entire pipeline of denoising and segmentation is presented as a solution. For denoising the images, the combination of conventional techniques with unsupervised techniques based on deep learning was explored. The best method for the removal of noise was the combination of traditional methods and PN2V using a GMM model. A UNet model was trained utilizing noisy pictures for segmentation. Then it was tested using both denoised and noisy images. The findings demonstrated an improvement of 9.4% in the dice score when the model was trained using noisy images.

Data engineering has become a powerful tool for machine learning applications over the last few years. In computer vision for generative AI, the necessity of large amounts of data for training models has become a significant bottleneck. Data augmentation is a technique with limitations, even if it is useful. Training models with synthetic data are a solution due to the flexibility and scalability of the data creation that they can offer. When creating a synthetic dataset, one of the biggest challenges, however, is generating accurate and valuable data that can guarantee that the samples are a factual representation of the area of interest or an image; therefore, validating the dataset by subject matter experts becomes crucial. Examples of the multiple applications this method can use are image captioning, question-answering applications, generative AI, and overall multimodal problems. For image captioning, an image and its description are needed. Regions-of-Interest (ROI) in the image can be associated with text, forming a multimodal relationship associating an ROI inside an image and describing it. This work proposes a methodology to create automatic ROI-description multimodal Ulcerative Colitis (UC) dataset construction. To deal with the requirement of a large dataset for model training, we introduce stable diffusion for generating images that represent these patch-level characteristics widely used to classify a sample into an MES score. We utilise the clinically accepted phenotypes for informed decision-making. These include ulcers, bleeding, and erosions. We use this dataset to train a transformer-based detection pipeline (DETR) to find the characteristic inside the raw UC image to generate an ROI and associate it with a text template that describes the region. Finally, we compare our results against a baseline ROI dataset that medical experts have validated.

Frequent monitoring is necessary to stratify individuals based on their likelihood of developing gastrointestinal (GI) cancer precursors. In the clinical practice, white-light imaging (WLI), and complimentary modalities such as narrow-band imaging (NBI) and fluorescence imaging are used to assess risk areas. However, conventional deep learning (DL) models have depleted performance due to domain gap when a model is trained on one modality and tested on a different one. In our earlier approach we used superpixel based method referred to as “SUPRA” to effectively learn domain-invariant information using color and space distances to generate groups of pixels. One of the main limitations of this early work is that the aggregation does not exploit structural information, making it sub-optimal for segmentation tasks, especially for polyps and heterogeneous color distributions. Therefore, in this work, we propose an approach for style-content disentanglement using instance normalization and instance selective whitening (ISW) for an improved domain generalization when combined with SUPRA. We evaluate our approach on two datasets: EndoUDA Barret’s Esophagus and EndoUDA polyps and compare its performance with previous three state-of-the-art (SOTA) methods. Our findings demonstrate a notable enhancement in performance compared to both baseline and state-of-the-art methods across the target domain data. Specifically, our approach exhibited improvements of 14%, 10%, 8%, and 18% over the baseline and three SOTA methods on the polyp dataset. Additionally, it surpassed the second best method (EndoUDA) on the BE dataset by nearly 2%.

Early wildfire detection is of paramount importance to avoid as much damage as possible to the environment, properties, and lives. Deep Learning (DL) models that can leverage both visible and infrared information have the potential to display state-of-the-art performance, with lower false-positive rates than existing techniques. However, most DL-based image fusion methods have not been evaluated in the domain of fire imagery. Additionally, to the best of our knowledge, no publicly available dataset contains visible-infrared fused fire images. There is a growing interest in DL-based image fusion techniques due to their reduced complexity. Due to the latter, we select three state-of-the-art, DL-based image fusion techniques and evaluate them for the specific task of fire image fusion. We compare the performance of these methods on selected metrics. Finally, we also present an extension to one of the said methods, that we called FIRe-GAN, that improves the generation of artificial infrared images and fused ones on selected metrics.

Precise instrument segmentation aids surgeons to navigate the body more easily and increases patient safety. While accurate tracking of surgical instruments in real-time plays a crucial role in minimally invasive computer-assisted surgeries, it is a challenging task to achieve, mainly due to: (1) a complex surgical environment, and (2) model design trade-off in terms of both optimal accuracy and speed. Deep learning gives us the opportunity to learn complex environment from large surgery scene environments and placements of these instruments in real world scenarios. The Robust Medical Instrument Segmentation 2019 challenge (ROBUST-MIS) provides more than 10,000 frames with surgical tools in different clinical settings. In this paper, we propose a light-weight single stage instance segmentation model complemented with a convolutional block attention module for achieving both faster and accurate inference. We further improve accuracy through data augmentation and optimal anchor localization strategies. To our knowledge, this is the first work that explicitly focuses on both real-time performance and improved accuracy. Our approach out-performed top team performances in the most recent edition of ROBUST-MIS challenge with over 44% improvement on area-based multi-instance dice metric MI_DSC and 39% on distance-based multi-instance normalized surface dice MI_NSD. We also demonstrate real-time performance (>60 frames-per-second) with different but competitive variants of our final approach.

This article presents a co-design methodology based on RecoMARTE, an extension to the well-known UML MARTE profile, which is used for the specification and automatic generation of Dynamic and Partially Reconfigurable Systems-on-Chip (DRSoC). This endeavor is part of a larger framework in which Model-Driven Engineering (MDE) techniques are extensively used for modeling and via model transformations, generating executable models, which are exploited by implementation tools to create reconfigurable systems. More specifically, the methodological aspects presented in this article are concerned with expediting the conception and implementation of the hardware platform and the integration of correct by construction reconfiguration controller. This article builds upon previous research by integrating previously separated endeavors to obtain a complete PR system generation chain, which aims at shielding the designer of many of the burdensome technological and tool-specific requirements. The methodology permits for the verification of the platform description at different stages in the development process (i.e., HDL for simulation, static FPGA implementation, controller simulation and verification). Furthermore, automation capabilities embedded in the flow enable the generation of the platform description and the integration of the reconfiguration controller executive seamlessly. In order to demonstrate the benefits of the proposed approach, we present a case study in which we target the creation of an image-processing application to be deployed onto an FPGA board. We present the required modeling strategies and we discuss how the generation chains are integrated with the back-end Xilinx tools (the most mature version of PR technology) to produce the necessary executable artifacts: VHDL for the platform description and a C description of the reconfiguration controller to be executed by an embedded processor.

This paper presents a framework which facilitates the parameterization and integration of IP cores into partially reconfigurable SoC platforms, departing from a high-level of abstraction. The approach is based in a Model-Driven Engineering (MDE) methodology, which exploits two widely used standards for Systems-on-Chip specification, MARTE and IP-XACT. The presented work deals with the deployment level of the MDE approach, in which the abstract components of the platform are first linked to the lower level IP-XACT counterparts. At this phase, information for parameterization and integration is readily available, and a synthesizable model can be obtained from the generated IP-XACT through model transformations. We detail how certain IP-XACT objects are exploited in our approach; the emphasis is given to the generation of IP cores in a Xilinx EDK environment. We provide a case study in which a complete DPR platform is modeled in MARTE and implemented in a FPGA.