Explainable Artificial Intelligence (XAI) has become increasingly vital for improving the transparency and reliability of neural network decisions. Transformer architectures have emerged as the state-of-the-art for various tasks across single modalities such as video, language, or signals, as well as for multimodal approaches. Although XAI methods for transformers are available, their potential impact during model training remains underexplored. Thus, we propose Explanation-guided Training (ET), leveraging an XAI method to identify salient input regions and guide the model to focus solely on these salient regions during training. We develop ET in a typical multimodal analysis framework using a multimodal transformer that operates on videos and signals. ET enhances the input by masking the non-salient regions for videos and enhances the signals with weights based on explanation scores for the sensor modality. Comparative evaluation with baseline vanilla training and the state-of-the-art XAI-based IFI method [1] shows that ET consistently outperforms them. We benchmark our method on the publicly available UCF50 video dataset to demonstrate that ET is better than vanilla training and IFI. A risk detection corpus comprising egocentric videos and wearable sensor data is used for multimodal evaluation. Our code is available at https://gitub.u-bordeaux.fr/mayyar/explain-to-train

The current paradigm of training deep neural networks relies on large, annotated and representative datasets. They assume a static world where the target domain does not change. However, in the real-world, data changes over time and is often available on the fly. Naive retraining on new data causes catastrophic forgetting and the network is unable to generalize on old data. Streaming learning is a type of incremental learning where networks learn sequentially and as soon as a sample is available from the data stream. Instead of training on every new sample, we propose an uncertainty based selection criteria to improve our previously proposed fast streaming learning method Move-to-Data (MTD), called Entropy-based MTD (EMTD). Besides, streaming learning methods have so far mostly used Convolutional Neural Networks (CNNs) but in recent times Vision Transformers (ViTs) have shown much better performances for many vision tasks. Therefore, we use ViT based Video Transformer to analyse MTD, EMTD and their gradient descent based "retargeting" steps. We have compared the performances of EMTD with MTD (w/wo retargeting) and a popular streaming learning method ExStream for the transformer. EMTD is able to outperform baseline MTD, and EMTD with retargeting achieves close results as ExStream and is ∼ 1.2 times faster.

Classification of Alzheimer’s disease from 3D structural Magnetic Resonance Imaging (sMRI) with deep neural networks has shown promising results in recent years. The decision interpretation of these networks is essential to aid medical experts to understand and rely on the results provided by such models. In this paper, we propose an adaptation of a recently developed feature-based explanation method and apply it to a 3D CNN architecture for the binary classification of Alzheimer’s disease and Normal Control from the hippocampal ROIs of brain sMRIs. We also compare our method to the state-of-the-art LRP method.

Identification of diseased kidney glomeruli and fibrotic regions remains subjective and time-consuming due to complete dependence on an expert kidney pathologist. In an attempt to automate the classification of glomeruli into normal and abnormal morphology and classification of fibrosis patches into mild, moderate and severe categories, we investigate three deep learning techniques: traditional transfer learning, pre-trained deep neural networks for feature extraction followed by supervised classification, and a novel Multi-Gaze Attention Network (MGANet) that uses multi-headed self-attention through parallel residual skip connections in a CNN architecture. Emperically, while the transfer learning models such as ResNet50, InceptionResNetV2, VGG19 and InceptionV3 acutely under-perform in the classification tasks, the Logistic Regression model augmented with features extracted from the InceptionResNetV2 shows promising results. Additionally, the experiments effectively ascertain that the proposed MGANet architecture outperforms both the former baseline techniques to establish the state of the art accuracy of 87.25% and 81.47% for glomerluli and fibrosis classification, respectively on the Renal Glomeruli Fibrosis Histopathological (RGFH) database.

A key challenge in renal diagnosis using digital pathology has been the scarcity of reliable annotated datasets that can act as a benchmark for histological investigations. This paper uses a novel medical image dataset, titled Glomeruli Classification Database (GCDB), consisting of renal glomeruli images bifurcated into binary classes of normal and abnormal morphology. Based on this dataset, we direct our pioneering efforts to explore suitable deep neural network techniques related to kidney tissue slide imaging so as to establish a state of the art in this relatively unexplored domain. The paper focuses on classifying normal and abnormal categories of glomeruli which are the vital blood filtration units of the kidney. The results obtained using publicly available transfer learning models are held in comparison with supervised classifiers configured with image features extracted from the last layers of pre-trained image classifiers. Contrary to popular belief, transfer learning models such as ResNet50 and InceptionV3 are empirically proved to under-perform for this particular task whereas the Logistic Regression model augmented with features from the InceptionResNetV2 show the most promising results on the GCDB dataset.