Histological examination remains the gold standard for breast cancer diagnosis, prognosis assessment and treatment guidance. Commercial molecular signature test, ONCOTYPEDX®is routinely used for luminal breast cancers to predict the probabilities of response to chemotherapy and disease recurrence. We attempted to predict RS using digital pathology and Weakly Supervised (WS) attention-based models like CLAM (Clustering-constrained Attention Multiple Instance Learning) [1] and TransMIL (Transformer based Correlated Multiple Instance Learning) [2] on our in-house dataset. In tissue samples, the malignant component is haphazardly admixed with the nonmalignant component in variable proportions. This represents a challenge for the WS attention-based models to identify high-valued diagnostic/prognostic areas within whole slide images (WSIs). To address this, we propose an interactive approach with a human in the middle (supervised) by creating a user-friendly Graphical User Interface (GUI) that allows a pathologist to provide feedback to heatmaps generated by any WS attention-based model. We incorporate the feedback from the GUI as expected scores and penalize its difference with the attention scores in the successive training process (current scores). We observe an improvement in RS prediction after the pathologist’s feedback - a 5% rise in AUC and 4% in accuracy for CLAM and a 4.5% increase in AUC and 3% in accuracy for TransMIL. We analyze the generated heatmaps and notice an improvement in cosine similarity between the expected scores and attention scores before and after the feedback - 5% and 10% increase for CLAM and TransMIL, respectively. The implementation of the proposed approach and the dataset is available for download1. Our adaptive, interactive feedback system harmonizes attention scores with expert intuition and instills higher confidence in the system’s predictions. This study establishes a potent synergy between AI and pathologists.

The analogy between Gaussian processes (GPs) and deep artificial neural networks (ANNs) has received a lot of interest, and has shown promise to unbox the blackbox of deep ANNs. Existing theoretical works put strict assumptions on the ANN (e.g. requiring all intermediate layers to be wide, or using specific activation functions). Accommodating those theoretical assumptions is hard in recent deep architectures, and those theoretical conditions need refinement as new deep architectures emerge. In this paper we derive an evidence lower-bound that encourages the GP's posterior to match the ANN's output without any requirement on the ANN. Using our method we find out that on 5 datasets, only a subset of those theoretical assumptions are sufficient. Indeed, in our experiments we used a normal ResNet-18 or feed-forward backbone with a single wide layer in the end. One limitation of training GPs is the lack of scalability with respect to the number of inducing points. We use novel computational techniques that allow us to train GPs with hundreds of thousands of inducing points and with GPU acceleration. As shown in our experiments, doing so has been essential to get a close match between the GPs and the ANNs on 5 datasets. We implement our method as a publicly available tool called GPEX: https://github.com/amirakbarnejad/gpex. On 5 datasets (4 image datasets, and 1 biological dataset) and ANNs with 2 types of functionality (classifier or attention-mechanism) we were able to find GPs whose outputs closely match those of the corresponding ANNs. After matching the GPs to the ANNs, we used the GPs' kernel functions to explain the ANNs' decisions. We provide more than 200 explanations (around 30 in the paper and the rest in the supplementary) which are highly interpretable by humans and show the ability of the obtained GPs to unbox the ANNs' decisions.

Automated cell detection and localization from microscopy images are significant tasks in biomedical research and clinical practice. In this paper, we design a new cell detection and localization algorithm that combines deep convolutional neural network (CNN) and compressed sensing (CS) or sparse coding (SC) for end-to-end training. We also derive, for the first time, a backpropagation rule, which is applicable to train any algorithm that implements a sparse code recovery layer. The key innovation behind our algorithm is that the cell detection task is structured as a point object detection task in computer vision, where the cell centers (i.e., point objects) occupy only a tiny fraction of the total number of pixels in an image. Thus, we can apply compressed sensing (or equivalently SC) to compactly represent a variable number of cells in a projected space. Subsequently, CNN regresses this compressed vector from the input microscopy image. The SC/CS recovery algorithm (L1optimization) can then recover sparse cell locations from the output of CNN. We train this entire processing pipeline end-to-end and demonstrate that end-to-end training improves accuracy over a training paradigm that treats CNN and CS-recovery layers separately. We have validated our algorithm on five benchmark datasets with excellent results.

The ability to detect certain types of cells in a microscopy image is important for a wide range of clinical applications. Cells often present huge variations in density and appearance, and often occupy only a small portion of an image. Consequently, general object detection methods in computer vision do not meet accuracy requirements: false / missed detections prevail. In this paper, we apply convolutional neural network (CNN) to regress a fixed length vector from a microscopy image. Then, L1minimization / compressed sensing (CS) recovers a variable number of cell locations from this fixed-length predicted vector. Our contribution in this work is combining CS with CNN to solve cell detection problem in a regression framework, which needs to handle a variable number of cells. Our method relies on the observation that the number of pixels indicating cell centroid locations is a tiny fraction of the total image size. Thus, we utilize this sparsity property by CS. The proposed method is evaluated with several state-of-the-art approaches on public cell datasets and it obtains superior or comparable performances.