It is well-known that severe pain and powerful pain medications cause short- and long-term damage to the developing nervous system of newborns. Caregivers routinely use physiological vital signs [Heart Rate (HR), Respiration Rate (RR), Oxygen Saturation (SR)] to monitor post-surgical pain in the Neonatal Intensive Care Unit (NICU). Here we present a novel approach that combines continuous, non-invasive monitoring of these vital signs and Computer Vision/Deep Learning to make automatic neonate pain detection with an accuracy of 74% AUC, 67.59% mAP. Further, we report for the first time our Early Pain Detection (EPD) approach that explores prediction of the time to onset of post-surgical pain in neonates. Our EPD can alert NICU workers to postoperative neonatal pain about 5 to 10 minutes prior to pain onset. In addition to alleviating the need for intermittent pain assessments by busy NICU nurses via long-term observation, our EPD approach creates a time window prior to pain onset for the use of less harmful pain mitigation strategies. Through effective pain mitigation prior to spinal sensitization, EPD could minimize or eliminate severe post-surgical pain and the consequential need for powerful analgesics in post-surgical neonates.

Accurately detecting pain in infants remains a complex challenge. Conventional deep neural networks used for analyzing infant cry sounds typically demand large labeled datasets, substantial computational power, and often lack interpretability. In this work, we introduce a novel approach that leverages OpenAI's vision-language model, GPT-4(V), combined with mel spectrogram-based representations of infant cries through prompting. This prompting strategy significantly reduces the dependence on large training datasets while enhancing transparency and interpretability. Using the USF-MNPAD-II dataset, our method achieves an accuracy of 83.33% with only 16 training samples, in contrast to the 4,914 samples required in the baseline model. To our knowledge, this represents the first application of few-shot prompting with vision-language models such as GPT-4o for infant pain classification.

Although concept-based approaches are widely used to explain a model's behavior and assess the contributions of different concepts in decision-making, identifying relevant concepts can be challenging for non-experts. This paper introduces a novel method that simplifies concept selection by leveraging the capabilities of a state-of-the-art large Vision-Language Model (VLM). Our method employs a VLM to select textual concepts that describe the classes in the target dataset. We then transform these influential textual concepts into human-readable image concepts using a text-to-image model. This process allows us to explain the targeted network in a post-hoc manner. Further, we use directional derivatives and concept activation vectors to quantify the importance of the generated concepts. We evaluate our method on a neonatal pain classification task, analyzing the sensitivity of the model's output for the generated concepts. The results demonstrate that the VLM not only generates coherent and meaningful concepts that are easily understandable by non-experts but also achieves performance comparable to that of natural image concepts without the need for additional annotation costs.

Current AI-based methods for the classification of cellular features in microscopy images require time- and labor-intensive processes for training models. Specific limitations include the need for a large amount of image data and major time commitments from domain experts for accurate ground truthing. We present a solution for overcoming these limitations using a state-of-the-art vision language model. Our approach uses GPT-4, the Vision Language Model (VLM) from OpenAI, for the analysis and classification of Iba-1 immuno-stained microglia cells in tissue sections through the mouse hippocampus. We used GPT-4 to classify a dataset of low-power (20x) images of Iba-1 immuno-stained microglia cells from tissue sections treated with saline or a potent neurotoxin (tri-methyl-tin, TMT). Rather than training with images from each class, the GPT-4 input consists of minimal ground-truth prompts for visual question answering. We introduce a novel human-in-the-loop approach to automate the selection of example image-text pairs as input prompts and generate explanatory text as the basis for separating images into distinct classes. We assess test accuracy and efficiency compared to the baseline results using a convolutional neural net applied to the same dataset. Compared to the baseline, the equivalence in accuracy (91%) and substantial (86%) improvement in throughput efficiency with considerably lower needs for input data or domain experts highlight the effectiveness of our new method for automatic image classification. Unlike traditional methods focused on labeling images with one-to-two-word tags, our pipeline generates understandable ground truth by incorporating explanatory text for each image.

The detection and segmentation of stained cells and nuclei are essential prerequisites for subsequent quantitative research for many diseases. Recently, deep learning has shown strong performance in many computer vision problems, including solutions for medical image analysis. Furthermore, accurate stereological quantification of microscopic structures in stained tissue sections plays a critical role in understanding human diseases and developing safe and effective treatments. In this article, we review the most recent deep learning approaches for cell (nuclei) detection and segmentation in cancer and Alzheimer's disease with an emphasis on deep learning approaches combined with unbiased stereology. Major challenges include accurate and reproducible cell detection and segmentation of microscopic images from stained sections. Finally, we discuss potential improvements and future trends in deep learning applied to cell detection and segmentation.

Deep Learning (DL)-based solutions have shown promising performance in assessing neonatal pain. However, the occlusion of the visual modality (face and body) is common in clinical settings due to several factors, including a prone sleeping position, low light, or swaddling. In such scenarios, other pain signals, such as audio signals, can be used as the major behavioral signs of pain. Although DL-based methods are proposed to assess pain from audio, these methods lack transparency and explainability (black box), which can decrease the user's trust in the automated decision. In this work, we visualize the neonate's audio signal as a spectrogram image to classify it as pain or no pain and present an instance-based approach for explaining the decision of the black-box model. Further, this work provides an analysis of the most helpful and harmful training instances using an influence score followed by assessing their impact on pain prediction. Experimental results demonstrate that the proposed approach can detect and remove harmful instances, eventually leading to a compressed dataset. Our results also show that the proposed work can add explainability to the current DL-based pain detection methods, which can enhance users' trust and provide a viable approach toward pain assessment in clinical settings.

Across basic research studies, cell counting requires significant human time and expertise. Trained experts use thin focal plane scanning to count (click) cells in stained biological tissue. This computer-assisted process (optical disector) requires a well-trained human to select a unique best z-plane of focus for counting cells of interest. Though accurate, this approach typically requires an hour per case and is prone to inter-and intra-rater errors. Our group has previously proposed deep learning (DL)-based methods to automate these counts using cell segmentation at high magnification. Here we propose a novel You Only Look Once (YOLO) model that performs cell detection on multi-channel z-plane images (disector stack). This automated Multiple Input Multiple Output (MIMO) version of the optical disector method uses an entire z-stack of microscopy images as its input, and outputs cell detections (counts) with a bounding box of each cell and class corresponding to the z-plane where the cell appears in best focus. Compared to the previous segmentation methods, the proposed method does not require time-and labor-intensive ground truth segmentation masks for training, while producing comparable accuracy to current segmentation-based automatic counts. The MIMO-YOLO method was evaluated on systematic-random samples of NeuN-stained tissue sections through the neocortex of mouse brains (n=7). Using a cross validation scheme, this method showed the ability to correctly count total neuron numbers with accuracy close to human experts and with 100% repeatability (Test-Retest).

Training deep learning models for unbiased stereology requires a large data set with associated ground truth. However manual ground truth annotation is tedious, time-consuming, and expert dependent. We propose an active deep learning method for automatic stereology counts using a snapshot ensemble approach. The method provides a confidence score for each mask in an unlabeled pool that reduces user verification to only images with high information content for training the deep learning model. The proposed method reduces the error rate to less than 1% for unbiased stereology cell counts on immunostained brain cells compared to manual stereology and requires ~25% less expert verification time compared to a previously proposed iterative deep learning approach.

Lack of enough labeled data is a major problem in building machine learning based models when the manual annotation (labeling) is error-prone, expensive, tedious, and time-consuming. In this paper, we introduce an iterative deep learning based method to improve segmentation and counting of cells based on unbiased stereology applied to regions of interest of extended depth of field (EDF) images. This method uses an existing machine learning algorithm called the adaptive segmentation algorithm (ASA) to generate masks (verified by a user) for EDF images to train deep learning models. Then an iterative deep learning approach is used to feed newly predicted and accepted deep learning masks/images (verified by a user) to the training set of the deep learning model. The error rate in unbiased stereology count of cells on an unseen test set reduced from about 3 % to less than 1 % after 5 iterations of the iterative deep learning based unbiased stereology process.

Decisions about cervical cancer diagnosis and classification currently require microscopic examination of cervical tissue by an expert pathologist. In the present study, which focused on full automation of this approach, we solely use nucleus-level features to classify tissues as normal or cancer. We propose Adaptive Nucleus Shape Modeling (ANSM) algorithm for nucleus-level analysis which consists of two steps to capture the nucleus-level information: adaptive multilevel thresholding segmentation; and shape approximation by ellipse fitting. After applying the proposed algorithm, the features are extracted for tissue classification. Experiments show that ANSM can achieve an accuracy of 93.33% with a false negative rate of zero in classifying cancer and healthy cervical tissues using nucleus texture features. This provides evidence that nucleus-level analysis is valuable in cervical histology image analysis.

To classify cervical cells as normal or cancer, the histological image must be segmented. After segmentation mean nuclear volume can be used to distinguish between normal and cancer cells. Due to the rapid reproduction of cancer cells, they have higher mean nuclear volume than typical normal cells. We propose a large ensemble of segmentations which separate normal and cancer cases based on the single feature of mean nuclear volume. Four basic segmentors with different parameters generate the segmentations. The mean nuclear volume is extracted from the segmentations. The dataset used for this paper contained multiple images from 30 normal and 32 cancer patients. Hematoxylin and eosin (H&E) was used to stain archival tissue sections from the normal cervix and cervical cancers. Results show it is possible to predict class with greater than 84% accuracy.

Design-based (unbiased) stereology provides an accurate, precise, and efficient method to quantify morphological parameters of biological microstructures, such as the total number of three-dimensional (3D) objects (cells) in stained tissue sections. The current requirement for extensive user interaction with commercially available computerized stereology systems limits the throughput of data collection. To increase the efficiency of this process, an algorithm was developed to automate data collection from stained objects in thick, transparent tissue sections. We present a novel approach to extract, count and classify stained objects of interest in 3D by linking them through a z-stack of images. Skeletonization and erosion are used to further segment the under segmented (overlapping) cells resulting from the extraction of out of focus cells in conjunction with in focus cells. Finally, 3D shape features, computed from the re-linked cells, are used for final classification of counted objects into “cells” and “not-cells”. We achieve a classification accuracy of 85% using SVM in a leave one-out experiment. The results demonstrate the effectiveness of our algorithm to count cells in 3D from thick, transparent tissue sections.