Reliable detection and classification of bacteria and other pathogens in the human body, animals, food, and water is crucial for improving and safeguarding public health. For instance, identifying the species and its antibiotic susceptibility is vital for effective bacterial infection treatment. Here we show that phase contrast time-lapse microscopy combined with deep learning is sufficient to classify four species of bacteria relevant to human health. The classification is performed on living bacteria and does not require fixation or staining, meaning that the bacterial species can be determined as the bacteria reproduce in a microfluidic device, enabling parallel determination of susceptibility to antibiotics. We assess the performance of convolutional neural networks and vision transformers, where the best model attained a class-average accuracy exceeding 98%. Our successful proof-of-principle results suggest that the methods should be challenged with data covering more species and clinically relevant isolates for future clinical use.

Fluorescence staining techniques, such as Cell Painting, together with fluorescence microscopy have proven invaluable for visualizing and quantifying the effects that drugs and other perturbations have on cultured cells. However, fluorescence microscopy is expensive, time-consuming, labor-intensive, and the stains applied can be cytotoxic, interfering with the activity under study. The simplest form of microscopy, brightfield microscopy, lacks these downsides, but the images produced have low contrast and the cellular compartments are difficult to discern. Nevertheless, by harnessing deep learning, these brightfield images may still be sufficient for various predictive purposes. In this study, we compared the predictive performance of models trained on fluorescence images to those trained on brightfield images for predicting the mechanism of action (MoA) of different drugs. We also extracted CellProfiler features from the fluorescence images and used them to benchmark the performance. Overall, we found comparable and largely correlated predictive performance for the two imaging modalities. This is promising for future studies of MoAs in time-lapse experiments for which using fluorescence images is problematic. Explorations based on explainable AI techniques also provided valuable insights regarding compounds that were better predicted by one modality over the other.

With the emergence of high throughput single cell techniques, the understanding of the molecular and cellular diversity of mammalian organs have rapidly increased. In order to understand the spatial organization of this diversity, single cell data is often integrated with spatial data to create probabilistic cell maps. However, targeted cell typing approaches relying on existing single cell data achieve incomplete and biased maps that could mask the true diversity present in a tissue slide. Here we applied a de novo technique to spatially resolve and characterize cellular diversity of in situ sequencing data during human heart development. We obtained and made accessible well defined spatial cell-type maps of fetal hearts from 4.5 to 9 post conception weeks, not biased by probabilistic cell typing approaches. With our analysis, we could characterize previously unreported molecular diversity within cardiomyocytes and epicardial cells and identified their characteristic expression signatures, comparing them with specific subpopulations found in single cell RNA sequencing datasets. We further characterized the differentiation trajectories of epicardial cells, identifying a clear spatial component on it. All in all, our study provides a novel technique for conducting de novo spatial-temporal analyses in developmental tissue samples and a useful resource for online exploration of cell-type differentiation during heart development at sub-cellular image resolution.

OBJECTIVE: Large-scale microscopy-based experiments often result in images with rich but sparse information content. An experienced microscopist can visually identify regions of interest (ROIs), but this becomes a cumbersome task with large datasets. Here we present SimSearch, a framework for quick and easy user-guided training of a deep neural model aimed at fast detection of ROIs in large-scale microscopy experiments. METHODS: The user manually selects a small number of patches representing different classes of ROIs. This is followed by feature extraction using a pre-trained deep-learning model, and interactive patch selection pruning, resulting in a smaller set of clean (user approved) and larger set of noisy (unapproved) training patches of ROIs and background. The pre-trained deep-learning model is thereafter first trained on the large set of noisy patches, followed by refined training using the clean patches. RESULTS: The framework is evaluated on fluorescence microscopy images from a large-scale drug screening experiment, brightfield images of immunohistochemistry-stained patient tissue samples, and malaria-infected human blood smears, as well as transmission electron microscopy images of cell sections. Compared to state-of-the-art and manual/visual assessment, the results show similar performance with maximal flexibility and minimal a priori information and user interaction. CONCLUSIONS: SimSearch quickly adapts to different data sets, which demonstrates the potential to speed up many microscopy-based experiments based on a small amount of user interaction. SIGNIFICANCE: SimSearch can help biologists quickly extract informative regions and perform analyses on large datasets helping increase the throughput in a microscopy experiment.

Immunohistochemical (IHC) analysis of tissue biopsies is currently used for clinical screening of solid cancers to assess protein expression. The large amount of image data produced from these tissue samples requires specialized computational pathology methods to perform integrative analysis. Even though proteins are traditionally studied independently, the study of protein co-expression may offer new insights towards patients' clinical and therapeutic decisions. To explore protein co-expression, we constructed a modular image analysis pipeline to spatially align tissue microarray (TMA) image slides, evaluate alignment quality, define tumor regions, and ultimately quantify protein expression, before and after tumor segmentation. The pipeline was built with open-source tools that can manage gigapixel slides. To evaluate the consensus between pathologist and computer, we characterized a cohort of 142 gastric cancer (GC) cases regarding the extent of E-cadherin and CD44v6 expression. We performed IHC analysis in consecutive TMA slides and compared the automated quantification with the pathologists' manual assessment. Our results show that automated quantification within tumor regions improves agreement with the pathologists' classification. A co-expression map was created to identify the cores co-expressing both proteins. The proposed pipeline provides not only computational tools forwarding current pathology practices to explore co-expression, but also a framework for merging data and transferring information in learning-based approaches to pathology.

With the increasing amount of image data collected from biomedical experiments there is an urgent need for smarter and more effective analysis methods. Many scientific questions require analysis of image sub-regions related to some specific biology. Finding such regions of interest (ROIs) at low resolution and limiting the data subjected to final quantification at full resolution can reduce computational requirements and save time. In this paper we propose a three-step pipeline: First, bounding boxes for ROIs are located at low resolution. Next, ROIs are subjected to semantic segmentation into sub-regions at mid-resolution. We also estimate the confidence of the segmented sub-regions. Finally, quantitative measurements are extracted at full resolution. We use deep learning for the first two steps in the pipeline and conformal prediction for confidence assessment. We show that limiting final quantitative analysis to sub-regions with full confidence reduces noise and increases separability of observed biological effects.

Image-based multiplexed in situ RNA detection makes it possible to map the spatial gene expression of hundreds to thousands of genes in parallel, and thus discern at the same time a large numbers of different cell types to better understand tissue development, heterogeneity, and disease. Fluorescent signals are detected over multiple fluorescent channels and imaging rounds and decoded in order to identify RNA molecules in their morphological context. Here we present a graph-based decoding approach that models the decoding process as a network flow problem jointly optimizing observation likelihoods and distances of signal detections, thus achieving robustness with respect to noise and spatial jitter of the fluorescent signals. We evaluated our method on synthetic data generated at different experimental conditions, and on real data of in situ RNA sequencing, comparing results with respect to alternative and gold standard image decoding pipelines.

We propose contrastive coding to learn shared, dense image representations, referred to as CoMIRs (Contrastive Multimodal Image Representations). CoMIRs enable the registration of multimodal images where existing registration methods often fail due to a lack of sufficiently similar image structures. CoMIRs reduce the multimodal registration problem to a monomodal one, in which general intensity-based, as well as feature-based, registration algorithms can be applied. The method involves training one neural network per modality on aligned images, using a contrastive loss based on noise-contrastive estimation (InfoNCE). Unlike other contrastive coding methods, used for, e.g., classification, our approach generates image-like representations that contain the information shared between modalities. We introduce a novel, hyperparameter-free modification to InfoNCE, to enforce rotational equivariance of the learnt representations, a property essential to the registration task. We assess the extent of achieved rotational equivariance and the stability of the representations with respect to weight initialization, training set, and hyperparameter settings, on a remote sensing dataset of RGB and near-infrared images. We evaluate the learnt representations through registration of a biomedical dataset of bright-field and second-harmonic generation microscopy images; two modalities with very little apparent correlation. The proposed approach based on CoMIRs significantly outperforms registration of representations created by GAN-based image-to-image translation, as well as a state-of-the-art, application-specific method which takes additional knowledge about the data into account. Code is available at: https://github.com/MIDA-group/CoMIR.

MOTIVATION: Visual assessment of scanned tissue samples and associated molecular markers, such as gene expression, requires easy interactive inspection at multiple resolutions. This requires smart handling of image pyramids and efficient distribution of different types of data across several levels of detail. RESULTS: We present TissUUmaps, enabling fast visualization and exploration of millions of data points overlaying a tissue sample. TissUUmaps can be used both as a web service or locally in any computer, and regions of interest as well as local statistics can be extracted and shared among users. AVAILABILITY AND IMPLEMENTATION: TissUUmaps is available on github at github.com/wahlby-lab/TissUUmaps. Several demos and video tutorials are available at http://tissuumaps.research.it.uu.se/howto.html. SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.

It is well known that cells in tissue display a large heterogeneity in gene expression due to differences in cell lineage origin and variation in the local environment. Traditional methods that analyze gene expression from bulk RNA extracts fail to accurately describe this heterogeneity because of their intrinsic limitation in cellular and spatial resolution. Also, information on histology in the form of tissue architecture and organization is lost in the process. Recently, new transcriptome-wide analysis technologies have enabled the study of RNA molecules directly in tissue samples, thus maintaining spatial resolution and complementing histological information with molecular information important for the understanding of many biological processes and potentially relevant for the clinical management of cancer patients. These new methods generally comprise three levels of analysis. At the first level, biochemical techniques are used to generate signals that can be imaged by different means of fluorescence microscopy. At the second level, images are subject to digital image processing and analysis in order to detect and identify the aforementioned signals. At the third level, the collected data are analyzed and transformed into interpretable information by statistical methods and visualization techniques relating them to each other, to spatial distribution, and to tissue morphology. In this review, we describe state-of-the-art techniques used at all three levels of analysis. Finally, we discuss future perspective in this fast-growing field of spatially resolved transcriptomics.

Development of new stochastic super-resolution methods together with fluorescence microscopy imaging enables visualization of biological processes at increasing spatial and temporal resolution. Quantitative evaluation of such imaging experiments call for computational analysis methods that localize the signals with high precision and recall. Furthermore, it is desirable that the methods are fast and possible to parallelize so that the ever increasing amounts of collected data can be handled in an efficient way. We herein address signal detection in super-resolution microscopy by approaches based on compressed sensing. We describe how a previously published approach can be parallelized, reducing processing time at least four times. We also evaluate the effect of a greedy optimization approach on signal recovery at high noise and molecule density. Furthermore, our evaluation reveals how previously published compressed sensing algorithms have a performance that degrades to that of a random signal detector at high molecule density. Finally, we show the approximation of the imaging system's point spread function affects recall and precision of signal detection, illustrating the importance of parameter optimization. We evaluate the methods on synthetic data with varying signal to noise ratio and increasing molecular density, and visualize performance on real super-resolution microscopy data from a time-lapse sequence of living cells.

Cancer diagnosis is based on visual examination under a microscope of tissue sections from biopsies. But whereas pathologists rely on tissue stains to identify morphological features, automated tissue recognition using color is fraught with problems that stem from image intensity variations due to variations in tissue preparation, variations in spectral signatures of the stained tissue, spectral overlap and spatial aliasing in acquisition, and noise at image acquisition. We present a blind method for color decomposition of histological images. The method decouples intensity from color information and bases the decomposition only on the tissue absorption characteristics of each stain. By modeling the charge-coupled device sensor noise, we improve the method accuracy. We extend current linear decomposition methods to include stained tissues where one spectral signature cannot be separated from all combinations of the other tissues' spectral signatures. We demonstrate both qualitatively and quantitatively that our method results in more accurate decompositions than methods based on non-negative matrix factorization and independent component analysis. The result is one density map for each stained tissue type that classifies portions of pixels into the correct stained tissue allowing accurate identification of morphological features that may be linked to cancer.

This paper describes a system for in vitro cell migration analysis. Adult neural stem/progenitor cells are studied using time-lapse microscopy and thereafter stained immunohistochemically to find and distinguish between undifferentiated glial progenitor cells and cells having differentiated into type-1 or type-2 astrocytes. The cells are automatically segmented and tracked throughout the time-lapse sequence. The evaluation showed that 88% of the cells were correctly segmented and tracked by the automatic system. Upon characterization of the cell migration by hidden Markov modeling, it was found that the motion of glial progenitor cells was random 2/3 of the time, while the type-2 astrocytes showed a directed movement 2/3 of the time. This finding indicates possibilities for cell-type specific identification and cell sorting of live cells based on specific movement patterns in individual cell populations, which will have valuable applications in neurobiological research