Abstract In recent years Vision language models (VLM) have significantly gained popularity to process and analyze medical imaging data. The features extracted by vision encoders from the VLMs can be used in different activities like embedding with relevant text prompts or forming unsupervised clusters to find hidden biomarkers. The major challenge in digital pathology is that a huge portion of available datasets are unlabeled or partially labelled which makes the segmentation or classification tasks difficult. Unsupervised learning is used to find hidden patterns from the unlabeled Whole Slide Images (WSIs) by extracting image embeddings and by finding significant features as potential biomarkers. In this study we introduce "Unsupervised VLM-Driven Biomarker Identification and Meta-Integration (UVLMBI)," a novel framework that is designed to leverage VLM capabilities in unsupervised biomarker discovery and integration against relevant clinical metadata. Our method is used to extract rich feature embeddings from whole slide images of LungMAP dataset, devoid of explicit labels, with an available sparse metadata. We applied a suite of state-of-the-art VLMs, with most of them pre-trained on pathology-related image datasets (UNI, Prov-Gigapath, Conch, and Plip), while some of the VLMs were used as generalized foundation model applicable to medical imaging data (such as Llava and Llava-next). Our approach begins with extraction of image embeddings from the image encoders of each VLMs. Subsequently multiple unsupervised clustering algorithms, such as K-Means, Agglomerative, Gaussian Mixture, and Spectral are applied on the embeddings. Based on the clustering performances, we identify the optimal clustering – VLM combinations for the later experiments. One key novelty of the UVLMBI is the integration of clinical metadata to enhance the biomarker discovery. We develop predictive models to identify relevant biomarkers by linking image-derived clusters to clinical metadata based on patient demographics, clinical history, and molecular data. Beyond the mere identification of putative biomarkers, the integration delivers information about their clinical relevance in the light of personalized medicine strategies. Our experimental results show the effectiveness of UVLMBI framework in finding clinically significant biomarkers and provide comprehensive evaluations for collections of VLMs in unsupervised medical image analysis. This framework opens new dimensions of advanced data-driven techniques in digital pathology and opens new ways of research and clinical applications. This study provides an exploratory investigation of using VLMs for unsupervised biomarker discovery, acknowledging the challenges in clustering performance with complex medical image datasets. Our experimental results on a set of WSIs from human developing lung tissues in combination with clinical metadata shows promising results and can lead to unsupervised biomarker discovery in computational pathology.

Analyzing histopathological images using state-of-the-art deep learning (DL) models requires specialized approaches due to the unique characteristics of these images. Recently, transformer architectures have been adopted to computer vision, treating images as sequences of patches, similar to words in natural language processing. Taking advantage of this, we employed transformers for histopathological image analysis. We utilized a knowledge distillation (KD) approach in which a student Vision Transformer (ViT) learns from a Convolutional Neural Network (CNN) teacher model. We assessed how the ViT’s attention heads were influenced by the signal received from the teacher. Building on that, this study further investigates the representations learned by the ViT’s attention heads. Our analysis reveals that many attention heads learn similar contexts or concepts, leading to redundancy in the learned representations. Additionally, by analyzing the mean of the maximum attention weights, we observed that only a few heads exhibit high confidence, indicating that they are critical for inference. To optimize the ViT, we performed head pruning at different rates, guided by ”head confidence” and ”head similarities”. Surprisingly, pruning certain heads not only maintained but, in some cases, improved model accuracy while also reducing training time when tested on breast cancer histopathological image classification.

Glioma is a malignant brain tumor that occurs in the central nervous system and is classified into high-grade glioblastoma multiforme (GBM) and low-grade glioma (LGG). GBM is a fatal malignant tumor with an extremely poor prognosis. In recent years, numerous studies have focused on the automatic classification of gliomas using histopathological images for diagnostic support. Among these, methods using multiple instance learning (MIL) have become mainstream. Many previous studies use pre-trained deep learning models as feature extractors for MIL. MIL processes multiple patch images at once, making low-cost end-to-end learning difficult. Therefore, we propose a MIL model using a reduced feature extractor. We use MobileNetV4 as the feature extractor and reduce its intermediate blocks to achieve a lightweight model. In this study, we incorporated the reduced feature extractor into five MIL models and compared the classification performance and computational cost. Experimental results using histopathological images of diffuse glioma patients from the Cancer Genome Atlas (TCGA) showed that the reduced models demonstrated classification performance equal to or better than the original models. Additionally, the reduced models have significantly lower computational costs and can easily achieve end-to-end learning. Therefore, our method is suggested to be useful for low-cost, end-to-end MIL classification using histopathological images.

ABSTRACT In this paper, we propose to use weighted ‐norm for approximating the solution of general ‐norm regularization problem for recovering hyperspectral images (HSI) corrupted by a mixture of Gaussian‐impulse noise. As a special case of , we design an optimization framework to accommodate the combined effect of different noise sources. An initial impulse noise pre‐detection phase decouples the raw noisy HSI data into impulse and Gaussian corrupted pixels. Gaussian corrupted pixels are handled by data‐fidelity term in while impulse corrupted pixels possess more Laplacian like behavior; modeled using . Solutions of problems involving in data fidelity and regularization terms complicate the optimization process but are less sensitive to the outlier pixels. On the other hand, the least square solutions for the data misfit are computationally efficient but generates solutions which are quite sensitive to the outlier pixels; which is the characteristic of impulse corrupted pixels. Therefore, in this paper, we decouple the set of pixels into two distinct parts; handled using two separate data fidelity terms. Total variation (TV) is used on the Casorati matrix representation of the input data to exploit similarity along both spatial and spectral dimensions. The resulting optimization problem is reformulated as iteratively reweighted least square for the general ‐norm problem for for data fidelity terms and for the TV regularization term. Experiments conducted over synthetically corrupted HSI data and images obtained from real HSI sensors confirm the suitability of the proposed weighted norm optimization framework (WNOF) over a wide range of degradation scenarios.

Abstract Digital images are corrupted with noise, and image denoising is an important step in image processing modules. In this review, the latest developments in filtering methods for color image restoration are analyzed. These algorithms are compared in terms of objective image quality measures and divided into major classes, such as spatial domain, switching and wavelet filtering methods. These classes are based on the particular methodology used in image denoising algorithms and further subdivided to show their classification in terms of noise models utilized, application style, and stages the filters applied in images. In particular, we present a review of filtering methods in color image denoising, published over the past two decades. Our classification and succinct descriptions of color image restoration by these mathematical filtering techniques and their characterizations can help choose the appropriate ones for various downstream image processing tasks.

Transcription factors read the genome, fundamentally connecting DNA sequence to gene expression across diverse cell types. Determining how, where, and when TFs bind chromatin will advance our understanding of gene regulatory networks and cellular behavior. The 2017 ENCODE-DREAM in vivo Transcription-Factor Binding Site (TFBS) Prediction Challenge highlighted the value of chromatin accessibility data to TFBS prediction, establishing state-of-the-art methods for TFBS prediction from DNase-seq. However, the more recent Assay-for-Transposase-Accessible-Chromatin (ATAC)-seq has surpassed DNase-seq as the most widely-used chromatin accessibility profiling method. Furthermore, ATAC-seq is the only such technique available at single-cell resolution from standard commercial platforms. While ATAC-seq datasets grow exponentially, suboptimal motif scanning is unfortunately the most common method for TFBS prediction from ATAC-seq. To enable community access to state-of-the-art TFBS prediction from ATAC-seq, we (1) curated an extensive benchmark dataset (127 TFs) for ATAC-seq model training and (2) built "maxATAC", a suite of user-friendly, deep neural network models for genome-wide TFBS prediction from ATAC-seq in any cell type. With models available for 127 human TFs, maxATAC is the largest collection of high-performance TFBS prediction models for ATAC-seq. maxATAC performance extends to primary cells and single-cell ATAC-seq, enabling improved TFBS prediction in vivo. We demonstrate maxATAC's capabilities by identifying TFBS associated with allele-dependent chromatin accessibility at atopic dermatitis genetic risk loci.

Cells and tissues respond to perturbations in multiple ways that can be sensitively reflected in the alterations of gene expression. Current approaches to finding and quantifying the effects of perturbations on cell-level responses over time disregard the temporal consistency of identifiable gene programs. To leverage the occurrence of these patterns for perturbation analyses, we developed CellDrift (https://github.com/KANG-BIOINFO/CellDrift), a generalized linear model-based functional data analysis method that is capable of identifying covarying temporal patterns of various cell types in response to perturbations. As compared to several other approaches, CellDrift demonstrated superior performance in the identification of temporally varied perturbation patterns and the ability to impute missing time points. We applied CellDrift to multiple longitudinal datasets, including COVID-19 disease progression and gastrointestinal tract development, and demonstrated its ability to identify specific gene programs associated with sequential biological processes, trajectories and outcomes.

Cytolytic T-cells play an essential role in the adaptive immune system by seeking out, binding and killing cells that present foreign antigens on their surface. An improved understanding of T-cell immunity will greatly aid in the development of new cancer immunotherapies and vaccines for life-threatening pathogens. Central to the design of such targeted therapies are computational methods to predict non-native peptides to elicit a T-cell response, however, we currently lack accurate immunogenicity inference methods. Another challenge is the ability to accurately simulate immunogenic peptides for specific human leukocyte antigen alleles, for both synthetic biological applications, and to augment real training datasets. Here, we propose a beta-binomial distribution approach to derive peptide immunogenic potential from sequence alone. We conducted systematic benchmarking of five traditional machine learning (ElasticNet, K-nearest neighbors, support vector machine, Random Forest and AdaBoost) and three deep learning models (convolutional neural network (CNN), Residual Net and graph neural network) using three independent prior validated immunogenic peptide collections (dengue virus, cancer neoantigen and SARS-CoV-2). We chose the CNN as the best prediction model, based on its adaptivity for small and large datasets and performance relative to existing methods. In addition to outperforming two highly used immunogenicity prediction algorithms, DeepImmuno-CNN correctly predicts which residues are most important for T-cell antigen recognition and predicts novel impacts of SARS-CoV-2 variants. Our independent generative adversarial network (GAN) approach, DeepImmuno-GAN, was further able to accurately simulate immunogenic peptides with physicochemical properties and immunogenicity predictions similar to that of real antigens. We provide DeepImmuno-CNN as source code and an easy-to-use web interface.

Visibility of outdoor images is usually limited due to haze, dust, smoke and other particles in air. Visibility limit can cause many difficulties for activities of transport, rescue, oceanography etc. Hence, image dehazing is very necessary. In this paper, we propose a single image dehazing method based on combination of adaptive histogram equalization, HSV color model and linearization of Gamma correction. In the experiments, we test the proposed method on hazy images of the TAU dataset. To assess dehazing quality, we utilize NIQE metric and compare to other dehazing methods. The results confirm that the proposed method dehazes effectively and can compete with other state-of-the-art dehazing methods.

Understanding of real scenes is an important task in augmented reality (AR). Identifying and comprehension of texts from real scene images is useful in implementing robust AR devices. Therefore, improving the quality of text images for better readability is very important and text images deconvolution is useful to increase the accuracy of AR pattern recognition algorithms. In this work, we propose an estimation method for the filtering operator within total variation deconvolution model. This method is applied for the texts image deconvolution problem from natural images. In the experiments, we use NIQE score - the blind quality assessment metric - to assess the deconvolution quality. We further compare the proposed method with other image deconvolution models such as the blind deconvolution, the Lucy and the Wiener methods. The experimental indicate that our deconvolution method works effectively for texts enhancement across different scenes with high quality results.

Deep learning models are now widely applied to various biomedical image analysis tasks such as the image segmentation and classification. However, automation of biomedical image analysis with deep learning is challenging since it requires highly specialized knowledge and large amounts of training data. In this work, we detail automatic multi-class segmentations using deep learning models for lung immunofluorescent (IF) confocal images, along with synthetic image generation of lung images for training these models. Analysis of lung imaging data is important for understanding the lung development at the molecular level and cross-sectional IF images are useful in identifying various structures of the lung. We tested multi-class segmentation using deep learning convolutional neural network (CNN) models with overwrap cropping method as preprocessing to make the dataset larger. Further, we generated synthetic images using deep convolution generative synthetic adversarial network (DCGAN) and use them in learned segmentation networks for creating segmentation masks. In terms of deep learning segmentation models, we adapted the state-of-the-art U-Net, SegNet, and DeepLabv3+ based models for multi-class segmentation from lung IF images. Our experimental results on these challenging lung IF images show that the highest dice score for training 98.7%, and testing 87.0% is obtained by an adapted multiclass U-Net method. Further, our synthetic image generation shows promise for future training paradigms in improving the segmentation of various lung structures in IF confocal images.

Regularization methods are used widely in image selective smoothing and edge preserving restoration of noisy images. Traditional methods utilize image gradients within regularization function for controlling the smoothing and can produce artifacts when noise levels are higher. In this paper, we consider a robust image adaptive exponent driven regularization for filtering noisy images with salient feature preservation. Our spatially adaptive variable exponent function depends on a continuous switch based on the eigenvalues of structure tensor which identifies noisy edges, and corners with higher accuracy. Structure tensor eigenvalues encode various image features and we consider a spatially varying continuous map which provides multiscale edge maps of natural images. By embedding the structure tensor-based exponent in a well-defined regularization model, we obtain denoising filters which are capable of obtaining good feature preserving image restoration. The GPU-based implementation computes the edge map in real time at 45-60 frames/s depending on the GPU card. Multiscale structure tensor-based spatially adaptive variable exponent provides reliable edge maps and compared with standard edge detectors it is robust under various noisy conditions. Moreover, filtering based on the multiscale variable exponent map method outperforms L0 sparse gradient-based image smoothing and related filters.

In this paper, we are proposing a 3D segmentation and interactive visualization workflow. The segmentation implementation uses a globally convex multiphase active contours without edges. This algorithm has been proven to be initialization independent due to their globally convex formulation and better than other approaches due to robustness to image variations and adaptive energy functionals. The workflow includes a flexible 3D visualization application that can handle very large volumes using multi-resolution hierarchical data formats following the segmentation. We also designed a custom fragment shader that is capable of meaningfully fusing the data from three different volumes: a segmented label volume, a mean value per voxel volume and a skull striped volume for effective visualization without modifying the segmented results. Giving researchers the access to a whole end to end pipeline, from 3D segmentation to custom real time interactive 3D visualization is, in our opinion, a powerful tool focused on an analyst/expert centric workflow.

A near-light perspective shape from shading (SfS) technique applied to endoscopy for 3D visualizations of the gastrointestinal tract regions is presented. By utilizing an extensible reflectance model, we study a robust Huber regularization function based variational SfS model. A balancing parameter is used for weighting the irradiance ad smoothness/regularization terms. Experimental results on different endoscopy systems show that we obtain 3D visualizations without shrinkage and dilation artifacts on mucosa tissues associated with other SfS models from the past.

In the field of histopathology, computer-assisted diagnosis systems are important in obtaining patient-specific diagnosis for various diseases and help define precision medicine. Therefore, many studies on automatic analysis methods for digital pathology images have been reported. One of the severe brain tumors is the Glioma can provide unique insights into identifying and grading disease stages. However, the number of tissue samples to be examined is enormous, and is a burden to pathologists because of the tedious manual evaluation required for efficient decision-making and diagnosis. Therefore, there is a strong demand for quick and automatic analysis to do that. In this study, we consider feature extraction and disease stage classification for Glioma images using automatic image analysis methods with deep learning techniques. By devising a custom made deep convolutional neural network (CNN) for disease stage classification we apply it on image data available on the cancer genome atlas for brain glioma in histopathology.

Segmenting microvascular structures is an important requirement in understanding angioadaptation by which vascular networks remodel their morphological structures. Accurate segmentation for separating microvasculature structures is important in quantifying remodeling process. In this work, we utilize a deep convolutional neural network (CNN) framework for obtaining robust segmentations of microvasculature from epifluorescence microscopy imagery of mice dura mater. Due to the inhomogeneous staining of the microvasculature, different binding properties of vessels under fluorescence dye, uneven contrast and low texture content, traditional vessel segmentation approaches obtain sub-optimal accuracy. We consider a deep CNN for the purpose keeping small vessel segments and handle the challenges posed by epifluorescence microscopy imaging modality. Experimental results on ovariectomized - ovary removed (OVX) - mice dura mater epifluorescence microscopy images show that the proposed modified CNN framework obtains an highest accuracy of 99% and better than other vessel segmentation methods.

Computer aided diagnosis (CAD) systems are important in obtaining precision medicine and patient driven solutions for various diseases. One of the main brain tumor is the Glioblastoma multiforme (GBM) and histopathological tissue images can provide unique insights into identifying and grading disease stages. In this work, we consider feature extraction and disease stage classification for brain tumor histopathological images using automatic image analysis methods. In particular we utilized automatic nuclei segmentation and labeling for histopathology image data obtained from The Cancer Genome Atlas (TCGA) and check for classification accuracy using support vector machine (SVM), Random Forests (RF). Our results indicate that we obtain classification accuracy 98.9% and 99.6% respectively.

Human epithelial (HEp-2) cell specimens are obtained from indirect immunofluorescence (IIF) imaging for diagnosis and management of autoimmune diseases. Analysis of HEp2 cells is important and in this work we consider automatic cell segmentation and classification using spatial and texture pattern features and random forest classifiers. In this paper, we summarize our efforts in classification and segmentation tasks proposed in ICPR 2016 contest. For the cell level staining pattern classification (Task 1), we utilized texture features such as rotational invariant co-occurrence (RIC) versions of the well-known local binary pattern (LBP), median binary pattern (MBP), joint adaptive median binary pattern (JAMBP), and motif labels (ML) along with other optimized features. We report the classification results utilizing different classifiers such as the k-nearest neighbors (kNN), support vector machine (SVM), and random forest (RF). We obtained the best accuracy of 94.26% for six cell classes with RIC-LBP combined with a motif pattern co-occurrence labels (MCL). For specimen level staining pattern classification (Task 2) we utilize a combination RIC-LBP with RF classifier and obtain 80% accuracy for seven classes. For cell segmentation (Task 4), we use our optimized multiscale spatial feature bank along with RF classifier for pixel-wise labeling to achieve an F-measure of 84.26% for 1008 images.

Computer aided diagnosis (CAD) systems are important in obtaining precision medicine and patient driven solutions for various diseases. One of the main brain tumor is the Glioblastoma multiforme (GBM) and histopathological tissue images can provide unique insights into identifying and grading disease stages. In this study, we consider nuclei segmentation method, feature extraction and disease stage classification for brain tumor histopathological images using automatic image analysis methods. In particular we utilized automatic nuclei segmentation and labeling for histopathology image data obtained from The Cancer Genome Atlas (TCGA) and check for significance of feature descriptors using K-S test and classification accuracy using support vector machine (SVM) and Random Forests (RF). Our results indicate that we obtain classification accuracy 98.6% and 99.8% in the case of Object-Level features and 82.1% and 86.1% in the case of Spatial Arrangement features, respectively.

This work discusses the performance of a color decomposition based active contours for segmenting cell nuclei from glioma histopathology. By combining a nuclear staining information obtained from color decomposition with fast variational active contours we obtain unsupervised segmentation of nuclei in histopathological images. Experimental results show promise when compared with different state of the art techniques.

Polyps in gastrointestinal (GI) tract can be precursors to cancer and detecting them early is important in determining their malignancy. Video capsule endoscopy imaging is a useful technology and provides visualization of GI without discomfort to the patient. We consider polyp localization from capsule endoscopy imagery using vascularization features. By using texture features computed from principle curvatures of the image surface, and multiscale directional vesselness stamping we obtain localization of polyps from a given video frame. We present our preliminary results for malignant and benign polyps from capsule endoscopy images.

This paper discusses the performance of feature descriptors for disease stage evaluation of Glioma images. In the field of histopathology, many evaluation methods for tissue images have been reported. However, pathologists have to analyze and evaluate many tissue images manually. In addition, the criteria of evaluation heavily depend on each pathologist's experience and feelings. From this background, studies on computational pathology using computer vision have been reported. The proposed feature descriptors were, however, applied to specified diseases only, and we do not know whether these descriptors will be effective to other tissues or not. This paper applied the feature descriptors defined by previous studies to the Glioma images and investigated the effectiveness of them by using a statistical method. We also discussed a method to distinguish low-grade from high-grade Glioma images by using the significant descriptors. After the experiments, more than 98% of Glioma images were classified correctly.

Edge preserving regularization using partial differential equation (PDE)-based methods although extensively studied and widely used for image restoration, still have limitations in adapting to local structures. We propose a spatially adaptive multiscale variable exponent-based anisotropic variational PDE method that overcomes current shortcomings, such as over smoothing and staircasing artifacts, while still retaining and enhancing edge structures across scale. Our innovative model automatically balances between Tikhonov and total variation (TV) regularization effects using scene content information by incorporating a spatially varying edge coherence exponent map constructed using the eigenvalues of the filtered structure tensor. The multiscale exponent model we develop leads to a novel restoration method that preserves edges better and provides selective denoising without generating artifacts for both additive and multiplicative noise models. Mathematical analysis of our proposed method in variable exponent space establishes the existence of a minimizer and its properties. The discretization method we use satisfies the maximum-minimum principle which guarantees that artificial edge regions are not created. Extensive experimental results using synthetic, and natural images indicate that the proposed multiscale Tikhonov-TV (MTTV) and dynamical MTTV methods perform better than many contemporary denoising algorithms in terms of several metrics, including signal-to-noise ratio improvement and structure preservation. Promising extensions to handle multiplicative noise models and multichannel imagery are also discussed.

Elastic body splines (EBS) belonging to the family of 3D splines were recently introduced to capture tissue deformations within a physical model-based approach for non-rigid biomedical image registration [1]. EBS model the displacement of points in a 3D homogeneous isotropic elastic body subject to forces. We propose a novel extension of using elastic body splines for learning driven figure-ground segmentation. The task of interactive image segmentation, with user provided foreground-background labeled seeds or samples, is formulated as learning an interpolating pixel classification function that is then used to assign labels for all unlabeled pixels in the image. The spline function we chose to model the supervised pixel classifier is the Gaussian elastic body spline (GEBS) which can use sparse scribbles from the user and has a closed form solution enabling a fast on-line implementation. Experimental results demonstrate the applicability of the GEBS approach for image segmentation. The GEBS method for interactive foreground image labeling shows promise and outperforms a previous approach using the thin-plate spline model.

Sparse representations have been advocated as a relevant advance in biometrics research. In this paper we propose a new algorithm for fusion at the data level of sparse representations, each one obtained from image patches. The main novelties are two-fold: 1) a dictionary fusion scheme is formalised, using the l1--- minimization with the gradient projection method; 2) the proposed representation and classification method does not require the non-overlapping condition of image patches from where individual dictionaries are obtained.

We study a multichannel version of nonlinear diffusion PDE which is used to restore noisy multispectral images. Weighted coupling of interchannel edges is done by utilizing fast total variation for each channel. Anisotropic intrachannel smoothing is included to denoise and preserve edges. Numerical results on noisy multispectral images show the advantage of the proposed hybrid approach.