3D segmentation of biological structures is critical in biomedical imaging, offering significant insights into structures and functions. This paper introduces a novel segmentation of biological images that couples Multi-Aperture representation with Transformers for 3D (MAT3D) segmentation. Our method integrates the global context-awareness of Transformer networks with the local feature extraction capabilities of Convolutional Neural Networks (CNNs), providing a comprehensive solution for accurately delineating complex biological structures. First, we evaluated the performance of the proposed technique on two public clinical datasets of ACDC and Synapse multi-organ segmentation, rendering superior Dice scores of 93.34±0.05 and 89.73±0.04, respectively, with fewer parameters compared to the published literature. Next, we assessed the performance of our technique on an organoid dataset comprising four breast cancer subtypes. The proposed method achieved a Dice 95.12±0.02 and a PQ score of 97.01±0.01, respectively. MAT3D also significantly reduces the parameters to 40 million. The code is available on https://github.com/sohaibcs1/MAT3D.

Automated 3D Radiological and Microscopy image analysis enables quantitative therapeutics assessment. However, accurate segmentation and smaller footprints are needed. To address these issues, the current study introduces the Multi-Aperture Transformer-Convolutional Network (MAFTC-Net) with a custom loss function incorporating principal curvatures to penalize topological dissimilarities. The multi-aperture modality represents each image patch as a series of images with smaller apertures along the same central field of view. Each aperture has a pyramid representation, with the advantage of maintaining the original image resolution. The MAFTC-Net encoder comprises four Swin Transformers, where a convolutional network processes the output of each transformer to accentuate local features within a global context. Next, the outputs of the transformer and convolution blocks are integrated with a 3D fusion block. MAFTC-Net was trained on the Synapse multi-organ, ACDC, and 3D Organoid datasets, achieving an average Dice score of 90.18±0.04, 93.55±0.05, and 95.13±0.03, respectively. The improved performance also comes with the added benefits of reduced complexity with 38.2 million parameters.

Profiling of Patient-Derived organoids is necessary for drug screening and precision medicine. This step requires accurate segmentation of three-dimensional cellular structures followed by protein readouts. While fully Convolutional Neural Networks are widely used in medical image segmentation, they struggle to capture long-range dependencies necessary for accurate segmentation. On the other hand, transformer models have shown promise in capturing long-range dependencies and self-similarities. Motivated by this, we present 3D-Organoid-SwinNet, a unique segmentation model explicitly designed for organoid semantic segmentation. We evaluated the performance of our technique using an Organoid dataset from four breast cancer subtypes. We demonstrated consistent top-tier performance in both the validation and testing phases, achieving a Dice score of 94.91 while reducing the number of parameters to 21 million. Our findings indicate that the proposed model offers a foundation for transformer-based models designed for high-content profiling of organoid models.

MOTIVATION: Organization of the organoid models, imaged in 3D with a confocal microscope, is an essential morphometric index to assess responses to stress or therapeutic targets. In fact, differentiating malignant and normal cells is often difficult in monolayer cultures. But in 3D culture, colony organization can provide a clear set of indices for differentiating malignant and normal cells. The limiting factors are delineating each cell in a 3D colony in the presence of perceptual boundaries between adjacent cells and heterogeneity associated with cells being at different cell cycles. RESULTS: In a previous paper, we defined a potential field for delineating adjacent nuclei, with perceptual boundaries, in 2D histology images by coupling three deep networks. This concept is now extended to 3D and simplified by an enhanced cost function that replaces three deep networks with one. Validation includes four cell lines with diverse mutations, and a comparative analysis with the UNet models of microscopy indicates an improved performance with the F1-score of 0.83. AVAILABILITY AND IMPLEMENTATION: All software and annotated images are available through GitHub and Bioinformatics online. The software includes the proposed method, UNet for microscopy that was extended to 3D and report generation for profiling colony organization. SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.

MOTIVATION: Our previous study has shown that ERBB2 is overexpressed in the organoid model of MCF10A when the stiffness of the microenvironment is increased to that of high mammographic density (MD). We now aim to identify key transcription factors (TFs) and functional enhancers that regulate processes associated with increased stiffness of the microenvironment in the organoid models of premalignant human mammary cell lines. RESULTS: 3D colony organizations and the cis-regulatory networks of two human mammary epithelial cell lines (184A1 and MCF10A) are investigated as a function of the increased stiffness of the microenvironment within the range of MD. The 3D colonies are imaged using confocal microscopy, and the morphometries of colony organizations and heterogeneity are quantified as a function of the stiffness of the microenvironment using BioSig3D. In a surrogate assay, colony organizations are profiled by transcriptomics. Transcriptome data are enriched by correlative analysis with the computed morphometric indices. Next, a subset of enriched data are processed against publicly available ChIP-Seq data using Model-based Analysis of Regulation of Gene Expression to predict regulatory transcription factors. This integrative analysis of morphometric and transcriptomic data predicted YY1 as one of the cis-regulators in both cell lines as a result of the increased stiffness of the microenvironment. Subsequent experiments validated that YY1 is expressed at protein and mRNA levels for MCF10A and 184A1, respectively. Also, there is a causal relationship between activation of YY1 and ERBB2 when YY1 is overexpressed at the protein level in MCF10A. SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.

MOTIVATION: Aberrant three-dimensional (3D) colony organization of premalignant human mammary epithelial cells (HMECs) is one of the indices of dysplasia. An experiment has been designed where the stiffness of the microenvironment, in 3D culture, has been set at either low or high level of mammographic density (MD) and the organoid models are exposed to 50 cGy X-ray radiation. This study utilizes published bioinformatics tools to quantify the frequency of aberrant colony formations by the combined stressors of stiffness and X-ray exposure. One of the goals is to develop a quantitative assay for evaluating the risk factors associated with women with high MD exposed to X-ray radiation. RESULTS: Analysis of 3D colony formations indicate that high stiffness, within the range of high MD, and X-ray radiation have an approximately additive effect on increasing the frequency of aberrant colony formations. Since both stiffness and X-ray radiation are DNA-damaging stressors, the additive effect of these stressors is also independently validated by profiling activin A-secreted protein. Secretion of activin A is known to be higher in tissues with a high MD as well as tumor cells. In addition, we show that increased stiffness of the microenvironment also induces phosphorylation of γH2AX-positive foci. The study uses two HMECs derived from a diseased tissue (e.g. MCF10A) and reduction mammoplasty of normal breast tissue (e.g. 184A1) to further demonstrate similar traits in the frequency of aberrant colony organization. SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.

MOTIVATION: Nuclear delineation and phenotypic profiling are important steps in the automated analysis of histology sections. However, these are challenging problems due to (i) technical variations (e.g. fixation, staining) that originate as a result of sample preparation; (ii) biological heterogeneity (e.g. vesicular versus high chromatin phenotypes, nuclear atypia) and (iii) overlapping nuclei. This Application-Note couples contextual information about the cellular organization with the individual signature of nuclei to improve performance. As a result, routine delineation of nuclei in H&E stained histology sections is enabled for either computer-aided pathology or integration with genome-wide molecular data. RESULTS: The method has been evaluated on two independent datasets. One dataset originates from our lab and includes H&E stained sections of brain and breast samples. The second dataset is publicly available through IEEE with a focus on gland-based tissue architecture. We report an approximate AJI of 0.592 and an F1-score 0.93 on both datasets. AVAILABILITY AND IMPLEMENTATION: The code-base, modified dataset and results are publicly available. SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.

The current trend in high throughput screening is the utilization of more complex model systems that mimic both structural and functional properties of cellular processes in vivo. In this context, 3D cell culture models have emerged as effective systems to study tumor initiation and cancer behavior, where colony organization represents distinct phenotypic signatures that enable differentiation of cancer cells in culture using phase imaging and in the absence of clinical markers. If the colony organization can be classified into different phenotypes, it will enable rapid drug screening using phase microscopy. In this paper, we propose a novel method based on locality-constrained dictionary learning for the discrimination of aberrant colony organization in phase images, which encodes original SIFT (Scale-Invariant Feature Transform) features into high dimensional sparse codes with locality-preserving landmark points on the nonlinear manifold, and summarizes the sparse features at various locations and scales through spatial pyramid matching for robust representation. Experimental results demonstrate the significant improvement of performance, compared to the state-of-art in the field.

Presents the introductory editorial for this issue of the publication.

When the amount of labeled data are limited, semisupervised learning can improve the learner's performance by also using the often easily available unlabeled data. In particular, a popular approach requires the learned function to be smooth on the underlying data manifold. By approximating this manifold as a weighted graph, such graph-based techniques can often achieve state-of-the-art performance. However, their high time and space complexities make them less attractive on large data sets. In this paper, we propose to scale up graph-based semisupervised learning using a set of sparse prototypes derived from the data. These prototypes serve as a small set of data representatives, which can be used to approximate the graph-based regularizer and to control model complexity. Consequently, both training and testing become much more efficient. Moreover, when the Gaussian kernel is used to define the graph affinity, a simple and principled method to select the prototypes can be obtained. Experiments on a number of real-world data sets demonstrate encouraging performance and scaling properties of the proposed approach. It also compares favorably with models learned via l1 -regularization at the same level of model sparsity. These results demonstrate the efficacy of the proposed approach in producing highly parsimonious and accurate models for semisupervised learning.

Image-based classification of histology sections plays an important role in predicting clinical outcomes. However this task is very challenging due to the presence of large technical variations (e.g., fixation, staining) and biological heterogeneities (e.g., cell type, cell state). In the field of biomedical imaging, for the purposes of visualization and/or quantification, different stains are typically used for different targets of interest (e.g., cellular/subcellular events), which generates multi-spectrum data (images) through various types of microscopes and, as a result, provides the possibility of learning biological-component-specific features by exploiting multispectral information. We propose a multispectral feature learning model that automatically learns a set of convolution filter banks from separate spectra to efficiently discover the intrinsic tissue morphometric signatures, based on convolutional sparse coding (CSC). The learned feature representations are then aggregated through the spatial pyramid matching framework (SPM) and finally classified using a linear SVM. The proposed system has been evaluated using two large-scale tumor cohorts, collected from The Cancer Genome Atlas (TCGA). Experimental results show that the proposed model 1) outperforms systems utilizing sparse coding for unsupervised feature learning (e.g., PSD-SPM [5]); 2) is competitive with systems built upon features with biological prior knowledge (e.g., SMLSPM [4]).

Image-based classification of tissue histology, in terms of different components (e.g., normal signature, categories of aberrant signatures), provides a series of indices for tumor composition. Subsequently, aggregation of these indices in each whole slide image (WSI) from a large cohort can provide predictive models of clinical outcome. However, the performance of the existing techniques is hindered as a result of large technical and biological variations that are always present in a large cohort. In this paper, we propose two algorithms for classification of tissue histology based on robust representations of morphometric context, which are built upon nuclear level morphometric features at various locations and scales within the spatial pyramid matching (SPM) framework. These methods have been evaluated on two distinct datasets of different tumor types collected from The Cancer Genome Atlas (TCGA), and the experimental results indicate that our methods are (i) extensible to different tumor types; (ii) robust in the presence of wide technical and biological variations; (iii) invariant to different nuclear segmentation strategies; and (iv) scalable with varying training sample size. In addition, our experiments suggest that enforcing sparsity, during the construction of morphometric context, further improves the performance of the system.

Image-based classification of tissue histology, in terms of distinct histopathology (e.g., tumor or necrosis regions), provides a series of indices for tumor composition. Furthermore, aggregation of these indices from each whole slide image (WSI) in a large cohort can provide predictive models of clinical outcome. However, the performance of the existing techniques is hindered as a result of large technical variations (e.g., fixation, staining) and biological heterogeneities (e.g., cell type, cell state) that are always present in a large cohort. We suggest that, compared with human engineered features widely adopted in existing systems, unsupervised feature learning is more tolerant to batch effect (e.g., technical variations associated with sample preparation) and pertinent features can be learned without user intervention. This leads to a novel approach for classification of tissue histology based on unsupervised feature learning and spatial pyramid matching (SPM), which utilize sparse tissue morphometric signatures at various locations and scales. This approach has been evaluated on two distinct datasets consisting of different tumor types collected from The Cancer Genome Atlas (TCGA), and the experimental results indicate that the proposed approach is (i) extensible to different tumor types; (ii) robust in the presence of wide technical variations and biological heterogeneities; and (iii) scalable with varying training sample sizes.

MOTIVATION: Our goal is to develop a screening platform for quantitative profiling of colony organizations in 3D cell culture models. The 3D cell culture models, which are also imaged in 3D, are functional assays that mimic the in vivo characteristics of the tissue architecture more faithfully than the 2D cultures. However, they also introduce significant computational challenges, with the main barriers being the effects of growth conditions, fixations and inherent complexities in segmentation that need to be resolved in the 3D volume. RESULTS: A segmentation strategy has been developed to delineate each nucleus in a colony that overcomes (i) the effects of growth conditions, (ii) variations in chromatin distribution and (iii) ambiguities formed by perceptual boundaries from adjacent nuclei. The strategy uses a cascade of geometric filters that are insensitive to spatial non-uniformity and partitions a clump of nuclei based on the grouping of points of maximum curvature at the interface of two neighboring nuclei. These points of maximum curvature are clustered together based on their coplanarity and proximity to define dissecting planes that separate the touching nuclei. The proposed curvature-based partitioning method is validated with both synthetic and real data, and is shown to have a superior performance against previous techniques. Validation and sensitivity analysis are coupled with the experimental design that includes a non-transformed cell line and three tumorigenic cell lines, which covers a wide range of phenotypic diversity in breast cancer. Colony profiling, derived from nuclear segmentation, reveals distinct indices for the morphogenesis of each cell line.

Automated analysis of whole mount tissue sections can provide insights into tumor subtypes and the underlying molecular basis of neoplasm. However, since tumor sections are collected from different laboratories, inherent technical and biological variations impede analysis for very large datasets such as The Cancer Genome Atlas (TCGA). Our objective is to characterize tumor histopathology, through the delineation of the nuclear regions, from hematoxylin and eosin (H&E) stained tissue sections. Such a representation can then be mined for intrinsic subtypes across a large dataset for prediction and molecular association. Furthermore, nuclear segmentation is formulated within a multi-reference graph framework with geodesic constraints, which enables computation of multidimensional representations, on a cell-by-cell basis, for functional enrichment and bioinformatics analysis. Here, we present a novel method, multi-reference graph cut (MRGC), for nuclear segmentation that overcomes technical variations associated with sample preparation by incorporating prior knowledge from manually annotated reference images and local image features. The proposed approach has been validated on manually annotated samples and then applied to a dataset of 377 Glioblastoma Multiforme (GBM) whole slide images from 146 patients. For the GBM cohort, multidimensional representation of the nuclear features and their organization have identified 1) statistically significant subtypes based on several morphometric indexes, 2) whether each subtype can be predictive or not, and 3) that the molecular correlates of predictive subtypes are consistent with the literature. Data and intermediaries for a number of tumor types (GBM, low grade glial, and kidney renal clear carcinoma) are available at: http://tcga.lbl.gov for correlation with TCGA molecular data. The website also provides an interface for panning and zooming of whole mount tissue sections with/without overlaid segmentation results for quality control.

BACKGROUND: Our goals are to develop a computational histopathology pipeline for characterizing tumor types that are being generated by The Cancer Genome Atlas (TCGA) for genomic association. TCGA is a national collaborative program where different tumor types are being collected, and each tumor is being characterized using a variety of genome-wide platforms. Here, we have developed a tumor-centric analytical pipeline to process tissue sections stained with hematoxylin and eosin (H&E) for visualization and cell-by-cell quantitative analysis. Thus far, analysis is limited to Glioblastoma Multiforme (GBM) and kidney renal clear cell carcinoma tissue sections. The final results are being distributed for subtyping and linking the histology sections to the genomic data. RESULTS: A computational pipeline has been designed to continuously update a local image database, with limited clinical information, from an NIH repository. Each image is partitioned into blocks, where each cell in the block is characterized through a multidimensional representation (e.g., nuclear size, cellularity). A subset of morphometric indices, representing potential underlying biological processes, can then be selected for subtyping and genomic association. Simultaneously, these subtypes can also be predictive of the outcome as a result of clinical treatments. Using the cellularity index and nuclear size, the computational pipeline has revealed five subtypes, and one subtype, corresponding to the extreme high cellularity, has shown to be a predictor of survival as a result of a more aggressive therapeutic regime. Further association of this subtype with the corresponding gene expression data has identified enrichment of (i) the immune response and AP-1 signaling pathways, and (ii) IFNG, TGFB1, PKC, Cytokine, and MAPK14 hubs. CONCLUSION: While subtyping is often performed with genome-wide molecular data, we have shown that it can also be applied to categorizing histology sections. Accordingly, we have identified a subtype that is a predictor of the outcome as a result of a therapeutic regime. Computed representation has become publicly available through our Web site.

In this paper, a novel approach is proposed for perceptual grouping and localization of ill-defined curvilinear structures. Our approach builds upon the tensor voting and the iterative voting frameworks. Its efficacy lies on iterative refinements of curvilinear structures by gradually shifting from an exploratory to an exploitative mode. Such a mode shifting is achieved by reducing the aperture of the tensor voting fields, which is shown to improve curve grouping and inference by enhancing the concentration of the votes over promising, salient structures. The proposed technique is validated on delineating adherens junctions that are imaged through fluorescence microscopy. However, the method is also applicable for screening other organisms based on characteristics of their cell wall structures. Adherens junctions maintain tissue structural integrity and cell-cell interactions. Visually, they exhibit fibrous patterns that may be diffused, heterogeneous in fluorescence intensity, or punctate and frequently perceptual. Besides the application to real data, the proposed method is compared to prior methods on synthetic and annotated real data, showing high precision rates.

MOTIVATION: Molecular association of phenotypic responses is an important step in hypothesis generation and for initiating design of new experiments. Current practices for associating gene expression data with multidimensional phenotypic data are typically (i) performed one-to-one, i.e. each gene is examined independently with a phenotypic index and (ii) tested with one stress condition at a time, i.e. different perturbations are analyzed separately. As a result, the complex coordination among the genes responsible for a phenotypic profile is potentially lost. More importantly, univariate analysis can potentially hide new insights into common mechanism of response. RESULTS: In this article, we propose a sparse, multitask regression model together with co-clustering analysis to explore the intrinsic grouping in associating the gene expression with phenotypic signatures. The global structure of association is captured by learning an intrinsic template that is shared among experimental conditions, with local perturbations introduced to integrate effects of therapeutic agents. We demonstrate the performance of our approach on both synthetic and experimental data. Synthetic data reveal that the multi-task regression has a superior reduction in the regression error when compared with traditional L(1)-and L(2)-regularized regression. On the other hand, experiments with cell cycle inhibitors over a panel of 14 breast cancer cell lines demonstrate the relevance of the computed molecular predictors with the cell cycle machinery, as well as the identification of hidden variables that are not captured by the baseline regression analysis. Accordingly, the system has identified CLCA2 as a hidden transcript and as a common mechanism of response for two therapeutic agents of CI-1040 and Iressa, which are currently in clinical use.

BACKGROUND: Methylation of CpG islands within the DNA promoter regions is one mechanism that leads to aberrant gene expression in cancer. In particular, the abnormal methylation of CpG islands may silence associated genes. Therefore, using high-throughput microarrays to measure CpG island methylation will lead to better understanding of tumor pathobiology and progression, while revealing potentially new biomarkers. We have examined a recently developed high-throughput technology for measuring genome-wide methylation patterns called mTACL. Here, we propose a computational pipeline for integrating gene expression and CpG island methylation profiles to identify epigenetically regulated genes for a panel of 45 breast cancer cell lines, which is widely used in the Integrative Cancer Biology Program (ICBP). The pipeline (i) reduces the dimensionality of the methylation data, (ii) associates the reduced methylation data with gene expression data, and (iii) ranks methylation-expression associations according to their epigenetic regulation. Dimensionality reduction is performed in two steps: (i) methylation sites are grouped across the genome to identify regions of interest, and (ii) methylation profiles are clustered within each region. Associations between the clustered methylation and the gene expression data sets generate candidate matches within a fixed neighborhood around each gene. Finally, the methylation-expression associations are ranked through a logistic regression, and their significance is quantified through permutation analysis. RESULTS: Our two-step dimensionality reduction compressed 90% of the original data, reducing 137,688 methylation sites to 14,505 clusters. Methylation-expression associations produced 18,312 correspondences, which were used to further analyze epigenetic regulation. Logistic regression was used to identify 58 genes from these correspondences that showed a statistically significant negative correlation between methylation profiles and gene expression in the panel of breast cancer cell lines. Subnetwork enrichment of these genes has identified 35 common regulators with 6 or more predicted markers. In addition to identifying epigenetically regulated genes, we show evidence of differentially expressed methylation patterns between the basal and luminal subtypes. CONCLUSIONS: Our results indicate that the proposed computational protocol is a viable platform for identifying epigenetically regulated genes. Our protocol has generated a list of predictors including COL1A2, TOP2A, TFF1, and VAV3, genes whose key roles in epigenetic regulation is documented in the literature. Subnetwork enrichment of these predicted markers further suggests that epigenetic regulation of individual genes occurs in a coordinated fashion and through common regulators.

Correlative analysis of molecular markers with phenotypic signatures is the simplest model for hypothesis generation. In this paper, a panel of 24 breast cell lines was grown in 3D culture, their morphology was imaged through phase contrast microscopy, and computational methods were developed to segment and represent each colony at multiple dimensions. Subsequently, subpopulations from these morphological responses were identified through consensus clustering to reveal three clusters of round, grape-like, and stellate phenotypes. In some cases, cell lines with particular pathobiological phenotypes clustered together (e.g., ERBB2 amplified cell lines sharing the same morphometric properties as the grape-like phenotype). Next, associations with molecular features were realized through (i) differential analysis within each morphological cluster, and (ii) regression analysis across the entire panel of cell lines. In both cases, the dominant genes that are predictive of the morphological signatures were identified. Specifically, PPARgamma has been associated with the invasive stellate morphological phenotype, which corresponds to triple-negative pathobiology. PPARgamma has been validated through two supporting biological assays.

Most tumors arise from epithelial tissues, such as mammary glands and lobules, and their initiation is associated with the disruption of a finely defined epithelial architecture. Progression from intraductal to invasive tumors is related to genetic mutations that occur at a subcellular level but manifest themselves as functional and morphological changes at the cellular and tissue scales, respectively. Elevated proliferation and loss of epithelial polarization are the two most noticeable changes in cell phenotypes during this process. As a result, many three-dimensional cultures of tumorigenic clones show highly aberrant morphologies when compared to regular epithelial monolayers enclosing the hollow lumen (acini). In order to shed light on phenotypic changes associated with tumor cells, we applied the bio-mechanical IBCell model of normal epithelial morphogenesis quantitatively matched to data acquired from the non-tumorigenic human mammary cell line, MCF10A. We then used a high-throughput simulation study to reveal how modifications in model parameters influence changes in the simulated architecture. Three parameters have been considered in our study, which define cell sensitivity to proliferative, apoptotic and cell-ECM adhesive cues. By mapping experimental morphologies of four MCF10A-derived cell lines carrying different oncogenic mutations onto the model parameter space, we identified changes in cellular processes potentially underlying structural modifications of these mutants. As a case study, we focused on MCF10A cells expressing an oncogenic mutant HER2-YVMA to quantitatively assess changes in cell doubling time, cell apoptotic rate, and cell sensitivity to ECM accumulation when compared to the parental non-tumorigenic cell line. By mapping in vitro mutant morphologies onto in silico ones we have generated a means of linking the morphological and molecular scales via computational modeling. Thus, IBCell in combination with 3D acini cultures can form a computational/experimental platform for suggesting the relationship between the histopathology of neoplastic lesions and their underlying molecular defects.

Cell membrane proteins play an important role in tissue architecture and cell-cell communication. We hypothesize that segmentation and multidimensional characterization of the distribution of cell membrane proteins, on a cell-by-cell basis, enable improved classification of treatment groups and identify important characteristics that can otherwise be hidden. We have developed a series of computational steps to 1) delineate cell membrane protein signals and associate them with a specific nucleus; 2) compute a coupled representation of the multiplexed DNA content with membrane proteins; 3) rank computed features associated with such a multidimensional representation; 4) visualize selected features for comparative evaluation through heatmaps; and 5) discriminate between treatment groups in an optimal fashion. The novelty of our method is in the segmentation of the membrane signal and the multidimensional representation of phenotypic signature on a cell-by-cell basis. To test the utility of this method, the proposed computational steps were applied to images of cells that have been irradiated with different radiation qualities in the presence and absence of other small molecules. These samples are labeled for their DNA content and E-cadherin membrane proteins. We demonstrate that multidimensional representations of cell-by-cell phenotypes improve predictive and visualization capabilities among different treatment groups, and identify hidden variables.

Practical data mining rarely falls exactly into the supervised learning scenario. Rather, the growing amount of unlabeled data poses a big challenge to large-scale semi-supervised learning (SSL). We note that the computational intensiveness of graph-based SSL arises largely from the manifold or graph regularization, which in turn lead to large models that are difficult to handle. To alleviate this, we proposed the prototype vector machine (PVM), a highly scalable, graph-based algorithm for large-scale SSL. Our key innovation is the use of "prototypes vectors" for efficient approximation on both the graph-based regularizer and model representation. The choice of prototypes are grounded upon two important criteria: they not only perform effective low-rank approximation of the kernel matrix, but also span a model suffering the minimum information loss compared with the complete model. We demonstrate encouraging performance and appealing scaling properties of the PVM on a number of machine learning benchmark data sets.

Bright field imaging of biological samples stained with antibodies and/or special stains provides a rapid protocol for visualizing various macromolecules. However, this method of sample staining and imaging is rarely employed for direct quantitative analysis due to variations in sample fixations, ambiguities introduced by color composition and the limited dynamic range of imaging instruments. We demonstrate that, through the decomposition of color signals, staining can be scored on a cell-by-cell basis. We have applied our method to fibroblasts grown from histologically normal breast tissue biopsies obtained from two distinct populations. Initially, nuclear regions are segmented through conversion of color images into gray scale, and detection of dark elliptic features. Subsequently, the strength of staining is quantified by a color decomposition model that is optimized by a graph cut algorithm. In rare cases where nuclear signal is significantly altered as a result of sample preparation, nuclear segmentation can be validated and corrected. Finally, segmented stained patterns are associated with each nuclear region following region-based tessellation. Compared to classical non-negative matrix factorization, proposed method: (i) improves color decomposition, (ii) has a better noise immunity, (iii) is more invariant to initial conditions and (iv) has a superior computing performance.

Color and texture have been widely used in image segmentation; however, their performance is often hindered by scene ambiguities, overlapping objects, or missing parts. In this paper, we propose an interactive image segmentation approach with shape prior models within a Bayesian framework. Interactive features, through mouse strokes, reduce ambiguities, and the incorporation of shape priors enhances quality of the segmentation where color and/or texture are not solely adequate. The novelties of our approach are in (i) formulating the segmentation problem in a well-defined Bayesian framework with multiple shape priors, (ii) efficiently estimating parameters of the Bayesian model, and (iii) multi-object segmentation through user-specified priors. We demonstrate the effectiveness of our method on a set of natural and synthetic images.

Many dynamical processes can be represented as directed attributed graphs or Petri nets where relationships between various entities are explicitly expressed. Signaling networks modeled as Petri nets are one class of such graphical modeling and representations. These networks encode how different protein in specific compartments, interact to create new protein products. Initially, the proteins and rules governing their interactions are curated from literature and then refined with experimental data. Variation in these networks occurs in topological structure, size, and weights associated on edges. Collectively, these variations are quite significant for manual and interactive analysis. Furthermore, as new information is added to these networks, the emergence of new computational models becomes paramount. From this perspective, hierarchical spectral methods are proposed and applied for inferring similarities and dissimilarities from an ensemble of graphs that corresponds to reaction networks. The technique has been implemented and tested on curated signaling networks that are derived for breast cancer cell lines

In this paper, we present a novel active contour model, in which the traditional gradient descent optimization is replaced by graph cut optimization. The basic idea is to first define an energy function according to curve evolution and then construct a graph with well selected edge weights based on the objective energy function, which is further optimized via graph cut algorithm. In this fashion, our model shares advantages of both level set method and graph cut algorithm, which are "topological" invariance, computational efficiency, and immunity to being stuck in the local minima. The model is validated on synthetic images, applied to two-class segmentation problem, and compared with the traditional active contour to demonstrate effectiveness of the technique. Finally, the method is applied to samples imaged with transmission electron microscopy that demonstrate complex textured patterns corresponding subcellular regions and micro-anatomy.

Saliency is an important perceptual cue that occurs at different levels of resolution. Important attributes of saliency are symmetry, continuity, and closure. Detection of these attributes is often hindered by noise, variation in scale, and incomplete information. This paper introduces the iterative voting method, which uses oriented kernels for inferring saliency as it relates to symmetry. A unique aspect of the technique is the kernel topography, which is refined and reoriented iteratively. The technique can cluster and group nonconvex perceptual circular symmetries along the radial line of an object's shape. It has an excellent noise immunity and is shown to be tolerant to perturbation in scale. The application of this technique to images obtained through various modes of microscopy is demonstrated. Furthermore, as a case example, the method has been applied to quantify kinetics of nuclear foci formation that are formed by phosphorylation of histone gammaH2AX following ionizing radiation. Iterative voting has been implemented in both 2-D and 3-D for multi image analysis.

A novel approach for reconstruction of sparse high-resolution data from lower-resolution dense spatio-temporal data is introduced. The basic idea is to compute the dense feature velocities from lower-resolution data and project them to the corresponding high-resolution data for computing the missing data. In this context, the basic flow equation is solved for intensity, as opposed to feature velocities at high resolution. Although the proposed technique is generic, we have applied our approach to sea surface temperature (SST) data at 18 km (low-resolution dense data) for computing the feature velocities and at 4 km (high-resolution sparse data) for interpolating the missing data. At low resolution, computation of the flow field is regularized and uses the incompressibility constraints for tracking fluid motion. At high resolution, computation of the intensity is regularized for continuity across multiple frames.

Organisms express their genomes in a cell-specific manner, resulting in a variety of cellular phenotypes or phenomes. Mapping cell phenomes under a variety of experimental conditions is necessary in order to understand the responses of organisms to stimuli. Representing such data requires an integrated view of experimental and informatic protocols. The proposed system, named BioSig, provides the foundation for cataloging cellular responses as a function of specific conditioning, treatment, staining, etc., for either fixed tissue or living cell studies. A data model has been developed to capture experimental variables and map them to image collections and their computed representation. This representation is hierarchical and spans across sample tissues, cells, and organelles, which are imaged with light microscopy. At each layer, content is represented with an attributed graph, which contains information about cellular morphology, protein localization, and cellular organization in tissue or cell culture. The Web-based multilayer informatics architecture uses the data model to provide guided workflow access for content exploration.

We have developed a "microscopy channel" to advertise a unique set of online scientific instruments and to let users join a particular session, perform an experiment, collaborate with other users, and collect data for further analysis. The channel is a collaborative problem solving environment (CPSE) that allows for both synchronous and asynchronous collaboration, as well as flow control for enhanced scalability. The flow control is a declarative feature that enhances software functionality at the experimental scale. Our testbed includes several unique electron and optical microscopes with applications ranging from material science to cell biology. We have built a system that leverages current commercial CORBA services, Web servers, and flow control specifications to meet diverse requirements for microscopy and experimental protocols. In this context, we have defined and enhanced instrument services (IS), exchange services (ES), computational services (CS), and declarative services (DS) that sit on top of CORBA and its enabling services (naming, trading, security, and notification). IS provides a layer of abstraction for controlling any type of microscope. ES provides a common set of utilities for information management and transaction. CS provides the analytical capabilities needed for online microscopy. DS provides mechanisms for flow control for improving the dynamic behavior of the system.

Mapping inter-cell signaling pathways requires an integrated view of experimental and informatic protocols. BioSig provides the foundation of cataloging inter-cell responses as a function of particular conditioning, treatment, staining, etc. for either in vivo or in vitro experiments. This paper outlines the system architecture, a functional data model for representing experimental protocols, algorithms for image analysis, and the required statistical analysis. The architecture provides remote shared operation of an inverted optical microscope, and couples instrument operation with images acquisition and annotation. The information is stored in an object-oriented database. The algorithms extract structural information such as morphology and organization, and map it to functional information such as inter-cellular responses. An example of usage of this system is included.

In this paper, we propose a new approach for interpolating curves (contour morphing) in time, which is a process of gradually changing a source curve (known) through intermediate curves (unknown) into a target curve (known). The novelty of our approach is in the deployment of a new regularization term and the corresponding Euler equation. Our method is applicable to implicit curve representation and it establishes a relationship between curve interpolation and a two dimensional function. This is achieved by minimizing the supremum of the gradient, which leads to the infinite Laplacian equation (ILE). ILE is optimal in the sense that interpolated curves are equally distributed along their normal direction. We point out that the existing distance field manipulation (DFM) methods are only an approximation to the proposed optimal solution and that the relationship between ILE and DFM is not local as it has been asserted before. The proposed interpolation can also be used to construct multiscale curve representation.

Our goal is to develop a feature based framework for data mining and forecasting from geophysical data fields. These data may be generated from either numerical simulation models or space based platforms. This paper focuses on pertinent features from sea surface temperature (SST) fields that are observed with the AVHRR satellite. Our contribution consist of three components: (1) A method for tracking feature velocities from from fluid motion with incompressibility constraint, (2) a method for localizing singular events such as vortices and saddle points from underlying feature velocities, and (3) application of our protocol to 12 years of high resolution real data to reveal novel seasonal and inter-annual trends based on computed events.

We propose an approach for surface recovery from planar sectional contours. The surface is reconstructed based on the so-called "equal importance criterion" which suggests that every point in the region contributes equally to the reconstruction process. The problem is then formulated in terms of a partial differential equation, and the solution is efficiently calculated from a distance transformation. To make the algorithm valid for different application purposes, both the isosurface and the primitive representations of the object surface are derived. The isosurface is constructed by a partial differential equation, which can be solved iteratively. The traditional distance interpolating method, which was used by several researchers for surface reconstruction, is an approximate solution of the PDE. The primitive representations are approximated by Voronoi diagram transformation of the surface space. Isosurfaces have the advantage that subsequent geometric analysis of the object can be easily carried out while primitive representation is easy to visualize. The proposed technique allows for surface recovery at any desired resolution, thus avoiding the inherent problems of correspondence, tiling, and branching.

We propose to detect singular-features in order to generate an intelligent summary of high resolution spatio-temporal data that are obtained from satellite-based observations of the ocean. Toward this objective, we extend the Horn-Schunck model of flow field computation to incorporate incompressibility for tracking fluid motion. This is expressed as a zero-divergence constraint in the variational problem and an efficient multigrid implementation of it is introduced. Additionally, we show an effective localization of event features, such as vortices and saddle points, in the velocity field that can be used for subsequent abstraction, query and statistical analysis.

A unique imaging modality based on equal thickness contours (ETC) has introduced a new opportunity for 3D shape reconstruction from multiple views. These ETC can be generated through an interference between transmitted and diffracted beams. We present a computational framework for representing each view of an object in terms of its object thickness and then integrating these representations into a 3D surface by algebraic reconstruction. In this framework, the object thickness is first derived from ideal contours and then extended to real data. For real data, the object thickness is inferred by grouping curve segments that correspond to points of second derivative maxima. At each step of the process, we use some form of regularization to ensure closeness to the original features as well as neighborhood continuity. We apply our approach to images of a submicron crystal structure obtained through a holographic process.

A new approach for segmentation of nuclei observed with an epi-fluorescence microscope is presented. The technique is model based and uses local feature activities such as step-edge segments, roof-edge segments, and concave corners to construct a set of initial hypotheses. These local feature activities are extracted using either local or global operators to form a possible set of hypotheses. Each hypothesis is expressed as a hyperquadric for better stability, compactness, and error handling. The search space is expressed as an assignment matrix with an appropriate cost function to ensure local adjacency, and global consistency. Each possible configuration of a set of nuclei defines a path, and the path with the least error corresponds to best representation. This result is then presented to an operator who verifies and eliminates a small number of errors.

Article DeepView: a channel for distributed microscopy and informatics Share on Authors: B. Parvin Information and Computing Sciences Division, Lawrence Berkeley National Laboratory, Berkeley, CA Information and Computing Sciences Division, Lawrence Berkeley National Laboratory, Berkeley, CAView Profile , J. Taylor Information and Computing Sciences Division, Lawrence Berkeley National Laboratory, Berkeley, CA Information and Computing Sciences Division, Lawrence Berkeley National Laboratory, Berkeley, CAView Profile , G. Cong Information and Computing Sciences Division, Lawrence Berkeley National Laboratory, Berkeley, CA Information and Computing Sciences Division, Lawrence Berkeley National Laboratory, Berkeley, CAView Profile , M. A. OKeefe Material Sciences Division, Lawrence Berkeley National Laboratory, Berkeley, CA Material Sciences Division, Lawrence Berkeley National Laboratory, Berkeley, CAView Profile , M. H. Barcellos-Hoff Life Sciences Division, Lawrence Berkeley National Laboratory, Berkeley, CA Life Sciences Division, Lawrence Berkeley National Laboratory, Berkeley, CAView Profile Authors Info & Claims SC '99: Proceedings of the 1999 ACM/IEEE conference on SupercomputingJanuary 1999 Pages 65–eshttps://doi.org/10.1145/331532.331597Online:01 January 1999Publication History 6citation214DownloadsMetricsTotal Citations6Total Downloads214Last 12 Months2Last 6 weeks0 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access

A unique imaging modality based on Equal Thickness Contours (ETC) has introduced a new opportunity for 3D shape reconstruction from multiple views. We present a computational framework for representing each view of an object in terms of its object thickness, and then integrating these representations into a 3D surface by algebraic reconstruction. The object thickness is inferred by grouping curve segments that correspond to points of second derivative maxima. At each step of the process, we use some form of regularization to ensure closeness to the original features, as well as neighborhood continuity. We apply our approach to images of a sub-micron crystal structure obtained through a holographic process.

A unique imaging modality based on equal thickness contours (ETC) has introduced a new opportunity for 3D shape reconstruction from multiple views. These ETCs are generated as a result of interference between transmitted and diffracted beams. We present a computational framework for representing each view of an object in terms of its object thickness, and then integrating these representations into a 3D surface by algebraic reconstruction. In this framework, the object thickness is first derived from ideal contours and then extended to real data. For real data, the object thickness is inferred by grouping curve segments that correspond to points of second derivative maxima. At each step of the process, we use some form of regularization to ensure closeness to the original features, as well as neighborhood continuity. We apply our approach to images of a sub-micron crystal structure obtained through a graphic process.

A curve evolution approach is proposed for corner enhancement. The constraint is that a new corner should not be generated. Starting from the general geometric heat flow (GGHF), we study under what conditions the GGHF satisfies the scale space causality criteria. The criteria of corner enhancement are also proposed. Then, a new curve evolution scheme which can enhance strong corners, suppress noise and satisfies the scale space criteria is presented.

This paper outlines the motivation, requirements, and architecture of a collaborative framework for distributed virtual microscopy. In this context, the requirements are specified in terms of (1) functionality, (2) scalability, (3) interactivity, and (4) safety and security. Functionality refers to what and how an instrument does something. Scalability refers to the number of instruments, vendor specific desktop workstations, analysis programs, and collaborators that can be accessed. Interactivity refers to how well the system can be steered either for static or dynamic experiments. Safety and security refers to safe operation of an instrument coupled with user authentication, privacy, and integrity of data communication. To meet these requirements, we introduce three types of services in the architecture: Instrument Services (IS), Exchange Services (ES), and Computational Services (CS). These services may reside on any host in the distributed system. The IS provide an abstraction for manipulating different types of microscopes; the ES provide common services that are required between different resources; and the CS provide analytical capabilities for data analysis and simulation. These services are brought together through CORBA and its enabling services, e.g., Event Services, Time Services, Naming Services, and Security Services. Two unique applications have been introduced into the CS for analyzing scientific images either for instrument control or recovery of a model for objects of interest. These include: insitu electron microscopy and recovery of 3D shape from holographic microscopy. The first application provides a near real-time processing of the video-stream for on-line quantitative analysis and the use of that information for closed-loop servo control. The second application reconstructs a 3D representation of an inclusion (a crystal structure in a matrix) from multiple views through holographic electron microscopy. These application require steering external stimuli or computational parameters for a particular result. In a sense, ``computational instruments'' (symmetric multiprocessors) interact closely with data generated from ``experimental instruments'' (unique microscopes) to conduct new experiments and bring new functionalities to these instruments. Both of these features exploit high-performance computing and low-latency networks to bring novel functionalities to unique scientific imaging instruments.

A new approach for reconstruction of 3D surfaces from 2D cross-sectional contours is presented. By using the so-called "Equal Importance Criterion," we reconstruct the surface based on the assumption that every point in the region contributes equally to the surface reconstruction process. In this context, the problem is formulated in terms of a partial differential equation (PDE), and we show that the solution for dense contours can be efficiently derived from distance transform. In the case of sparse contours, we add a regularization term to insure smoothness in surface recovery. The proposed technique allows for surface recovery at any desired resolution. The main advantage of the proposed method is that inherent problems due to correspondence, tiling, and branching are avoided. Furthermore, the computed high resolution surface is better represented for subsequent geometric analysis. We present results on both synthetic and real data.

A framework for automated micro-manipulation of microscopic objects using visual information is developed and applied to diverse problems in biology and material science. These applications include microdissection of stretched and immobilized DNA molecules using an optical microscope and dynamic studies of crystal structure formation observed with a high power transmission electron microscope. Manipulation in this context refers not only to mechanical positioning or dissection of a specimen, but also to the variation of environmental parameters that elicit a response in the specimen. The common thread between these applications is a collection of computational techniques for image compression, autofocusing, object detection, tracking, and servo loop control. A second feature of our system is a distributed software architecture that allows the proposed dynamic experiments to be conducted remotely over the wide area network. In this context, the visual information processing is used to compensate for dynamic changes and the latencies in the wide area network. This system is now available over the global Internet.

In this paper, we present a system for detection and tracking of tubular molecules in images. The automatic detection and characterization of the shape, location, and motion of these molecules can enable new laboratory protocols in several scientific disciplines. The uniqueness of the proposed system is twofold: At the macro level, the novelty of the system lies in the integration of object localization and tracking using geometric properties; at the micro level, in the use of high and low level constraints to model the detection and tracking subsystem. The underlying philosophy for object detection is to extract perceptually significant features from the pixel level image, and then use these high level cues to refine the precise boundaries. In the case of tubular molecules, the perceptually significant features are antiparallel line segments or, equivalently, their axis of symmetries. The axis of symmetry infers a coarse description of the object in terms of a bounding polygon. The polygon then provides the necessary boundary condition for the refinement process, which is based on dynamic programming. For tracking the object in a time sequence of images, the refined contour is then projected onto each consecutive frame.>

We present a system for detection, tracking and representation of tubular objects in images. The uniqueness of the proposed system is twofold: at the macro level, the novelty of the system lies in the integration of object localization and tracking using geometric properties; at the micro level, is the use of high and low level constraints to model the detection and tracking subsystem. The underlying philosophy for object detection is to extract perceptually significant features from the pixel level image, and then use these high level cues to refine the precise boundaries. In the case of tubular objects, the perceptually significant features are anti-parallel line segments or, equivalently, their axis of symmetries. The axis of symmetry infers a coarse description of the object in terms of a bounding polygon. The polygon then provides the necessary boundary condition for the refinement process, which is based on dynamic programming. For tracking the object in a time sequence of images, the refined contour is then projected onto each consecutive frame. In addition, the system provides an axis of symmetry representation of object for subsequent scientific analysis.>

Pinta is a system for segmentation and visualization of anatomical structures obtained from serial sections reconstructed from magnetic resonance imaging. The system approaches the segmentation problem by assigning each volumetric region to an anatomical structure. This is accomplished by satisfying constraints at the pixel level, slice level, and volumetric level. Each slice is represented by an attributed graph, where nodes correspond to regions and links correspond to the relations between regions. These regions are obtained by grouping pixels based on similarity and proximity. The slice level attributed graphs are then coerced to form a volumetric attributed graph, where volumetric consistency can be verified. The volumetric graph is used to ensure consistency from symbolic representations obtained from individual slices. The system allows errors to be made at the slice level, yet removes them when the volumetric consistency cannot be verified. Once the segmentation is complete, the 3D surfaces of the brain can be constructed and visualized.>

The authors present a method to produce an integrated description of an object given multiple range views without registration. The multiple view representation is in the form of a B-rep (boundary representation). They describe each view of the object as an attributed graph whose nodes are surface patches and links are the relations between surfaces. Any two attributed graphs, each corresponding to a given view, are matched, and the rigid motion transformation between them is computed. The basic strategy for multiple view integration is composed of two aspects: first, a composite graph which contains the bounding surfaces and their corresponding attributes is built and then these surfaces are intersected so that the edges and vertices corresponding to the B-rep description are identified. Results on objects with polyhedral as well as quadratic curved surfaces are presented.>

A unified framework for object description and correspondence in range images is proposed. This is based on modeling the ambiguities with a dynamic system at every level of hierarchy. The critical issue in object description is boundary completion. The authors show how this step can be modeled as the interaction of long and short term variables operating on partial contours, represented by parametric splines. In a sense, this network operates on an evolutionary basis where boundary completion is modeled as a synergistic process, and mutation is modeled as differential decay. Once reliable surface features are extracted, the same framework, with appropriate constraints, can find the corresponding surfaces and compute the transformation.>

A computational approach for solving the correspondence problem between different views of objects in range images is presented. This is modeled as a layered constraint satisfaction network which can be implemented on a parallel analog neural network. In this approach, each view of an object is represented by an attributed graph with nodes as surfaces and their bounding vertices, and links as relations between adjacent surfaces. The matching strategy is a two-step process. Each step is formulated with a constraint satisfaction network, and implemented on a Hopfield network. At each level, a set of local, adjacency and global constraints is specified, and an appropriate energy function to be minimized is defined. At the first level of this hierarchy, surface patches are matched and clusters of rotation transformations are hypothesized. At the second level, the computed rotation transformation is applied to the corresponding vertices, and the translation vector is computed.>