MOTIVATION: High-throughput time-lapse imaging is a fundamental tool for efficient living cell profiling at single-cell resolution. Label-free phase-contrast video microscopy enables noninvasive, nontoxic, and long-term imaging. The tradeoff between speed and throughput, however, implies that despite the state-of-the-art autofocusing algorithms, out-of-focus cells are unavoidable due to the migratory nature of immune cells (velocities >10 μm/min). Here, we propose PostFocus to (i) identify out-of-focus images within time-lapse sequences with a classifier, and (ii) deploy a de-noising diffusion probabilistic model to yield reliable in-focus images. RESULTS: De-noising diffusion probabilistic model outperformed deep discriminative models with a superior performance on the whole image and around cell boundaries. In addition, PostFocus improves the accuracy of image analysis (cell and contact detection) and the yield of usable videos. AVAILABILITY AND IMPLEMENTATION: Open-source code and sample data are available at: https://github.com/kwu14victor/PostFocus.

MOTIVATION: Reliable label-free methods are needed for detecting and profiling apoptotic events in time-lapse cell-cell interaction assays. Prior studies relied on fluorescent markers of apoptosis, e.g. Annexin-V, that provide an inconsistent and late indication of apoptotic onset for human melanoma cells. Our motivation is to improve the detection of apoptosis by directly detecting apoptotic bodies in a label-free manner. RESULTS: Our trained ResNet50 network identified nanowells containing apoptotic bodies with 92% accuracy and predicted the onset of apoptosis with an error of one frame (5 min/frame). Our apoptotic body segmentation yielded an IoU accuracy of 75%, allowing associative identification of apoptotic cells. Our method detected apoptosis events, 70% of which were not detected by Annexin-V staining. AVAILABILITY AND IMPLEMENTATION: Open-source code and sample data provided at https://github.com/kwu14victor/ApoBDproject.

Automatic and accurate classification of apoptosis, or programmed cell death, will facilitate cell biology research. The state-of-the-art approaches in apoptosis classification use deep convolutional neural networks (CNNs). However, these networks are not efficient in encoding the part-whole relationships, thus requiring a large number of training samples to achieve robust generalization. This paper proposes an efficient variant of capsule networks (CapsNets) as an alternative to CNNs. Extensive experimental results demonstrate that the proposed CapsNets achieve competitive performances in target cell apoptosis classification, while significantly outperforming CNNs when the number of training samples is small. To utilize temporal information within microscopy videos, we propose a recurrent CapsNet constructed by stacking a CapsNet and a bi-directional long short-term recurrent structure. Our experiments show that when considering temporal constraints, the recurrent CapsNet achieves 93.8% accuracy and makes significantly more consistent prediction than NNs.

MOTIVATION: Automated profiling of cell-cell interactions from high-throughput time-lapse imaging microscopy data of cells in nanowell grids (TIMING) has led to fundamental insights into cell-cell interactions in immunotherapy. This application note aims to enable widespread adoption of TIMING by (i) enabling the computations to occur on a desktop computer with a graphical processing unit instead of a server; (ii) enabling image acquisition and analysis to occur in the laboratory avoiding network data transfers to/from a server and (iii) providing a comprehensive graphical user interface. RESULTS: On a desktop computer, TIMING 2.0 takes 5 s/block/image frame, four times faster than our previous method on the same computer, and twice as fast as our previous method (TIMING) running on a Dell PowerEdge server. The cell segmentation accuracy (f-number = 0.993) is superior to our previous method (f-number = 0.821). A graphical user interface provides the ability to inspect the video analysis results, make corrective edits efficiently (one-click editing of an entire nanowell video sequence in 5-10 s) and display a summary of the cell killing efficacy measurements. AVAILABILITY AND IMPLEMENTATION: Open source Python software (GPL v3 license), instruction manual, sample data and sample results are included with the Supplement (https://github.com/RoysamLab/TIMING2). SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.

MOTIVATION: Current spectral unmixing methods for multiplex fluorescence microscopy have a limited ability to cope with high spectral overlap as they only analyze spectral information over individual pixels. Here, we present adaptive Morphologically Constrained Spectral Unmixing (MCSU) algorithms that overcome this limitation by exploiting morphological differences between sub-cellular structures, and their local spatial context. RESULTS: The proposed method was effective at improving spectral unmixing performance by exploiting: (i) a set of dictionary-based models for object morphologies learned from the image data; and (ii) models of spatial context learned from the image data using a total variation algorithm. The method was evaluated on multi-spectral images of multiplex-labeled pancreatic ductal adenocarcinoma (PDAC) tissue samples. The former constraint ensures that neighbouring pixels correspond to morphologically similar structures, and the latter constraint ensures that neighbouring pixels have similar spectral signatures. The average Mean Squared Error (MSE) and Signal Reconstruction Error (SRE) ratio of the proposed method was 39.6% less and 8% more, respectively, compared to the best of all other algorithms that do not exploit these spatial constraints. AVAILABILITY AND IMPLEMENTATION: Open source software (MATLAB). CONTACT: [email protected]. SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.

In this letter, a no-reference (NR) hybrid image quality assessment (IQA) metric based on cartoon-texture decomposition (CTD) is presented. Focusing on images distorted by both blur and noise, the method takes properties of CTD to separate image into cartoon part with salient edges and texture part with noises. Then, the blur degree and noise level can be estimated separately from different image parts, combine with a joint effect prediction between blur and noise distortions, we present a cartoon-texture decomposition-based blind metric (CTDBBM). Comparative studies with classical full-reference IQA metrics and state-of-the-art NR metrics are conducted on multidistortion image database: LIVEMD. Experimental results show that the CTDBBM performs well and has a high consistency with the human opinions given in the database.

MOTIVATION: The arbor morphologies of brain microglia are important indicators of cell activation. This article fills the need for accurate, robust, adaptive and scalable methods for reconstructing 3-D microglial arbors and quantitatively mapping microglia activation states over extended brain tissue regions. RESULTS: Thick rat brain sections (100-300 µm) were multiplex immunolabeled for IBA1 and Hoechst, and imaged by step-and-image confocal microscopy with automated 3-D image mosaicing, producing seamless images of extended brain regions (e.g. 5903 × 9874 × 229 voxels). An over-complete dictionary-based model was learned for the image-specific local structure of microglial processes. The microglial arbors were reconstructed seamlessly using an automated and scalable algorithm that exploits microglia-specific constraints. This method detected 80.1 and 92.8% more centered arbor points, and 53.5 and 55.5% fewer spurious points than existing vesselness and LoG-based methods, respectively, and the traces were 13.1 and 15.5% more accurate based on the DIADEM metric. The arbor morphologies were quantified using Scorcioni's L-measure. Coifman's harmonic co-clustering revealed four morphologically distinct classes that concord with known microglia activation patterns. This enabled us to map spatial distributions of microglial activation and cell abundances. AVAILABILITY AND IMPLEMENTATION: Experimental protocols, sample datasets, scalable open-source multi-threaded software implementation (C++, MATLAB) in the electronic supplement, and website (www.farsight-toolkit.org). http://www.farsight-toolkit.org/wiki/Population-scale_Three-dimensional_Reconstruction_and_Quanti-tative_Profiling_of_Microglia_Arbors CONTACT: [email protected] SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.

MOTIVATION: There is a need for effective automated methods for profiling dynamic cell-cell interactions with single-cell resolution from high-throughput time-lapse imaging data, especially, the interactions between immune effector cells and tumor cells in adoptive immunotherapy. RESULTS: Fluorescently labeled human T cells, natural killer cells (NK), and various target cells (NALM6, K562, EL4) were co-incubated on polydimethylsiloxane arrays of sub-nanoliter wells (nanowells), and imaged using multi-channel time-lapse microscopy. The proposed cell segmentation and tracking algorithms account for cell variability and exploit the nanowell confinement property to increase the yield of correctly analyzed nanowells from 45% (existing algorithms) to 98% for wells containing one effector and a single target, enabling automated quantification of cell locations, morphologies, movements, interactions, and deaths without the need for manual proofreading. Automated analysis of recordings from 12 different experiments demonstrated automated nanowell delineation accuracy >99%, automated cell segmentation accuracy >95%, and automated cell tracking accuracy of 90%, with default parameters, despite variations in illumination, staining, imaging noise, cell morphology, and cell clustering. An example analysis revealed that NK cells efficiently discriminate between live and dead targets by altering the duration of conjugation. The data also demonstrated that cytotoxic cells display higher motility than non-killers, both before and during contact. CONTACT: [email protected] or [email protected] SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.

This paper presents unsupervised algorithms for discovering previously unknown subspace trends in high-dimensional data sets without the benefit of prior information. A subspace trend is a sustained pattern of gradual/progressive changes within an unknown subset of feature dimensions. A fundamental challenge to subspace trend discovery is the presence of irrelevant data dimensions, noise, outliers, and confusion from multiple subspace trends driven by independent factors that are mixed in with each other. These factors can obscure the trends in conventional dimension reduction & projection based data visualizations. To overcome these limitations, we propose a novel graph-theoretic neighborhood similarity measure for detecting concordant progressive changes across data dimensions. Using this measure, we present an unsupervised algorithm for trend-relevant feature selection, subspace trend discovery, quantification of trend strength, and validation. Our method successfully identified verifiable subspace trends in diverse synthetic and real-world biomedical datasets. Visualizations derived from the selected trend-relevant features revealed biologically meaningful hidden subspace trend(s) that were obscured by irrelevant features and noise. Although our examples are drawn from the biological domain, the proposed algorithm is broadly applicable to exploratory analysis of high-dimensional data including visualization, hypothesis generation, knowledge discovery, and prediction in diverse other applications.

We briefly identify several critical issues in current computational neuroscience, and present our opinions on potential solutions based on bioimage informatics, especially automated image computing.

One of the major goals in biomedical image processing is accurate segmentation of networks embedded in volumetric data sets. Biological networks are composed of a meshwork of thin filaments that span large volumes of tissue. Examples of these structures include neurons and microvasculature, which can take the form of both hierarchical trees and fully connected networks, depending on the imaging modality and resolution. Network function depends on both the geometric structure and connectivity. Therefore, there is considerable demand for algorithms that segment biological networks embedded in three-dimensional data. While a large number of tracking and segmentation algorithms have been published, most of these do not generalize well across data sets. One of the major reasons for the lack of general-purpose algorithms is the limited availability of metrics that can be used to quantitatively compare their effectiveness against a pre-constructed ground-truth. In this paper, we propose a robust metric for measuring and visualizing the differences between network models. Our algorithm takes into account both geometry and connectivity to measure network similarity. These metrics are then mapped back onto an explicit model for visualization.

This paper presents an algorithm for the extraction of interest points in hyperspectral images. Interest points are spatial features of the image that capture information from their neighbors, are distinctive and stable under transformations such as translation and rotation, are helpful in data reduction, and reduce the computational burden of various algorithms such as image registration by replacing an exhaustive search over the entire image domain by a probe into a concise set of highly informative points. Interest points have been applied to problems in computer vision, including image matching, recognition, 3-D reconstruction, and change detection. Interest point operators for monochromatic images were proposed more than a decade ago and have extensively been studied. An interest point operator seeks out points in an image that are structurally distinct, invariant to imaging conditions, and stable under geometric transformations. An extension of Lowe's scale-invariant feature transform (SIFT) to vector images is proposed here. The approach takes the vectorial nature of the hyperspectral images into account. Furthermore, the multiscale representation of the image is generated by vector nonlinear diffusion, which leads to improved detection, because it better preserves edges in the image as opposed to Gaussian blurring, which is used in Lowe's original approach. Experiments with hyperspectral images of the same and different resolutions that were collected with the Airborne Hyperspectral Imaging System (AISA) and Hyperion sensors are presented. Evaluation of the proposed approach using repeatability criterion and image registration is carried out. Comparisons with other approaches that were described in the literature are presented.

We present novel approaches for fully automated extraction of tree-like tubular structures from 3-D image stacks. A 4-D Open-Curve Active Contour (Snake) model is proposed for simultaneous 3-D centerline tracing and local radius estimation. An image energy term, stretching term, and a novel region-based radial energy term constitute the energy to be minimized. This combination of energy terms allows the 4-D open-curve snake model, starting from an automatically detected seed point, to stretch along and fit the tubular structures like neurites and blood vessels. A graph-based curve completion approach is proposed to merge possible fragments caused by discontinuities in the tree structures. After tree structure extraction, the centerlines serve as the starting points for a Fast Marching segmentation for which the stopping time is automatically chosen. We illustrate the performance of our method with various datasets.

This paper presents robust 3-D algorithms to segment vasculature that is imaged by labeling laminae, rather than the lumenal volume. The signal is weak, sparse, noisy, nonuniform, low-contrast, and exhibits gaps and spectral artifacts, so adaptive thresholding and Hessian filtering based methods are not effective. The structure deviates from a tubular geometry, so tracing algorithms are not effective. We propose a four step approach. The first step detects candidate voxels using a robust hypothesis test based on a model that assumes Poisson noise and locally planar geometry. The second step performs an adaptive region growth to extract weakly labeled and fine vessels while rejecting spectral artifacts. To enable interactive visualization and estimation of features such as statistical confidence, local curvature, local thickness, and local normal, we perform the third step. In the third step, we construct an accurate mesh representation using marching tetrahedra, volume-preserving smoothing, and adaptive decimation algorithms. To enable topological analysis and efficient validation, we describe a method to estimate vessel centerlines using a ray casting and vote accumulation algorithm which forms the final step of our algorithm. Our algorithm lends itself to parallel processing, and yielded an 8 x speedup on a graphics processor (GPU). On synthetic data, our meshes had average error per face (EPF) values of (0.1-1.6) voxels per mesh face for peak signal-to-noise ratios from (110-28 dB). Separately, the error from decimating the mesh to less than 1% of its original size, the EPF was less than 1 voxel/face. When validated on real datasets, the average recall and precision values were found to be 94.66% and 94.84%, respectively.

An algorithmic information-theoretic method is presented for object-level summarization of meaningful changes in image sequences. Object extraction and tracking data are represented as an attributed tracking graph (ATG). Time courses of object states are compared using an adaptive information distance measure, aided by a closed-form multidimensional quantization. The notion of meaningful summarization is captured by using the gap statistic to estimate the randomness deficiency from algorithmic statistics. The summary is the clustering result and feature subset that maximize the gap statistic. This approach was validated on four bioimaging applications: 1) It was applied to a synthetic data set containing two populations of cells differing in the rate of growth, for which it correctly identified the two populations and the single feature out of 23 that separated them; 2) it was applied to 59 movies of three types of neuroprosthetic devices being inserted in the brain tissue at three speeds each, for which it correctly identified insertion speed as the primary factor affecting tissue strain; 3) when applied to movies of cultured neural progenitor cells, it correctly distinguished neurons from progenitors without requiring the use of a fixative stain; and 4) when analyzing intracellular molecular transport in cultured neurons undergoing axon specification, it automatically confirmed the role of kinesins in axon specification.

Interest points are widely used as point-features for image matching. This paper describes robust and efficient algorithms to extract multiscale interest points in hyperspectral images in which structural information is distributed across several spectral bands. The formulation is based on a Gaussian scale-space representation of the hyperspectral data cube, and the use of a principal components decomposition to combine information efficiently across spectral bands. A spectral distance measure is used to characterize spatial relations between neighboring hyperspectral pixels. In addition, we describe methods for preprocessing a pair of hyperspectral images, clustering the spectral signatures of interest points, and using the resulting data for matching points under simple geometric transformations. The stability of the resulting interest points in time-lapse satellite images was determined to be in the range of 52% to 75% in the testing data set that were acquired from variety of landforms like coastal islands of La Parguera, Chesapeake Bay, the Cuprite Mining District of Nevada, and agricultural field images of Kansas and Oklahoma, and thus, they can be used as a foundation for image matching and related image analysis tasks.

This updates an earlier publication by the authors describing a robust framework for detecting vasculature in noisy retinal fundus images. We improved the handling of the "central reflex" phenomenon in which a vessel has a "hollow" appearance. This is particularly pronounced in dual-wavelength images acquired at 570 and 600 nm for retinal oximetry. It is prominent in the 600 nm images that are sensitive to the blood oxygen content. Improved segmentation of these vessels is needed to improve oximetry. We show that the use of a generalized dual-Gaussian model for the vessel intensity profile instead of the Gaussian yields a significant improvement. Our method can account for variations in the strength of the central reflex, the relative contrast, width, orientation, scale, and imaging noise. It also enables the classification of regular and central reflex vessels. The proposed method yielded a sensitivity of 72% compared to 38% by the algorithm of Can et al., and 60% by the robust detection based on a single-Gaussian model. The specificity for the methods were 95%, 97%, and 98%, respectively.

Retinal clinicians and researchers make extensive use of images, and the current emphasis is on digital imaging of the retinal fundus. The goal of this paper is to introduce a system, known as retinal image vessel extraction and registration system, which provides the community of retinal clinicians, researchers, and study directors an integrated suite of advanced digital retinal image analysis tools over the Internet. The capabilities include vasculature tracing and morphometry, joint (simultaneous) montaging of multiple retinal fields, cross-modality registration (color/red-free fundus photographs and fluorescein angiograms), and generation of flicker animations for visualization of changes from longitudinal image sequences. Each capability has been carefully validated in our previous research work. The integrated Internet-based system can enable significant advances in retina-related clinical diagnosis, visualization of the complete fundus at full resolution from multiple low-angle views, analysis of longitudinal changes, research on the retinal vasculature, and objective, quantitative computer-assisted scoring of clinical trials imagery. It could pave the way for future screening services from optometry facilities.

This paper presents methods to model complex vasculature in three-dimensional (3-D) images using cylindroidal superellipsoids, along with robust estimation and detection algorithms for automated image analysis. This model offers an explicit, low-order parameterization, enabling joint estimation of boundary, centerlines, and local pose. It provides a geometric framework for directed vessel traversal, and extraction of topological information like branch point locations and connectivity. M-estimators provide robust region-based statistics that are used to drive the superellipsoid toward a vessel boundary. A robust likelihood ratio test is used to differentiate between noise, artifacts, and other complex unmodeled structures, thereby verifying the model estimate. The proposed methodology behaves well across scale-space, shows a high degree of insensitivity to adjacent structures and implicitly handles branching. When evaluated on synthetic imagery mimicking specific structural complexities in tumor microvasculature, it consistently produces ubvoxel accuracy estimates of centerlines and widths in the presence of closely-adjacent vessels, branch points, and noise. An edit-based validation demonstrated a precision level of 96.6% at a recall level of 95.4%. Overall, it is robust enough for large-scale application.

An automated method is presented for selecting optimal parameter settings for vessel/neurite segmentation algorithms using the minimum description length principle and a recursive random search algorithm. It trades off a probabilistic measure of image-content coverage against its conciseness. It enables nonexpert users to select parameter settings objectively, without knowledge of underlying algorithms, broadening the applicability of the segmentation algorithm, and delivering higher morphometric accuracy. It enables adaptation of parameters across batches of images. It simplifies the user interface to just one optional parameter and reduces the cost of technical support. Finally, the method is modular, extensible, and amenable to parallel computation. The method is applied to 223 images of human retinas and cultured neurons, from four different sources, using a single segmentation algorithm with eight parameters. Improvements in segmentation quality compared to default settings using 1000 iterations ranged from 4.7%-21%. Paired t-tests showed that improvements are statistically significant (p < 0.0005). Most of the improvement occurred in the first 44 iterations. Improvements in description lengths and agreement with the ground truth were strongly correlated (p = 0.78).

Detecting regions of change in multiple images of the same scene taken at different times is of widespread interest due to a large number of applications in diverse disciplines, including remote sensing, surveillance, medical diagnosis and treatment, civil infrastructure, and underwater sensing. This paper presents a systematic survey of the common processing steps and core decision rules in modern change detection algorithms, including significance and hypothesis testing, predictive models, the shading model, and background modeling. We also discuss important preprocessing methods, approaches to enforcing the consistency of the change mask, and principles for evaluating and comparing the performance of change detection algorithms. It is hoped that our classification of algorithms into a relatively small number of categories will provide useful guidance to the algorithm designer.

This paper presents a set of algorithms for robust detection of vasculature in noisy retinal video images. Three methods are studied for effective handling of outliers. The first method is based on Huber's censored likelihood ratio test. The second is based on the use of a alpha-trimmed test statistic. The third is based on robust model selection algorithms. All of these algorithms rely on a mathematical model for the vasculature that accounts for the expected variations in intensity/texture profile, width, orientation, scale, and imaging noise. These unknown parameters are estimated implicitly within a robust detection and estimation framework. The proposed algorithms are also useful as nonlinear vessel enhancement filters. The proposed algorithms were evaluated over carefully constructed phantom images, where the ground truth is known a priori, as well as clinically recorded images for which the ground truth was manually compiled. A comparative evaluation of the proposed approaches is presented. Collectively, these methods outperformed prior approaches based on Chaudhuri et al. (1989) matched filtering, as well as the verification methods used by prior exploratory tracing algorithms, such as the work of Can et aL (1999). The Huber censored likelihood test yielded the best overall improvement, with a 145.7% improvement over the exploratory tracing algorithm, and a 43.7% improvement in detection rates over the matched filter.

A model-based algorithm, termed exclusion region and position refinement (ERPR), is presented for improving the accuracy and repeatability of estimating the locations where vascular structures branch and cross over, in the context of human retinal images. The goal is two fold. First, accurate morphometry of branching and crossover points (landmarks) in neuronal/vascular structure is important to several areas of biology and medicine. Second, these points are valuable as landmarks for image registration, so improved accuracy and repeatability in estimating their locations and signatures leads to more reliable image registration for applications such as change detection and mosaicing. The ERPR algorithm is shown to reduce the median location error from 2.04 pixels down to 1.1 pixels, while improving the median spread (a measure of repeatability) from 2.09 pixels down to 1.05 pixels. Errors in estimating vessel orientations were similarly reduced from 7.2 degrees down to 3.8 degrees.

This paper addresses the problem of migrating large and complex computer vision code bases that have been developed off-line, into efficient real-time implementations avoiding the need for rewriting the software, and the associated costs. Creative linking strategies based on Linux loadable kernel modules are presented to create a simultaneous realization of real-time and off-line frame rate computer vision systems from a single code base. In this approach, systemic predictability is achieved by inserting time-critical components of a user-level executable directly into the kernel as a virtual device driver. This effectively emulates a single process space model that is nonpreemptable, nonpageable, and that has direct access to a powerful set of system-level services. This overall approach is shown to provide the basis for building a predictable frame-rate vision system using commercial off-the-shelf hardware and a standard uniprocessor Linux operating system. Experiments on a frame-rate vision system designed for computer-assisted laser retinal surgery show that this method reduces the variance of observed per-frame central processing unit cycle counts by two orders of magnitude. The conclusion is that when predictable application algorithms are used, it is possible to efficiently migrate to a predictable frame-rate computer vision system.

This paper describes an algorithm to continually and accurately estimate the absolute location of a diagnostic or surgical tool (such as a laser) pointed at the human retina, from a series of image frames. We treat the problem as a registration problem using diagnostic images to build a spatial map of the retina and then registering each online image against this map. Since the image location where the laser strikes the retina is easily found, this registration determines the position of the laser in the global coordinate system defined by the spatial map. For each online image, the algorithm computes similarity invariants, locally valid despite the curved nature of the retina, from constellations of vascular landmarks. These are detected using a high-speed algorithm that iteratively traces the blood vessel structure. Invariant indexing establishes initial correspondences between landmarks from the online image and landmarks stored in the spatial map. Robust alignment and verification steps extend the similarity transformation computed from these initial correspondences to a global, high-order transformation. In initial experimentation, the method has achieved 100 percent success on 1024 /spl times/ 1024 retina images. With a version of the tracing algorithm optimized for speed on 512 /spl times/ 512 images, the computation time is only 51 milliseconds per image on a 900MHz PentiumIII processor and a 97 percent success rate is achieved. The median registration error in either case is about 1 pixel.

This paper presents a method to exploit rank statistics to improve fully automatic tracing of neurons from noisy digital confocal microscope images. Previously proposed exploratory tracing (vectorization) algorithms work by recursively following the neuronal topology, guided by responses of multiple directional correlation kernels. These algorithms were found to fail when the data was of lower quality (noisier, less contrast, weak signal, or more discontinuous structures). This type of data is commonly encountered in the study of neuronal growth on microfabricated surfaces. We show that by partitioning the correlation kernels in the tracing algorithm into multiple subkernels, and using the median of their responses as the guiding criterion improves the tracing precision from 41% to 89% for low-quality data, with a 5% improvement in recall. Improved handling was observed for artifacts such as discontinuities and/or hollowness of structures. The new algorithms require slightly higher amounts of computation, but are still acceptably fast, typically consuming less than 2 seconds on a personal computer (Pentium III, 500 MHz, 128 MB). They produce labeling for all somas present in the field, and a graph-theoretic representation of all dendritic/axonal structures that can be edited. Topological and size measurements such as area, length, and tortuosity are derived readily. The efficiency, accuracy, and fully-automated nature of the proposed method makes it attractive for large-scale applications such as high-throughput assays in the pharmaceutical industry, and study of neuron growth on nano/micro-fabricated structures. A careful quantitative validation of the proposed algorithms is provided against manually derived tracing, using a performance measure that combines the precision and recall metrics.

Motivated by the problem of retinal image registration, this paper introduces and analyzes a new registration algorithm called Dual-Bootstrap Iterative Closest Point (Dual-Bootstrap ICP). The approach is to start from one or more initial, low-order estimates that are only accurate in small image regions, called bootstrap regions. In each bootstrap region, the algorithm iteratively: 1) refines the transformation estimate using constraints only from within the bootstrap region; 2) expands the bootstrap region; and 3) tests to see if a higher order transformation model can be used, stopping when the region expands to cover the overlap between images. Steps 1): and 3), the bootstrap steps, are governed by the covariance matrix of the estimated transformation. Estimation refinement [Step 2)] uses a novel robust version of the ICP algorithm. In registering retinal image pairs, Dual-Bootstrap ICP is initialized by automatically matching individual vascular landmarks, and it aligns images based on detected blood vessel centerlines. The resulting quadratic transformations are accurate to less than a pixel. On tests involving approximately 6000 image pairs, it successfully registered 99.5% of the pairs containing at least one common landmark, and 100% of the pairs containing at least one common landmark and at least 35% image overlap.

This paper describes a robust hierarchical algorithm for fully-automatic registration of a pair of images of the curved human retina photographed by a fundus microscope. Accurate registration is essential for mosaic synthesis, change detection, and design of computer-aided instrumentation. Central to the algorithm is a 12-parameter interimage transformation derived by modeling the retina as a rigid quadratic surface with unknown parameters. The parameters are estimated by matching vascular landmarks by recursively tracing the blood vessel structure. The parameter estimation technique, which could be generalized to other applications, is a hierarchy of models and methods, making the algorithm robust to unmatchable image features and mismatches between features caused by large interframe motions. Experiments involving 3,000 image pairs from 16 different healthy eyes were performed. Final registration errors less than a pixel are routinely achieved. The speed, accuracy, and ability to handle small overlaps compare favorably with retinal image registration techniques published in the literature.

An algorithm for constructing image mosaics from multiple, uncalibrated, weak-perspective views of the human retina is presented and analyzed. It builds on an algorithm for registering pairs of retinal images using a noninvertible, 12-parameter, quadratic image transformation model and hierarchical, robust estimation. The major innovation presented is a linear, feature-based, noniterative method for jointly estimating consistent transformations of all images onto the mosaic "anchor image." Constraints for this estimation are derived from pairwise registration both directly with the anchor image and indirectly between pairs of nonanchor images. An incremental, graph-based technique constructs the set of registered image pairs used in the solution. The estimation technique allows images that do not overlap the anchor frame to be successfully mosaiced, a valuable capability for mosaicing images of the retinal periphery. Experimental analysis on data sets from 16 eyes shows the average overall median transformation error in final mosaic to be 0.76 pixels. The technique is simpler, more accurate, and offers broader coverage than previously published methods.

Algorithms are presented for fully automatic three-dimensional (3-D) tracing of neurons that are imaged by fluorescence confocal microscopy. Unlike previous voxel-based skeletonization methods, the present approach works by recursively following the neuronal topology, using a set of 4 x N2 directional kernels (e.g., N = 32), guided by a generalized 3-D cylinder model. This method extends our prior work on exploratory tracing of retinal vasculature to 3-D space. Since the centerlines are of primary interest, the 3-D extension can be accomplished by four rather than six sets of kernels. Additional modifications, such as dynamic adaptation of the correlation kernels, and adaptive step size estimation, were introduced for achieving robustness to photon noise, varying contrast, and apparent discontinuity and/or hollowness of structures. The end product is a labeling of all somas present, graph-theoretic representations of all dendritic/axonal structures, and image statistics such as soma volume and centroid, soma interconnectivity, the longest branch, and lengths of all graph branches originating from a soma. This method is able to work directly with unprocessed confocal images, without expensive deconvolution or other preprocessing. It is much faster that skeletonization, typically consuming less than a minute to trace a 70-MB image on a 500-MHz computer. These properties make it attractive for large-scale automated tissue studies that require rapid on-line image analysis, such as high-throughput neurobiology/angiogenesis assays, and initiatives such as the Human Brain Project.

The paper describes a fast feature-based algorithm for accurately estimating the absolute location of a surgical tool (a laser), pointed at the curved human retina, from a series of image frames. The method is capable of a 91% success rate with just a third of the feature extraction computation, taking 37 ms overall per image on a 900 MHz Pentium III. The success rate approaches 100% when the feature extraction is allowed to run to completion. The median error is 0.92 pixels with 512/spl times/512 8-bit image frames. Making a significant break from prior incremental tracking-based efforts, we propose a framework that involves extensive offline precomputation to build a "spatial map" and high-speed online "spatial referencing" to rapidly register each surgical image with this spatial map. The spatial referencing technique is designed around the idea of quasi-invariant indexing. Similarity invariants, locally valid despite the curved nature of the retina, are computed from constellations of vascular landmarks. These are detected using a high-speed algorithm that recursively traces the blood vessel structure. Invariant indexing establishes initial correspondences between landmarks from the online image and landmarks stored in the spatial map. Alignment and verification steps gradually extend the similarity transformation computed from these initial correspondences to a global, high-order transformation. The spatial map is precomputed to contain mosaics, distance maps and an invariant database, all designed to make these spatial referencing computations extremely fast.

Recently, this group published fast algorithms for automatic tracing (vectorization) of the vasculature in live retinal angiograms, and for the extraction of visual landmarks formed by vascular bifurcations and crossings. These landmarks are used for feature-based image matching for controlling a computer-assisted laser retinal surgery instrument currently under development. This paper describes methods to schedule the vascular tracing computations to maximize the rate of growth of quality of the partial tracing results within a frame cycle. There are two main advantages. First, progressive image matching from partially extracted landmark sets can be faster, and provide an earlier indication of matching failure. Second, the likelihood of successful image matching is greatly improved since the extracted landmarks are of the highest quality for the given computational budget. The scheduling method is based on quantitative measures for the computational work and the quality of landmarks. A coarse grid-based analysis of the image is used to generate seed points for the tracing computations, along with estimates of local edge strengths, orientations, and vessel thickness. These estimates are used to define criteria for real-time preemptive scheduling of the tracing computations. It is shown that the optimal schedule can only be achieved in perfect hindsight, and is thus unrealizable. This leads to scheduling heuristics that approximate the behavior of the optimal algorithm. One such approximation produced approximately 400% improvement in the quality of the partial results at a defined milestone, as compared to random scheduling. The resulting algorithm can be readily implemented on conventional and multiple-processor systems, and is being applied to computer-assisted laser retinal surgery.

Methods are presented for increasing the coverage and accuracy of image mosaics constructed from multiple, uncalibrated, weak-perspective views of the human retina. Extending our previous algorithm for registering pairs of images using a non-invertible, 12-parameter, quadratic image transformation model and a hierarchical, robust estimation technique, two important innovations are presented. The first is a linear, non-iterative method for jointly estimating the transformations of all images onto the mosaic. This employs constraints derived from pairwise matching between the non-mosaic image frames. It allows the transformations to be estimated for images that do not overlap the mosaic anchor frame, and results in mutually consistent transformations for all images. This means the mosaics can cover a much broader area of the retinal surface, even though the transformation model is not closed under composition. This capability is particularly valuable for mosaicing the retinal periphery in the context of diseases such as AIDS/CMV. The second innovation is a method to improve the accuracy of the pairwise matches as well as the joint estimation by refining the feature locations and by adding new features based on the transformation estimates themselves. For matching image frames of size 1024/spl times/1024, this cuts the registration error from the range of 1 to 3 pixels to about 0.55 pixels. The overall transformation error in final mosaic construction is 0.80 pixels based on experiments over a large set of eyes.

This paper describes computer vision algorithms to assist in retinal laser surgery, which is widely used to treat leading blindness causing conditions but only has a 50% success rate, mostly due to a lack of spatial mapping and reckoning capabilities in current instruments. The novel technique described here automatically constructs a composite (mosaic) image of the retina from a sequence of incomplete views. This mosaic will be useful to ophthalmologists for both diagnosis and surgery. The new technique goes beyond published methods in both the medical and computer vision literatures because it is fully automated, models the patient-dependent curvature of the retina, handles large interframe motions, and does not require calibration. At the heart of the technique is a 12-parameter image transformation model derived by modeling the retina as a quadratic surface and assuming a weak perspective camera, and rigid motion. Estimating the parameters of this transformation model requires robustness to unmatchable image features and mismatches between features caused by large interframe motions. The described estimation technique is a hierarchy of models and methods: the initial match set is pruned based on a 0th order transformation estimated using a similarity-weighted histogram; a 1st order affine transformation is estimated using the reduced match set and least-median of squares; and the final, 2nd order 12-parameter transformation is estimated using an M-estimator initialized from the 1st order results. Initial experimental results show the method to be robust and accurate in accounting for the unknown retinal curvature in a fully automatic manner while preserving image details.

Algorithms are presented for rapid, automatic, robust, adaptive, and accurate tracing of retinal vasculature and analysis of intersections and crossovers. This method improves upon prior work in several ways: 1) automatic adaptation from frame to frame without manual initialization/adjustment, with few tunable parameters; 2) robust operation on image sequences exhibiting natural variability, poor and varying imaging conditions, including over/under-exposure, low contrast, and artifacts such as glare; 3) does not require the vasculature to be connected, so it can handle partial views; and 4) operation is efficient enough for use on unspecialized hardware, and amenable to deadline-driven computing, being able to produce a rapidly and monotonically improving sequence of usable partial results. Increased computation can be traded for superior tracing performance. Its efficiency comes from direct processing on gray-level data without any preprocessing, and from processing only a minimally necessary fraction of pixels in an exploratory manner, avoiding low-level image-wide operations such as thresholding, edge detection, and morphological processing. These properties make the algorithm suited to real-time, on-line (live) processing and is being applied to computer-assisted laser retinal surgery.

Effective 3-D image processing algorithms are presented for automatic counting and analysis of cells in anisotropic 3-D biological images that are collected by laser-scanning confocal microscopes. In these instruments, the x-y resolution is much better than the resolution along the z axis, hence the voxels (pixels in 3-D) are anisotropic. In this work, the images are pre-processed by a 3-D extension of an anisotropic diffusion algorithm, and the resulting images are binarized by a clustering based segmentation algorithm. As a result of binary segmentation, some regions consist of individual objects while others are multi-object clusters. An extension of Vincent and Soille's watershed algorithm (1991) to anisotropic 3D spaces is used to separate such cell clusters. The watershed algorithm is applied on marker functions that are generated using a combination of 3-D morphological inverse distance functions and 3-D image gradients. Cell measurements, such as volume, average intensity and locations, are calculated on the result of watershed segmentation. This algorithm has been successfully applied to the automated analysis of cell populations from a variety of biological studies involving large numbers of tissue samples.

Although laser retinal surgery is the best available treatment for a variety of ophthalmic disorders, the current procedure has a success rate of less than 50%. A computer-controlled instrument is under development for minimizing this failure rate. This paper presents core image-processing methods used for this system. The image-processing problems of interest in this paper are (i) the computation of a retinal map from video retinal angiograms, and (ii) real-time eye tracking and location determination during laser surgery. The algorithms described here comprise of a fast method for detecting retinal landmark feature points, a fast method for matching feature-tagged point sets, feature-based method for montaging retinal angiograms, and an algorithm for real-time image tracking.

Presents algorithms for segmenting three-dimensional (3-D) brightfield microscope images of thick and overlapped regions of Pap smears, acquired using a specially-developed high-speed 3-D microscope system. Algorithms for segmenting these images require a careful tradeoff between sophistication and processing speed, due to extreme image variability and the large volume of data involved. These challenges are overcome by applying a sequence of algorithms including an adaptive clustering algorithm that exploits local contrast and focus features, a boundary extraction and refinement algorithm based on gray-level thinning, a 3-D extension of the watershed algorithm to separate overlapping objects, and a boundary selection algorithm that takes into account a priori known characteristics of nuclei. It has been successfully applied on a variety of images.

Methods for conducting model-based computer vision from low-SNR (=/<1 dB) image data are presented. Conventional algorithms break down in this regime due to a cascading of noise artifacts, and inconsistencies arising from the lack of optimal interaction between high- and low-level processing. These problems are addressed by solving low-level problems such as intensity estimation, segmentation, and boundary estimation jointly (synergistically) with intermediate-level problems such as the estimation of position, magnification, and orientation, and high-level problems such as object identification and scene interpretation. This is achieved by formulating a single objective function that incorporates all the data and object models, and a hierarchy of constraints in a Bayesian framework. All image-processing operations, including those that exploit the low and high-level variables to satisfy multi-level pattern constraints, result directly from a parallel multi-trajectory global optimization algorithm. Experiments with simulated low-count (7-9 photons/pixel) 2-D Poisson images demonstrate that compared to non-joint methods, a joint solution not only results in more reliable scene interpretation, but also a superior estimation of low-level imaging variables. Typically, most object parameters are estimated to within a 5% accuracy even with overlap and partial occlusion.

A method to reduce the average operation time, and increase the transfer rate of RAID-5 disk arrays is presented, along with quantitative models for these paramaers as a funcrion of load. The modified controller allocates moderate sized files (1-20 sectors), and parity information for groups of consecutive sectors, entirely on one disk, avoiding the compounding of latency and queuing delaysfor multiple disks.

A stable deterministic approach is presented for incorporating unit-simplex constraints based on a hierarchical deformable-template structure. This approach (i) guarantees strict confinement of the search to the unit-simplex constraint set without introducing unwanted constraints; (ii) leads to a hierarchical, rather than a global, network interconnection structure; (iii) allows multiresolution processing; and (iv) allows easy closed-form incorporation of certain other inherently global constraints, such as general recursive symmetries. Selected examples are presented which illustrate and demonstrate large-scale application of the template method.

A general method is proposed for incorporating rule-based constraints corresponding to regular languages into stochastic inference problems, thereby allowing for a unified representation of stochastic and syntactic pattern constraints. The authors' approach establishes the formal connection of rules to Chomsky grammars and generalizes the original work of Shannon on the encoding of rule-based channel sequences to Markov chains of maximum entropy. This maximum entropy probabilistic view leads to Gibbs representations with potentials which have their number of minima growing at precisely the exponential rate that the language of deterministically constrained sequences grow. These representations are coupled to stochastic diffusion algorithms, which sample the language-constrained sequences by visiting the energy minima according to the underlying Gibbs probability law. This coupling yields the result that fully parallel stochastic cellular automata can be derived to generate samples from the rule-based constraint sets. The production rules and neighborhood state structure of the language of sequences directly determine the necessary connection structures of the required parallel computing surface. Representations of this type have been mapped to the DAP-510 massively parallel processor consisting of 1024 mesh-connected bit-serial processing elements for performing automated segmentation of electron-micrograph images.

Earlier work on parallel computation of generalized Bayesian hypothesis tests for hierarchical image reconstruction on massively parallel processor arrays is extended to incorporate pattern constraints specified with Boolean functions defined on symbolic imaging variables. This is based on a stochastic representation for memoryless Boolean functions following U. Grenander's work (1984) on metric pattern theory. Its application is presented to the segmentation of low-contrast textured images through an extension of J. Besag's (1986) ICM segmentation algorithm, and to image reconstruction with large point spread. Hierarchical image reconstruction in time-of-flight positron emission tomography at low count levels is described.>

The authors present a single formulation for constrained imaging that fuses the problem of joint estimation of the continuous parameters using MAP (maximum a posteriori) and conditional-mean estimators with that of performing generalized Bayes hypothesis testing for the symbolic imaging variables. Coupling this with recent results on representing regular grammars via Gibbs' distributions makes it possible to incorporate into a single hierarchical framework the stochastic constraints relevant to continuous-valued parameters as well as language-theoretic constraints on the symbolic variables. The authors also present a method for performing the required computations on a massively parallel architecture, which makes it possible to update every variable at every level in the hierarchy in parallel. The conclusions obtained are supported with results for a Poisson imaging problem computed on a DAP-500 massively parallel processor with 1024 processing elements.>

A constrained maximum-likelihood estimator is derived by incorporating a rotationally invariant roughness penalty proposed by I.J. Good (1981) into the likelihood functional. This leads to a set of nonlinear differential equations the solution of which is a spline-smoothing of the data. The nonlinear partial differential equations are mapped onto a grid via finite differences, and it is shown that the resulting computations possess a high degree of parallelism as well as locality in the data-passage, which allows an efficient implementation on a 48-by-48 mesh-connected array of NCR GAPP processors. The smooth reconstruction of the intensity functions of Poisson point processes is demonstrated in two dimensions.>