Automatic tracking of viral and intracellular structures displayed as spots with varying sizes in fluorescence microscopy images is an important task to quantify cellular processes. We propose a novel probabilistic tracking approach for multiple particle tracking based on multi-detector and multi-scale data fusion as well as Bayesian smoothing. The approach integrates results from multiple detectors using a novel intensity-based covariance intersection method which takes into account information about the image intensities, positions, and uncertainties. The method ensures a consistent estimate of multiple fused particle detections and does not require an optimization step. Our probabilistic tracking approach performs data fusion of detections from classical and deep learning methods as well as exploits single-scale and multi-scale detections. In addition, we use Bayesian smoothing to fuse information of predictions from both past and future time points. We evaluated our approach using image data of the Particle Tracking Challenge and achieved state-of-the-art results or outperformed previous methods. Our method was also assessed on challenging live cell fluorescence microscopy image data of viral and cellular proteins expressed in hepatitis C virus-infected cells and chromatin structures in non-infected cells, acquired at different spatial-temporal resolutions. We found that the proposed approach outperforms existing methods.

Cell nuclei segmentation is challenging due to shape variation and closely clustered or partially overlapping objects. Most previous methods are not globally optimal, limited to elliptical models, or are computationally expensive. In this work, we introduce a globally optimal approach based on deformable shape models and global energy minimization for cell nuclei segmentation and cluster splitting. We propose an implicit parameterization of deformable shape models and show that it leads to a convex energy. Convex energy minimization yields the global solution independently of the initialization, is fast, and robust. To jointly perform cell nuclei segmentation and cluster splitting, we developed a novel iterative global energy minimization method, which leverages the inherent property of superadditivity of the convex energy. This property exploits the lower bound of the energy of the union of the models and improves the computational efficiency. Our method provably determines a solution close to global optimality. In addition, we derive a closed-form solution of the proposed global minimization based on the superadditivity property for non-clustered cell nuclei. We evaluated our method using fluorescence microscopy images of five different cell types comprising various challenges, and performed a quantitative comparison with previous methods. Our method achieved state-of-the-art or improved performance.

Automatic tracking of particles in time-lapse fluorescence microscopy images is essential for quantifying the dynamic behavior of subcellular structures and virus structures. We introduce a novel particle tracking approach based on a deep recurrent neural network architecture that exploits past and future information in both forward and backward direction. Assignment probabilities are determined jointly across multiple detections, and the probability of missing detections is computed. In addition, existence probabilities are determined by the network to handle track initiation and termination. For correspondence finding, track hypotheses are propagated to future time points so that information at later time points can be used to resolve ambiguities. A handcrafted similarity measure and handcrafted motion features are not necessary. Manually labeled data is not required for network training. We evaluated the performance of our approach using image data of the Particle Tracking Challenge as well as real fluorescence microscopy image sequences of virus structures. It turned out that the proposed approach outperforms previous methods.

Non-rigid registration of cell nuclei in time-lapse microscopy images can be achieved through estimating the deformation fields using optical flow methods. In contrast to local optical flow models employed in the existing non-rigid registration methods, we introduce approaches based on a global optical flow model. Our registration model consists of a data fidelity term and a regularization term. We compared different regularizers for the deformation fields and found that a convex quadratic function is more suitable than non-convex ones. To improve the robustness, we propose an adaptive weighting scheme based on the statistics of the noise in fluorescence microscopy images as well as a combined local-global scheme. Moreover, we extend the global method by exploiting high-order image features. The best suitable high-order features are determined through learning two generative image models, namely, fields of experts and convolutional Gaussian restricted Boltzmann machine, whose model formulations are both consistent with the assumption of high-order feature constancy in the registration model. Using multiple data sets of real 2D and 3D live cell microscopy image sequences as well as synthetic image data, we demonstrate that our proposed approach outperforms the previous methods in terms of both registration accuracy and computational efficiency.

We propose a novel minimal path method for the segmentation of 2D and 3D line structures. Minimal path methods perform propagation of a wavefront emanating from a start point at a speed derived from image features, followed by path extraction using backtracing. Usually, the computation of the speed and the propagation of the wave are two separate steps, and point features are used to compute a static speed. We introduce a new continuous minimal path method which steers the wave propagation progressively using dynamic speed based on path features. We present three instances of our method, using an appearance feature of the path, a geometric feature based on the curvature of the path, and a joint appearance and geometric feature based on the tangent of the wavefront. These features have not been used in previous continuous minimal path methods. We compute the features dynamically during the wave propagation, and also efficiently using a fast numerical scheme and a low-dimensional parameter space. Our method does not suffer from discretization or metrication errors. We performed qualitative and quantitative evaluations using 2D and 3D images from different application areas.

The analysis of the pure motion of subnuclear structures without influence of the cell nucleus motion and deformation is essential in live cell imaging. In this paper, we propose a 2-D contour-based image registration approach for compensation of nucleus motion and deformation in fluorescence microscopy time-lapse sequences. The proposed approach extends our previous approach, which uses a static elasticity model to register cell images. Compared with that scheme, the new approach employs a dynamic elasticity model for the forward simulation of nucleus motion and deformation based on the motion of its contours. The contour matching process is embedded as a constraint into the system of equations describing the elastic behavior of the nucleus. This results in better performance in terms of the registration accuracy. Our approach was successfully applied to real live cell microscopy image sequences of different types of cells including image data that was specifically designed and acquired for evaluation of cell image registration methods. An experimental comparison with the existing contour-based registration methods and an intensity-based registration method has been performed. We also studied the dependence of the results on the choice of method parameters.

To gain a better understanding of cellular and molecular processes, it is important to quantitatively analyze the motion of subcellular particles in live cell microscopy image sequences. Since, generally, the subcellular particles move and cell nuclei move as well as deform, it is important to decouple the movement of particles from that of the cell nuclei using non-rigid registration methods. We have developed a diffeomorphic multi-frame approach for non-rigid registration of cell nuclei in 2D and 3D live cell fluorescence microscopy images. Our non-rigid registration approach is based on local optic flow estimation, exploits information from multiple consecutive image frames, and determines diffeomorphic transformations in the log-domain, which allows efficient computation of the inverse transformations. To register single images of an image sequence to a reference image, we use a temporally weighted mean image, which is constructed based on inverse transformations and multiple consecutive frames. Using multiple consecutive frames improves the registration accuracy compared to pairwise registration, and using a temporally weighted mean image significantly reduces the computation time compared with previous work. In addition, we use a flow boundary preserving method for regularization of computed deformation vector fields, which prevents from over-smoothing compared to standard Gaussian filtering. Our approach has been successfully applied to 2D and 3D synthetic as well as real live cell microscopy image sequences, and an experimental comparison with non-rigid pairwise, multi-frame, and temporal groupwise registration has been carried out.

We propose a novel hybrid approach for automatic 3D segmentation and quantification of high-resolution 7 Tesla magnetic resonance angiography (MRA) images of the human cerebral vasculature. Our approach consists of two main steps. First, a 3D model-based approach is used to segment and quantify thick vessels and most parts of thin vessels. Second, remaining vessel gaps of the first step in low-contrast and noisy regions are completed using a 3D minimal path approach, which exploits directional information. We present two novel minimal path approaches. The first is an explicit approach based on energy minimization using probabilistic sampling, and the second is an implicit approach based on fast marching with anisotropic directional prior. We conducted an extensive evaluation with over 2300 3D synthetic images and 40 real 3D 7 Tesla MRA images. Quantitative and qualitative evaluation shows that our approach achieves superior results compared with a previous minimal path approach. Furthermore, our approach was successfully used in two clinical studies on stroke and vascular dementia.

Automatic fluorescent particle tracking is an essential task to study the dynamics of a large number of biological structures at a sub-cellular level. We have developed a probabilistic particle tracking approach based on multi-scale detection and two-step multi-frame association. The multi-scale detection scheme allows coping with particles in close proximity. For finding associations, we have developed a two-step multi-frame algorithm, which is based on a temporally semiglobal formulation as well as spatially local and global optimization. In the first step, reliable associations are determined for each particle individually in local neighborhoods. In the second step, the global spatial information over multiple frames is exploited jointly to determine optimal associations. The multi-scale detection scheme and the multi-frame association finding algorithm have been combined with a probabilistic tracking approach based on the Kalman filter. We have successfully applied our probabilistic tracking approach to synthetic as well as real microscopy image sequences of virus particles and quantified the performance. We found that the proposed approach outperforms previous approaches.

Tracking subcellular structures as well as viral structures displayed as 'particles' in fluorescence microscopy images yields quantitative information on the underlying dynamical processes. We have developed an approach for tracking multiple fluorescent particles based on probabilistic data association. The approach combines a localization scheme that uses a bottom-up strategy based on the spot-enhancing filter as well as a top-down strategy based on an ellipsoidal sampling scheme that uses the Gaussian probability distributions computed by a Kalman filter. The localization scheme yields multiple measurements that are incorporated into the Kalman filter via a combined innovation, where the association probabilities are interpreted as weights calculated using an image likelihood. To track objects in close proximity, we compute the support of each image position relative to the neighboring objects of a tracked object and use this support to recalculate the weights. To cope with multiple motion models, we integrated the interacting multiple model algorithm. The approach has been successfully applied to synthetic 2-D and 3-D images as well as to real 2-D and 3-D microscopy images, and the performance has been quantified. In addition, the approach was successfully applied to the 2-D and 3-D image data of the recent Particle Tracking Challenge at the IEEE International Symposium on Biomedical Imaging (ISBI) 2012.

MOTIVATION: Automatic tracking of cells in multidimensional time-lapse fluorescence microscopy is an important task in many biomedical applications. A novel framework for objective evaluation of cell tracking algorithms has been established under the auspices of the IEEE International Symposium on Biomedical Imaging 2013 Cell Tracking Challenge. In this article, we present the logistics, datasets, methods and results of the challenge and lay down the principles for future uses of this benchmark. RESULTS: The main contributions of the challenge include the creation of a comprehensive video dataset repository and the definition of objective measures for comparison and ranking of the algorithms. With this benchmark, six algorithms covering a variety of segmentation and tracking paradigms have been compared and ranked based on their performance on both synthetic and real datasets. Given the diversity of the datasets, we do not declare a single winner of the challenge. Instead, we present and discuss the results for each individual dataset separately. AVAILABILITY AND IMPLEMENTATION: The challenge Web site (http://www.codesolorzano.com/celltrackingchallenge) provides access to the training and competition datasets, along with the ground truth of the training videos. It also provides access to Windows and Linux executable files of the evaluation software and most of the algorithms that competed in the challenge.

Characterizing the activating and inhibiting effect of protein-protein interactions (PPI) is fundamental to gain insight into the complex signaling system of a human cell. A plethora of methods has been suggested to infer PPI from data on a large scale, but none of them is able to characterize the effect of this interaction. Here, we present a novel computational development that employs mitotic phenotypes of a genome-wide RNAi knockdown screen and enables identifying the activating and inhibiting effects of PPIs. Exemplarily, we applied our technique to a knockdown screen of HeLa cells cultivated at standard conditions. Using a machine learning approach, we obtained high accuracy (82% AUC of the receiver operating characteristics) by cross-validation using 6,870 known activating and inhibiting PPIs as gold standard. We predicted de novo unknown activating and inhibiting effects for 1,954 PPIs in HeLa cells covering the ten major signaling pathways of the Kyoto Encyclopedia of Genes and Genomes, and made these predictions publicly available in a database. We finally demonstrate that the predicted effects can be used to cluster knockdown genes of similar biological processes in coherent subgroups. The characterization of the activating or inhibiting effect of individual PPIs opens up new perspectives for the interpretation of large datasets of PPIs and thus considerably increases the value of PPIs as an integrated resource for studying the detailed function of signaling pathways of the cellular system of interest.

The entry process of virus particles into cells is decisive for infection. In this work, we investigate fusion of virus particles with the cell membrane via time-lapse fluorescence microscopy. To automatically identify fusion for single particles based on their intensity over time, we have developed a layered probabilistic approach. The approach decomposes the action of a single particle into three abstractions: the intensity over time, the underlying temporal intensity model, as well as a high level behavior. Each abstraction corresponds to a layer and these layers are represented via stochastic hybrid systems and hidden Markov models. We use a maxbelief strategy to efficiently combine both representations. To compute estimates for the abstractions we use a hybrid particle filter and the Viterbi algorithm. Based on synthetic image sequences, we characterize the performance of the approach as a function of the image noise. We also characterize the performance as a function of the tracking error. We have also successfully applied the approach to real image sequences displaying pseudotyped HIV-1 particles in contact with host cells and compared the experimental results with ground truth obtained by manual analysis.

BACKGROUND: High-content, high-throughput RNA interference (RNAi) offers unprecedented possibilities to elucidate gene function and involvement in biological processes. Microscopy based screening allows phenotypic observations at the level of individual cells. It was recently shown that a cell's population context significantly influences results. However, standard analysis methods for cellular screens do not currently take individual cell data into account unless this is important for the phenotype of interest, i.e. when studying cell morphology. RESULTS: We present a method that normalizes and statistically scores microscopy based RNAi screens, exploiting individual cell information of hundreds of cells per knockdown. Each cell's individual population context is employed in normalization. We present results on two infection screens for hepatitis C and dengue virus, both showing considerable effects on observed phenotypes due to population context. In addition, we show on a non-virus screen that these effects can be found also in RNAi data in the absence of any virus. Using our approach to normalize against these effects we achieve improved performance in comparison to an analysis without this normalization and hit scoring strategy. Furthermore, our approach results in the identification of considerably more significantly enriched pathways in hepatitis C virus replication than using a standard analysis approach. CONCLUSIONS: Using a cell-based analysis and normalization for population context, we achieve improved sensitivity and specificity not only on a individual protein level, but especially also on a pathway level. This leads to the identification of new host dependency factors of the hepatitis C and dengue viruses and higher reproducibility of results.

The observed motion of subcellular particles in fluorescence microscopy image sequences of live cells is generally a superposition of the motion and deformation of the cell and the motion of the particles. Decoupling the two types of movements to enable accurate classification of the particle motion requires the application of registration algorithms. We have developed an intensity-based approach for nonrigid registration of multichannel microscopy image sequences of cell nuclei. First, based on 3-D synthetic images we demonstrate that cell nucleus deformations change the observed motion types of particles and that our approach allows to recover the original motion. Second, we have successfully applied our approach to register 2-D and 3-D real microscopy image sequences. A quantitative experimental comparison with previous approaches for nonrigid registration of cell microscopy has also been performed.

MOTIVATION: Detecting human proteins that are involved in virus entry and replication is facilitated by modern high-throughput RNAi screening technology. However, hit lists from different laboratories have shown only little consistency. This may be caused by not only experimental discrepancies, but also not fully explored possibilities of the data analysis. We wanted to improve reliability of such screens by combining a population analysis of infected cells with an established dye intensity readout. RESULTS: Viral infection is mainly spread by cell-cell contacts and clustering of infected cells can be observed during spreading of the infection in situ and in vivo. We employed this clustering feature to define knockdowns which harm viral infection efficiency of human Hepatitis C Virus. Images of knocked down cells for 719 human kinase genes were analyzed with an established point pattern analysis method (Ripley's K-function) to detect knockdowns in which virally infected cells did not show any clustering and therefore were hindered to spread their infection to their neighboring cells. The results were compared with a statistical analysis using a common intensity readout of the GFP-expressing viruses and a luciferase-based secondary screen yielding five promising host factors which may suit as potential targets for drug therapy. CONCLUSION: We report of an alternative method for high-throughput imaging methods to detect host factors being relevant for the infection efficiency of viruses. The method is generic and has the potential to be used for a large variety of different viruses and treatments being screened by imaging techniques.

We introduce a new model-based approach for automatic quantification of colocalizations in multichannel 3D microscopy images. The approach uses different 3D parametric intensity models in conjunction with a model fitting scheme to localize and quantify subcellular structures with high accuracy. The central idea is to determine colocalizations between different channels based on the estimated geometry of the subcellular structures as well as to differentiate between different types of colocalizations. A statistical analysis was performed to assess the significance of the determined colocalizations. This approach was used to successfully analyze about 500 three-channel 3D microscopy images of human soft tissue tumors and controls.

We present an intensity-based nonrigid registration approach for the normalization of 3-D multichannel microscopy images of cell nuclei. A main problem with cell nuclei images is that the intensity structure of different nuclei differs very much; thus, an intensity-based registration scheme cannot be used directly. Instead, we first perform a segmentation of the images from the cell nucleus channel, smooth the resulting images by a Gaussian filter, and then apply an intensity-based registration algorithm. The obtained transformation is applied to the images from the nucleus channel as well as to the images from the other channels. To improve the convergence rate of the algorithm, we propose an adaptive step length optimization scheme and also employ a multiresolution scheme. Our approach has been successfully applied using 2-D cell-like synthetic images, 3-D phantom images as well as 3-D multichannel microscopy images representing different chromosome territories and gene regions. We also describe an extension of our approach, which is applied for the registration of 3D + t (4-D) image series of moving cell nuclei.

Investigation of 3D chromatin structure in interphase cell nuclei is important for the understanding of genome function. For a reconstruction of the 3D architecture of the human genome, systematic fluorescent in situ hybridization in combination with 3D confocal laser scanning microscopy is applied. The position of two or three genomic loci plus the overall nuclear shape were simultaneously recorded, resulting in statistical series of pair and triple loci combinations probed along the human chromosome 1 q-arm. For interpretation of statistical distributions of geometrical features (e.g. distances, angles, etc.) resulting from finite point sampling experiments, a Monte-Carlo-based approach to numerical computation of geometrical probability density functions (PDFs) for arbitrarily-shaped confined spatial domains is developed. Simulated PDFs are used as bench marks for evaluation of experimental PDFs and quantitative analysis of dimension and shape of probed 3D chromatin regions. Preliminary results of our numerical simulations show that the proposed numerical model is capable to reproduce experimental observations, and support the assumption of confined random folding of 3D chromatin fiber in interphase cell nuclei

We introduce a new approach for 3-D segmentation and quantification of vessels. The approach is based on a 3-D cylindrical parametric intensity model, which is directly fitted to the image intensities through an incremental process based on a Kalman filter. Segmentation results are the vessel centerline and shape, i.e., we estimate the local vessel radius, the 3-D position and 3-D orientation, the contrast, as well as the fitting error. We carried out an extensive validation using 3-D synthetic images and also compared the new approach with an approach based on a Gaussian model. In addition, the new model has been successfully applied to segment vessels from 3-D MRA and computed tomography angiography image data. In particular, we compared our approach with an approach based on the randomized Hough transform. Moreover, a validation of the segmentation results based on ground truth provided by a radiologist confirms the accuracy of the new approach. Our experiments show that the new model yields superior results in estimating the vessel radius compared to previous approaches based on a Gaussian model as well as the Hough transform.

We investigate different approaches for efficient feature space reduction and compare different methods for cell classification. The application context is the development of automatic methods for analysing fluorescence microscopy images with the goal to identify those genes that are involved in the mitosis of human cells (cell division). We distinguish four cell classes comprising interphase cells, mitotic cells, apoptotic cells, and cells with clustered nuclei. Feature space reduction was performed using the Principal Component Analysis and Independent Component Analysis methods. Six classification methods were examined including unsupervised clustering algorithms such as K-means, Hard Competitive Learning, and Neural Gas as well as Hierarchical Clustering, Support Vector Machines, and Random Forests classifiers. Detailed results on the cell image classification accuracy and computational efficiency achieved using different feature sets and different classification methods are reported.

We present a novel approach to the coarse segmentation of tubular structures in three-dimensional (3-D) image data. Our algorithm, which requires only few initial values and minimal user interaction, can be used to initialize complex deformable models and is based on an extension of the randomized hough transform (RHT), a robust method for low-dimensional parametric object detection. Tubular structures are modeled as generalized cylinders. By means of a discrete Kalman filter, they are tracked through 3-D space. Our extensions to the RHT are a feature adaptive selection of the sample size, expectation-dependent weighting of the input data, and a novel 3-D parameterization for straight elliptical cylinders. Experimental results obtained for 3-D synthetic as well as for 3-D medical images demonstrate the robustness of our approach w.r.t. image noise. We present the successful segmentation of tubular anatomical structures such as the aortic arc and the spinal cord.

We consider elastic image registration based on a set of corresponding anatomical point landmarks and approximating thin-plate splines. This approach is an extension of the original interpolating thin-plate spline approach and allows to take into account landmark localization errors. The extension is important for clinical applications since landmark extraction is always prone to error. Our approach is based on a minimizing functional and can cope with isotropic as well as anisotropic landmark errors. In particular, in the latter case it is possible to include different types of landmarks, e.g., unique point landmarks as well as arbitrary edge points. Also, the scheme is general with respect to the image dimension and the order of smoothness of the underlying functional. Optimal affine transformations as well as interpolating thin-plate splines are special cases of this scheme. To localize landmarks we use a semi-automatic approach which is based on three-dimensional (3-D) differential operators. Experimental results are presented for two-dimensional as well as 3-D tomographic images of the human brain.

We introduce radial basis functions with compact support for elastic registration of medical images. With these basis functions the influence of a landmark on the registration result is limited to a circle in 2D and, respectively, to a sphere in 3D. Therefore, the registration can be locally constrained which especially allows to deal with rather local changes in medical images due to, e.g., tumor resection. An important property of the used RBFs is that they are positive definite. Thus, the solvability of the resulting system of equations is always guaranteed. We demonstrate our approach for synthetic as well as for 2D and 3D tomographic images.

The accuracy of image-guided neurosurgery generally suffers from brain deformations due to intraoperative changes. These deformations cause significant changes of the anatomical geometry (organ shape and spatial interorgan relations), thus making intraoperative navigation based on preoperative images error prone. In order to improve the navigation accuracy, we developed a biomechanical model of the human head based on the finite element method, which can be employed for the correction of preoperative images to cope with the deformations occurring during surgical interventions. At the current stage of development, the two-dimensional (2-D) implementation of the model comprises two different materials, though the theory holds for the three-dimensional (3-D) case and is capable of dealing with an arbitrary number of different materials. For the correction of a preoperative image, a set of homologous landmarks must be specified which determine correspondences. These correspondences can be easily integrated into the model and are maintained throughout the computation of the deformation of the preoperative image. The necessary material parameter values have been determined through a comprehensive literature study. Our approach has been tested for the case of synthetic images and yields physically plausible deformation results. Additionally, we carried out registration experiments with a preoperative MR image of the human head and a corresponding postoperative image simulating an intraoperative image. We found that our approach yields good prediction results, even in the case when correspondences are given in a relatively small area of the image only.

The paper presentsanewparameter-free approach to non-rigid image registration, where displacements, obtained through a mapping of boundary structures in the source and target image, are incorporated as hard constraints for elastic image deformation. As a consequence, our approach does not contain any parameters of the deformation model (elastic constants). The approach guarantees the exact correspondence of boundary structures after elastic transformation provided that correct input data are available. We describe a linear and an incremental model, the latter model allows to cope also with large deformations. Experimental results for 2-D and 3-D synthetic as well as real medical images are presented. 1

We describe a new method to evaluate corner extraction schemes using invariance methods. Since the locations of centers in an image depend both on the intrinsic parameters of the camera and the relative position and orientation of the object with respect to the camera, the exact positions of corners in an image are generally not known. To circumvent the need for this knowledge, we use sets of points (instead of individual points) extracted from images of polyhedral objects and projective invariants to calculate a manifold of constraints on the coordinates of the corners. We then estimate the variance of the detected corners from the distance of the coordinate vector to this manifold. This is independent of the camera parameters and the relative position and orientation between the camera and the object. Five different kinds of corner extraction schemes are investigated. The purpose of the paper is to show that invariance methods can effectively be used to make this comparison rather than to make a thorough comparison of different corner extraction schemes.

An approach that uses a volume model consisting of cylinders for model-based recognition of pedestrians in real-world images is presented. The human body is represented by a volume model, and medical motion data are used for simulating the movement of walking. This knowledge is exploited to determine the 3-D position, as well as the posture of an observed person. By applying a Kalman filter, the model parameters in consecutive images are incrementally estimated. The approach is tested on real image data.>