In this paper, we propose a novel method to estimate the confidence of a registration that does not require any ground truth, is independent from the registration algorithm and the resulting confidence is correlated with the amount of registration error. We first apply a local search to match patterns between the registered image pairs. Local search induces a cost space per voxel which we explore further to estimate the confidence of the registration similar to confidence estimation algorithms for stereo matching. We test our method on both synthetically generated registration errors and on real registrations with ground truth. The experimental results show that our confidence measure can estimate registration errors and it is correlated with local errors.

Stereo confidence measures are one of the most popular research topics in stereo vision. These measures give an indication about the certainty of the matching. The main aim of using confidence measures is to filter the erroneous disparity estimations at the end of the matching process. However, they can also be incorporated at the initial step of the matching process to obtain accurate estimations before the cost aggregation. In this paper, we propose to utilize stereo confidence measures for fusing different similarity measures in order to obtain robust estimations for aggregation. Since stereo similarity measures perform differently in varying conditions, the confidence-guided fusion of them makes stereo matching more robust against errors. We evaluate the performance of our algorithm in comparison to different similarity measures on the Middleburry benchmark stereo test set. The results show significant improvements on the accuracy of initial disparity estimations with our fusion strategy compared to different similarity measures.

Depth sensors such as Kinect fail to find the depth of transparent objects which makes 3D reconstruction of such objects a challenge. The refinement algorithms for Kinect depth maps either do not address transparency or they only provide sparse depth on such objects which is inadequate for dense 3D reconstruction. In order to solve this problem, we propose a fully-connected CRF based hybrid refinement algorithm. We incorporate stereo cues from cross-modal stereo between IR and RGB cameras of the Kinect and Kinect's depth map. Our algorithm does not require any additional cameras and still provides dense depth estimations of transparent objects and specular surfaces with high accuracy.

State-of-the-art stereo matching algorithms estimate disparities using local block-matching, and subsequently refine the disparity estimates by introducing smoothness constraints and performing global energy minimization. Such algorithms are hampered by the inability of local block-matching algorithms to deal with repetitive patterns. This paper presents an approach that overcomes this problem by incorporating the disparity obtained from matching SURF key points between stereo image pairs. The algorithm provides further robustness to problems with repetitive pattern by penalizing the discrepancy between the initial and final disparity estimates in the global energy minimization. Evaluation of our approach on the Middleburry data set results shows that the our approach is more robust against repetitive patterns than existing approaches.

Fluorescence loss in photobleaching (FLIP) is a method to study compartment connectivity in living cells. A FLIP sequence is obtained by alternatively bleaching a spot in a cell and acquiring an image of the complete cell. Connectivity is estimated by comparing fluorescence signal attenuation in different cell parts. The measurements of the fluorescence attenuation are hampered by the low signal to noise ratio of the FLIP sequences, by sudden sample shifts and by sample drift. This paper describes a method that estimates the attenuation by modeling photobleaching as exponentially decaying signals. Sudden motion artifacts are minimized by registering the frames of a FLIP sequence to target frames based on the estimated model and by removing frames that contain deformations. Linear motion (sample drift) is reduced by minimizing the entropy of the estimated attenuation coefficients. Experiments on 16 in vivo FLIP sequences of muscle cells in Drosophila show that the proposed method results in fluorescence attenuations similar to the manually identified gold standard, but with standard deviations of approximately 50 times smaller. As a result of this higher precision, cell compartment edges and details such as cell nuclei become clearly discernible. The main value of this method is that it uses a model of the bleaching process to correct motion and that the model based fluorescence intensity and attenuation estimates can be interpreted easily. The proposed method is fully automatic, and runs in approximately one minute per sequence, making it suitable for unsupervised batch processing of large data series.

This paper discusses a number of image analysis challenges emerging from longitudinal small animal molecular imaging studies. Three steps towards a quantitative 3D analysis of follow-up small animal imaging are presented: whole-body articulated registration, change visualization in follow-up data and fusion of optical and 3D structural imaging data. Several application examples are presented in the context of translational cancer research.

Human perception of sign language can serve as inspiration for the improvement of automatic recognition systems. Experiments with human signers show that sign language signs contain redundancy over time. In this article, experiments are conducted to investigate whether comparable redundancies also exist for an automatic sign language recognition system. Such redundancies could be exploited, for example, by reserving more processing resources for the more informative phases of a sign, or by discarding uninformative phases. In the experiments, an automatic system is trained and tested on isolated fragments of sign language signs. The stimuli used were similar to those of the human signer experiments, allowing us to compare the results. The experiments show that redundancy over time exists for the automatic recognizer. The central phase of a sign is the most informative phase, and the first half of a sign is sufficient to achieve a recognition performance similar to that of the entire sign. These findings concur with the results of the human signer studies. However, there are differences as well, most notably the fact that human signers score better on the early phases of a sign than the automatic system. The results can be used to improve the automatic recognizer, by using only the most informative phases of a sign as input.

The analysis of multi-timepoint whole-body small animal CT data is greatly complicated by the varying posture of the subject at different timepoints. Due to these variations, correctly relating and comparing corresponding regions of interest is challenging.In addition, occlusion may prevent effective visualization of these regions of interest. To address these problems, we have developed a method that fully automatically maps the data to a standardized layout of sub-volumes, based on an articulated atlas registration. We have dubbed this process articulated planar reformation, or APR. A sub-volume can be interactively selected for closer inspection and can be compared with the corresponding sub-volume at the other timepoints, employing a number of different comparative visualization approaches. We provide an additional tool that highlights possibly interesting areas based on the change of bone density between timepoints. Furthermore we allow visualization of the local registration error, to give an indication of the accuracy of the registration. We have evaluated our approach on a case that exhibits cancer-induced bone resorption.

Object detection is a critical step in automated surveillance. A common approach to constructing object detectors consists of annotating large datasets and using them to train the detectors. However, due to inevitable limitations of a typical training data set, such supervised approach is unsuitable for building generic surveillance systems applicable to a wide variety of scenes and camera setups. In our previous work we proposed an unsupervised method for learning and detecting the dominant object class in a general dynamic scene observed by a static camera. In this paper, we investigate the possibilities to expand the applicability of this method to the problem of multiple dominant object classes. We propose an idea how to approach this expansion, and perform a proof-of-concept evaluation of this idea using a representative surveillance video sequence.

In this study we compare human and machine acceptability judgments for extreme variations in sign productions. We gathered acceptability judgments of 26 signers and scores of three different automatic gesture recognition (AGR) algorithms that could potentially be used for automatic acceptability judgments, in which case the correlation between human ratings and AGR scores may serve as an dasiaacceptability performancepsila measure. We found high human-human correlations, high AGR-AGR correlations, but low human-AGR correlations. Furthermore, in a comparison between acceptability and classification performance of the different AGR methods, classification performance was found to be an unreliable predictor of acceptability performance.

We have developed a prototype for a learning environment for deaf and hard of hearing children. This demonstration consists of hands-on experience with the prototype. In total, there are three exercises: 1) an introduction of all pictures and corresponding signs, 2) multiple choice sign-to-picture and 3) performing the sign that corresponds to the picture shown on the screen. The live recognition from a wide-angle stereo camera provides immediate feedback for the third exercise where the sign must be performed.

We present a method to automatically construct a sign language classifier for a previously unseen sign. The only required input of a new sign is one example, performed by a sign language tutor. The method works by comparing the measurements of the new sign to signs that have been trained on a large number of persons. The parameters of the respective trained classifier models are used to construct a classification model for the new sign. We show that the performance of a classifier constructed from an instructed sign is significantly better than that of dynamic time warping (DTW) with the same sign. Using only a single example, the proposed method has a performance comparable to a regular training with five examples, while being more stable because of the larger source of information.

To recognize speech, handwriting or sign language, many hybrid approaches have been proposed that combine Dynamic Time Warping (DTW) or Hidden Markov Models (HMM) with discriminative classifiers. However, all methods rely directly on the likelihood models of DTW/HMM. We hypothesize that time warping and classification should be separated because of conflicting likelihood modelling demands. To overcome these restrictions, we propose to use Statistical DTW (SDTW) only for time warping, while classifying the warped features with a different method. Two novel statistical classifiers are proposed (CDFD and Q-DFFM), both using a selection of discriminative features (DF), and are shown to outperform HMM and SDTW. However, we have found that combining likelihoods of multiple models in a second classification stage degrades performance of the proposed classifiers, while improving performance with HMM and SDTW. A proof-of-concept experiment, combining DFFM mappings of multiple SDTW models with SDTW likelihoods, shows that also for model-combining, hybrid classification can provide significant improvement over SDTW. Although recognition is mainly based on 3D hand motion features, these results can be expected to generalize to recognition with more detailed measurements such as hand/body pose and facial expression.

Object detection is a crucial step in automating monitoring and surveillance. A classical approach to object detection employs supervised learning methods, which are effective in well-defined narrow application scopes. In this paper we propose a framework for detecting moving objects in video, which first learns autonomously and on-line the characteristic features of typical object appearances at various parts of the observed scene. The collected knowledge is then used to calibrate the system for the given scene, and to separate isolated appearances of a dominant moving object from other events. Compared to the supervised detectors, the proposed framework is self-adaptable, and therefore able to handle large diversity of objects and situations, typical for general surveillance and monitoring applications. We demonstrate the effectiveness of our framework by employing it to isolate pedestrians in public places and cars on a highway.

A 3D visual hand gesture recognition method is proposed that detects correctly performed signs from stereo camera input. Hand tracking is based on skin detection with an adaptive chrominance model to get high accuracy. Informative high level motion properties are extracted to simplify the classification task. Each example is mapped onto a fixed reference sign by Dynamic Time Warping, to get precise time correspondences. The classification is done by combining weak classifiers based on robust statistics. Each base classifier assumes a uniform distribution of a single feature, determined by winsorization on the noisy training set. The operating point of the classifier is determined by stretching the uniform distributions of the base classifiers instead of changing the threshold on the total posterior likelihood. In a cross validation with 120 signs performed by 70 different persons, 95% of the test signs were correctly detected at a false positive rate of 5%.

This paper addresses the problem of deriving measures that express the degree of the reliability of motion vectors estimated by a block-matching motion estimation method. We express the block matching motion estimation scheme in the probabilistic framework as a maximum-likelihood estimation scheme. Subsequently, we derive the confidence measures in terms of the a posteriori probabilities and the likelihoods of the estimated vectors. The assumptions about the type of the likelihood, that is, about the underlying conditional probability distribution of the motion compensated intensity differences (e.g., Laplacian) are derived from the objective criterion of the block-matching estimator. All parameters are estimated from data that are derived as by-product of the motion estimation scheme and our method, practically, introduces no additional computational cost. The derivation of the confidence measures is incorporated in a multiscale scheme. Experimental results are presented for image sequences with known ground-truth motion.

Estimating the number of persons in a public place provides useful information for video-based surveillance and monitoring applications. In the case of oblique camera setup, counting is either achieved by detecting individuals or by statistically establishing relations between values of simple image features (e.g. amount of moving pixels, edge density, etc.) to the number of people. While the methods of the first category exhibit poor accuracy in cases of occlusions, the second category of methods are sensitive to perspective distortions, and require people to move in order to be counted. In this paper we investigate the possibilities of developing a robust statistical method for people counting. To maximize its applicability scope, we choose-in contrast to the majority of existing methods from this category-not to require prior learning of categories corresponding to different number of people. Second, we search for a suitable way of correcting the perspective distortion. Finally, we link the estimation to a confidence value that takes into account the known factors being of influence on the result. The confidence is then used to refine final results.

Usually, object detection is performed directly on (normalized) gray values or gray primitives like gradients or Haar-like features. In that case the learning of relationships between gray primitives, that describe the structure of the object, is the complete responsibility of the classifier. We propose to apply more knowledge about the image structure in the preprocessing step, by computing local isophote directions and curvatures, in order to supply the classifier with much more informative image structure features. However, a periodic feature space, like orientation, is unsuited for common classification methods. Therefore, we split orientation into two more suitable components. Experiments show that the isophote features result in better detection performance than intensities, gradients or Haar-like features.

We present a method for improving the temporal stability of video object segmentation algorithms for 3D-TV applications. First, two quantitative measures to evaluate temporal stability without ground-truth are presented. Then, a pseudo-3D curve evolution method, which spatio-temporally stabilizes the estimated object segments is introduced. Temporal stability is achieved by re-distributing existing object segmentation errors such that they are less disturbing when the scene is rendered and viewed in 3D. Our starting point is the hypothesis that if making segmentation errors are inevitable, they should be made in a temporally consistent way for 3D TV applications. This hypothesis is supported by the experiments, which show that there is significant improvement in segmentation quality both in terms of the objective quantitative measures and in terms of the viewing comfort in subjective perceptual tests. This shows that it is possible to increase the object segmentation quality without increasing the actual segmentation accuracy.

In this paper, we describe the view representation and reconstruction module for an image-based telepresence system, based on a viewpoint-adaptation scheme and the image-based rendering technique. For real-time three-dimensional synthesis, a parallel version of our multistep view reconstruction algorithm is realized on dedicated hardware based on Trimedia digital signal processors. A new scalable representation, named middle view stereo representation (MVSR), is further constructed to be able to adapt to the bandwidth requirement of the broadcasting network. Experiments show the promising performance of the parallel realization of the multistep view reconstruction method and the power of the MVSR representation.

In this paper, we propose a novel area-based algorithm for stereo correspondence estimation. The new dense correspondence estimation algorithm, which we call HACM (Hierarchical Adaptive Curve Matching), is suitable for stereo analysis of both static and dynamic scenes. The core of this method is the derivation of a pixel-based dense adaptive curve representation. For each pixel in the concerned 2-D images, the changes are characterized roughly in the local surface shape, the texture direction, and the luminance properties, all at a slight overhead cost. The matching is carried out in a hierarchical manner that incorporates several novel ideas that improve its efficiency and robustness. Experiments conducted on both synthetic and natural stereo pairs, and natural video streams, reveal the favorable performance of this new algorithm.

This paper addresses the problem of deriving measures that express the degree of the reliability of motion vectors estimated by a block matching motion estimation method. We express the block matching motion estimation scheme in the probabilistic framework and derive the confidence measures in terms of the a-posteriori probabilities of the estimated vectors. The type of the underlying conditional probability distribution of the motion compensated intensity differences (e.g. Laplacian) is derived from the objective criterion of the block matching estimator. All parameters are estimated from data that are derived as a by-product of the motion estimation scheme. The derivation is incorporated in a multiscale scheme. Experimental results are presented for image sequences with known ground-truth motion.

Abstract Selection of the best set of scales is problematic when developing signal‐driven approaches for pixel‐based image segmentation. Often, different possibly conflicting criteria need to be fulfilled in order to obtain the best trade‐off between uncertainty (variance) and location accuracy. The optimal set of scales depends on several factors: the noise level present in the image material, the prior distribution of the different types of segments, the class‐conditional distributions associated with each type of segment as well as the actual size of the (connected) segments. We analyse, theoretically and through experiments, the possibility of using the overall and class‐conditional error rates as criteria for selecting the optimal sampling of the linear and morphological scale spaces. It is shown that the overall error rate is optimized by taking the prior class distribution in the image material into account. However, a uniform (ignorant) prior distribution ensures constant class‐conditional error rates. Consequently, we advocate for a uniform prior class distribution when an uncommitted, scale‐invariant segmentation approach is desired. Experiments with a neural net classifier developed for segmentation of dynamic magnetic resonance (MR) images, acquired with a paramagnetic tracer, support the theoretical results. Furthermore, the experiments show that the addition of spatial features to the classifier, extracted from the linear or morphological scale spaces, improves the segmentation result compared to a signal‐driven approach based solely on the dynamic MR signal. The segmentation results obtained from the two types of features are compared using two novel quality measures that characterize spatial properties of labelled images. Copyright © 2002 John Wiley & Sons, Ltd.

In this paper we propose a middle view stereo representation (MVSR) to efficiently utilize the bandwidth and ease the implementation of a view-adaptive teleconference system. We show that the MVSR can be used to reconstruct a high quality view from an arbitrary pose, using occlusion information. Its modular property makes real-time implementation feasible and due to the parallel setup latency can be kept low.

This paper addresses the problem of spatio-temporal segmentation of video sequences. An initial intensity segmentation method (watershed segmentation) provides a number of initial segments which are subsequently labeled, with a known number of labels, according to motion information. The label field is modeled as a Markov random field where the statistical spatial and and temporal interactions are expressed on the basis of the initial watershed segments. The labeling criterion is the maximization of the conditional a posteriori probability of the label field given the motion hypotheses, the estimate of the label field of the previous frame, and the image intensities. For the optimization, an iterative motion estimation-labeling algorithm is proposed and experimental results are presented.

Traditional definitions of edges and corners are limited in the sense that they restrict their application. In this paper we generalize the concepts of edges and corners to 1D features and 2D features, respectively. The phase congruency theory is chosen as a mathematical system to model the generalized concepts. The basics of the phase congruency theory along with the local energy model are described. The extraction schemes of 2D features from 2D gray level images are studied and discussed as a demonstration of our generalization idea. The performance evaluation and the preliminary improvement show that this kind of generalization on features and the modeling are successful and promising.

We propose a generic algorithm for the geometrically correct reconstruction of 3-D scenes on stereo displays with viewpoint adaptation. This forms the basis of multiviewpoint systems, which are currently the most promising candidates for real-time implementations of 3-D visual communication systems. The reconstruction algorithm needs 3-D tracking of the viewers' eyes with respect to the display. We analyze the effect of eye-tracking errors. A simple bound is derived, below which reconstruction errors cannot be observed. We design a multiviewpoint system using a recently introduced image-based scene representation. The design formed the basis of the real-time multiviewpoint system that was recently built in the European PANORAMA project. Experiments with both natural and synthetic scenes show that the proposed reconstruction algorithm performs well. The experiments are performed by computer simulations and the real-time PANORAMA system.

This paper addresses the problem of segmentation of image sequences into moving objects. The segmentation (label) field is modeled by applying a Markov random field on regions provided by a watershed algorithm. The algorithm iterates between a motion estimation and a segmentation step. Temporal consistency is introduced by tracking the segmentation in time.

We present an algorithm for the synthesis of multi viewpoint images at non-intermediate positions, based on stereoscopic images. We consider the synthesis of images from virtual camera positions and the synthesis of images for scene reconstruction using stereo displays. The algorithm provides scene reconstruction without geometric distortion and without any restriction to the position of the viewer. All synthesized images are based on extrapolation of a single source image and a single disparity field. This provides low use of bandwidth and compatibility with mono video systems. With teleconferencing images, the generated views were subjectively evaluated as good for viewing positions not more than one half camera baseline from the centre position. Objectively, reconstructed left and right images have PSNR values of 41 dB.

Several implementations of a recursive disparity estimator have been investigated in which bi-directional spatial predictions as well as temporal predictions are used. The recursive disparity estimator is showed to be very attractive for real time multi-view interpolation applications because of the use of the predictions and small search ranges or update sets. The estimated disparity fields are smooth, causing less visible artifacts in the interpolated intermediate viewpoints. Moreover, the recursive estimator is attractive for hardware implementation. Only a small number of updates have to be checked, thereby considerably reducing the number of operations to be executed.