We propose a fast and accurate surface reconstruction algorithm for unorganized point clouds using an implicit representation. Recent learning methods are either singleobject representations with small neural models that allow for high surface details but require per-object training or generalized representations that require larger models and generalize to newer shapes but lack details, and inference is slow. We propose a new implicit representation for general 3D shapes that is faster than all the baselines at their optimum resolution, with only a marginal loss in performance compared to the state-of-the-art. We achieve the best accuracy-speed trade-off using three key contributions. Many implicit methods extract features from the point cloud to classify whether a query point is inside or outside the object. First, to speed up the reconstruction, we show that this feature extraction does not need to use the query point at an early stage (lazy query). Second, we use a parallel multi-scale grid representation to develop robust features for different noise levels and input resolutions. Finally, we show that attention across scales can provide improved reconstruction results. The code will be made available.

Head pose estimation has become a crucial area of research in computer vision given its usefulness in a wide range of applications, including robotics, surveillance, or driver attention monitoring. One of the most difficult challenges in this field is managing head occlusions that frequently take place in real-world scenarios. In this paper, we propose a novel and efficient framework that is robust in real world head occlusion scenarios. In particular, we propose an unsupervised latent embedding clustering with regression and classification components for each pose angle. The model optimizes latent feature representations for occluded and non-occluded images through a clustering term while improving fine-grained angle predictions. Experimental evaluation on in-the-wild head pose benchmark datasets reveal competitive performance in comparison to state-of-the-art methodologies with the advantage of having a significant data reduction. We observe a substantial improvement in occluded head pose estimation. Also, an ablation study is conducted to ascertain the impact of the clustering term within our proposed framework.

Intelligent agents are showing increasing promise for clinical decision-making in a variety of healthcare settings. While a substantial body of work has contributed to the best strategies to convey these agents’ decisions to clinicians, few have considered the impact of personalizing and customizing these communications on the clinicians’ performance and receptiveness. This raises the question of how intelligent agents should adapt their tone in accordance with their target audience. We designed two approaches to communicate the decisions of an intelligent agent for breast cancer diagnosis with different tones: a suggestive (non-assertive) tone and an imposing (assertive) one. We used an intelligent agent to inform about: (1) number of detected findings; (2) cancer severity on each breast and per medical imaging modality; (3) visual scale representing severity estimates; (4) the sensitivity and specificity of the agent; and (5) clinical arguments of the patient, such as pathological co-variables. Our results demonstrate that assertiveness plays an important role in how this communication is perceived and its benefits. We show that personalizing assertiveness according to the professional experience of each clinician can reduce medical errors and increase satisfaction, bringing a novel perspective to the design of adaptive communication between intelligent agents and clinicians.

In this paper, we developed BreastScreening-AI within two scenarios for the classification of multimodal beast images: (1) Clinician-Only; and (2) Clinician-AI. The novelty relies on the introduction of a deep learning method into a real clinical workflow for medical imaging diagnosis. We attempt to address three high-level goals in the two above scenarios. Concretely, how clinicians: i) accept and interact with these systems, revealing whether are explanations and functionalities required; ii) are receptive to the introduction of AI-assisted systems, by providing benefits from mitigating the clinical error; and iii) are affected by the AI assistance. We conduct an extensive evaluation embracing the following experimental stages: (a) patient selection with different severities, (b) qualitative and quantitative analysis for the chosen patients under the two different scenarios. We address the high-level goals through a real-world case study of 45 clinicians from nine institutions. We compare the diagnostic and observe the superiority of the Clinician-AI scenario, as we obtained a decrease of 27% for False-Positives and 4% for False-Negatives. Through an extensive experimental study, we conclude that the proposed design techniques positively impact the expectations and perceptive satisfaction of 91% clinicians, while decreasing the time-to-diagnose by 3 min per patient.

Deep neural networks have been tremendously successful at segmenting objects in images. However, it has been shown they still have limitations on challenging problems such as the segmentation of medical images. The main reason behind this lower success resides in the reduced size of the object in the image. In this paper we overcome this limitation through a cyclic collaborative framework, CyCoSeg. The proposed framework is based on a deep active shape model (D-ASM), which provides prior information about the shape of the object, and a semantic segmentation network (SSN). These two models collaborate to reach the desired segmentation by influencing each other: SSN helps D-ASM identify relevant keypoints in the image through an Expectation Maximization formulation, while D-ASM provides a segmentation proposal that guides the SSN. This cycle is repeated until both models converge. Extensive experimental evaluation shows CyCoSeg boosts the performance of the baseline models, including several popular SSNs, while avoiding major architectural modifications. The effectiveness of our method is demonstrated on the left ventricle segmentation on two benchmark datasets, where our approach achieves one of the most competitive results in segmentation accuracy. Furthermore, its generalization is demonstrated for lungs and kidneys segmentation in CT scans.

Video analysis often requires locating and tracking target objects. In some applications, the localization system has access to the full video, which allows fine-grain motion information to be estimated. This paper proposes capturing this information through motion fields and using it to improve the localization results. The learned motion fields act as a model-agnostic temporal regularizer that can be used with any localization system based on keypoints. Unlike optical flow-based strategies, our motion fields are estimated from the model domain, based on the trajectories described by the object keypoints. Therefore, they are not affected by poor imaging conditions. The benefits of the proposed strategy are shown on three applications: 1) segmentation of cardiac magnetic resonance; 2) facial model alignment; and 3) vehicle tracking. In each case, combining popular localization methods with the proposed regularizer leads to improvement in overall accuracies and reduces gross errors.

This paper describes the field research, design and comparative deployment of a multimodal medical imaging user interface for breast screening. The main contributions described here are threefold: 1) The design of an advanced visual interface for multimodal diagnosis of breast cancer (BreastScreening); 2) Insights from the field comparison of Single-Modality vs Multi-Modality screening of breast cancer diagnosis with 31 clinicians and 566 images; and 3) The visualization of the two main types of breast lesions in the following image modalities: (i) MammoGraphy (MG) in both Craniocaudal (CC) and Mediolateral oblique (MLO) views; (ii) UltraSound (US); and (iii) Magnetic Resonance Imaging (MRI). We summarize our work with recommendations from the radiologists for guiding the future design of medical imaging interfaces.

We present 3DRegNet, a novel deep learning architecture for the registration of 3D scans. Given a set of 3D point correspondences, we build a deep neural network to address the following two challenges: (i) classification of the point correspondences into inliers/outliers, and (ii) regression of the motion parameters that align the scans into a common reference frame. With regard to regression, we present two alternative approaches: (i) a Deep Neural Network (DNN) registration and (ii) a Procrustes approach using SVD to estimate the transformation. Our correspondence-based approach achieves a higher speedup compared to competing baselines. We further propose the use of a refinement network, which consists of a smaller 3DRegNet as a refinement to improve the accuracy of the registration. Extensive experiments on two challenging datasets demonstrate that we outperform other methods and achieve state-of-the-art results.

This paper proposes a novel approach for the non-rigid segmentation of deformable objects in image sequences, which is based on one-shot segmentation that unifies rigid detection and non-rigid segmentation using elastic regularization. The domain of application is the segmentation of a visual object that temporally undergoes a rigid transformation (e.g., affine transformation) and a non-rigid transformation (i.e., contour deformation). The majority of segmentation approaches to solve this problem are generally based on two steps that run in sequence: a rigid detection, followed by a non-rigid segmentation. In this paper, we propose a new approach, where both the rigid and non-rigid segmentation are performed in a single shot using a sparse low-dimensional manifold that represents the visual object deformations. Given the multi-modality of these deformations, the manifold partitions the training data into several patches, where each patch provides a segmentation proposal during the inference process. These multiple segmentation proposals are merged using the classification results produced by deep belief networks (DBN) that compute the confidence on each segmentation proposal. Thus, an ensemble of DBN classifiers is used for estimating the final segmentation. Compared to current methods proposed in the field, our proposed approach is advantageous in four aspects: (i) it is a unified framework to produce rigid and non-rigid segmentations; (ii) it uses an ensemble classification process, which can help the segmentation robustness; (iii) it provides a significant reduction in terms of the number of dimensions of the rigid and non-rigid segmentations search spaces, compared to current approaches that divide these two problems; and (iv) this lower dimensionality of the search space can also reduce the need for large annotated training sets to be used for estimating the DBN models. Experiments on the problem of left ventricle endocardial segmentation from ultrasound images, and lip segmentation from frontal facial images using the extended Cohn-Kanade (CK+) database, demonstrate the potential of the methodology through qualitative and quantitative evaluations, and the ability to reduce the search and training complexities without a significant impact on the segmentation accuracy.

Object recognition and localization is still a very challenging problem, despite recent advances in deep learning (DL) approaches, especially for objects with varying shapes and appearances. Statistical models, such as an Active Shape Model (ASM), rely on a parametric model of the object, allowing an easy incorporation of prior knowledge about shape and appearance in a principled way. To take advantage of these benefits, this paper proposes a new ASM framework that addresses two tasks: (i) comparing the performance of several image features used to extract observations from an input image; and (ii) improving the performance of the model fitting by relying on a probabilistic framework that allows the use of multiple observations and is robust to the presence of outliers. The goal in (i) is to maximize the quality of the observations by exploring a wide set of handcrafted features (HOG, SIFT, and texture templates) and more recent DL-based features. Regarding (ii), we use the Generalized Expectation-Maximization algorithm to deal with outliers and to extend the fitting process to multiple observations. The proposed framework is evaluated in the context of facial landmark fitting and the segmentation of the endocardium of the left ventricle in cardiac magnetic resonance volumes. We experimentally observe that the proposed approach is robust not only to outliers, but also to adverse initialization conditions and to large search regions (from where the observations are extracted from the image). Furthermore, the results of the proposed combination of the ASM with DL-based features are competitive with more recent DL approaches (e.g. FCN [1], U-Net [2] and CNN Cascade [3]), showing that it is possible to combine the benefits of statistical models and DL into a new deep ASM probabilistic framework.

Providing reliable descriptions of the agents in a video scene is an essential task in many applications, such as surveillance. However, most works focus solely on the characterization of pedestrians, which is not sufficient to describe complex scenes, where a variety of vehicles (e.g., bikes and cars) are also present. In this work we address this limitation and propose a framework based on switching motion fields to efficiently characterize the different agents in a scene. Our method achieves a balanced accuracy of 91.9% on the identification of bikers and pedestrian classes on three challenging scenarios, and provides comprehensive information about their behaviors.

Pedestrian detection is one of the most explored topics in computer vision and robotics. The use of deep learning methods allowed the development of new and highly competitive algorithms. Deep Reinforcement Learning has proved to be within the state-of-the-art in terms of both detection in perspective cameras and robotics applications. However, for detection in omnidirectional cameras, the literature is still scarce, mostly because of their high levels of distortion. This paper presents a novel and efficient technique for robust pedestrian detection in omnidirectional images. The proposed method uses deep Reinforcement Learning that takes advantage of the distortion in the image. By considering the 3D bounding boxes and their distorted projections into the image, our method is able to provide the pedestrian’s position in the world, in contrast to the image positions provided by most state-of-the-art methods for perspective cameras. Our method avoids the need of pre-processing steps to remove the distortion, which is computationally expensive. Beyond the novel solution, our method compares favorably with the state-of-the-art methodologies that do not consider the underlying distortion for the detection task.

Obtaining the segmentation of an object in a sequence of images is usually achieved using a tracking methodology. However, in some applications, the whole sequence is available beforehand. This means that the segmentations can be determined simultaneously for all the frames in the sequence and taking into account the motion of the object. This paper proposes a new framework to incorporate motion information in the segmentation of image sequences using an active shape model (ASM). The motion of the object is modeled using a vector field, which is learned and refined online as the segmentation algorithm proceeds. The vector field is determined from the trajectories described by ASM points throughout the sequence. The vector field, in turn, influences the estimation of the ASM parameters by acting as a regularizer, ensuring that the segmentations are in agreement with the expected motion. The results show that coupling these models during the segmentation leads to an increase in performance, in particular by guarantee more consistent segmentations and by avoid gross errors in more challenging frames.

In this work, we present a context-aware ensemble fusion framework based on soft-biometric features, for long term person re-identification (Re-ID) in wild surveillance scenarios. The characteristics of a person that best correlate to its identity depend strongly on the view point. For instance, a person with a short stride gait is better perceived from a lateral view, whereas a person with a large chest is more distinct from a frontal view. Thus we associate context to the viewing direction of walking people in a surveillance scenario and choose the best features for each case. Using the MS KinectTM sensor v.2, we collect data from walking subjects and extract associated anthropometric and gait features. Each context is analysed with a Feature selection technique (Sequential Forward Selection) so that only the most relevant features for the context are retained. Then, individual context-specific classifiers are trained leveraging those selected features. Finally, we propose a contextaware ensemble fusion strategy, which we term as 'Contextspecific score-level fusion', based on the adaptive weighted sum of the results of individual classifiers. The proposed contextaware Re-ID framework demonstrate significant performance improvement both in terms of speed (up to 4.5 times faster) and accuracy (up to 17% rank-1 Re-ID rate) compared to the Context-unaware systems. From the study, we show that gait features are better for lateral views and anthropometric features are better for frontal views, confirming the results of previous studies.

This paper proposes a novel methodology to describe the trajectories performed by pedestrians in long-range surveillance scenarios. The proposed approach describes the trajectories by using sparse motion/vector fields together with a space-varying switching mechanism embedded in a Hidden Markov Model framework. Despite the diversity of motion patterns that may occur in a given scenario, the observed trajectories do not lie in the entire surveilled area. Instead, they are constrained to patterns corresponding to typical motions. To achieve a compact representation, we propose a sparse model estimated using the ℓ1norm applied to the log prior distribution of the vector fields. Experimental evaluation is conducted in real scenarios, and testify the usefulness of the proposed approach in modeling typical trajectories that occur in a far-field surveillance setup.

This paper proposes a new approach for the segmentation of the endocardium of the left ventricle using short axis magnetic resonance (MR) images. The proposed method comprises two main stages. First, each image is converted to polar coordinates, and an edge map is computed from the transformed image. Then, the contour of the left ventricle (LV) is estimated by computing the optimal path along the edge map, using a dynamic programming approach. The system is evaluated on a public database comprising 660 magnetic resonance volumes and the results testify its usefulness both in terms of running time and accuracy. The proposed methodology is able to segment a whole volume in 1.5 seconds achieving an average Dice similarity coefficient of 85.9% (8.3%), which compares favorably with related state-of-the-art methods.

A fundamental step in medical diagnosis for patient follow-up relies on the ability of radiologists to perform a trusty diagnostic from acquired images. Basically, the diagnosis strongly depends on the visual inspection over the shape of the lesions. As datasets increase in size, such visual evaluation becomes harder. For this reason, it is crucial to introduce easy-to-use interfaces that help the radiologists to perform a reliable visual inspection and allow the efficient delineation of the lesions. We will explore the radiologist's receptivity to the current touch environment solution. The advantages of touch are threefold: (i) the time performance is superior regarding the traditional use, (ii) it has more intuitive control and, (iii) for less time, the user interface delivers more information per action, concerning annotations. From our studies, we conclude that the radiologists still exhibit a resistance to change from traditional to touch based interfaces in current clinical setups.

We propose a new combination of deep belief networks and sparse manifold learning strategies for the 2D segmentation of non-rigid visual objects. With this novel combination, we aim to reduce the training and inference complexities while maintaining the accuracy of machine learning-based non-rigid segmentation methodologies. Typical non-rigid object segmentation methodologies divide the problem into a rigid detection followed by a non-rigid segmentation, where the low dimensionality of the rigid detection allows for a robust training (i.e., a training that does not require a vast amount of annotated images to estimate robust appearance and shape models) and a fast search process during inference. Therefore, it is desirable that the dimensionality of this rigid transformation space is as small as possible in order to enhance the advantages brought by the aforementioned division of the problem. In this paper, we propose the use of sparse manifolds to reduce the dimensionality of the rigid detection space. Furthermore, we propose the use of deep belief networks to allow for a training process that can produce robust appearance models without the need of large annotated training sets. We test our approach in the segmentation of the left ventricle of the heart from ultrasound images and lips from frontal face images. Our experiments show that the use of sparse manifolds and deep belief networks for the rigid detection stage leads to segmentation results that are as accurate as the current state of the art, but with lower search complexity and training processes that require a small amount of annotated training data.

We describe an automated methodology for the analysis of unregistered cranio-caudal (CC) and medio-lateral oblique (MLO) mammography views in order to estimate the patient's risk of developing breast cancer. The main innovation behind this methodology lies in the use of deep learning models for the problem of jointly classifying unregistered mammogram views and respective segmentation maps of breast lesions (i.e., masses and micro-calcifications). This is a holistic methodology that can classify a whole mammographic exam, containing the CC and MLO views and the segmentation maps, as opposed to the classification of individual lesions, which is the dominant approach in the field. We also demonstrate that the proposed system is capable of using the segmentation maps generated by automated mass and micro-calcification detection systems, and still producing accurate results. The semi-automated approach (using manually defined mass and micro-calcification segmentation maps) is tested on two publicly available data sets (INbreast and DDSM), and results show that the volume under ROC surface (VUS) for a 3-class problem (normal tissue, benign, and malignant) is over 0.9, the area under ROC curve (AUC) for the 2-class "benign versus malignant" problem is over 0.9, and for the 2-class breast screening problem (malignancy versus normal/benign) is also over 0.9. For the fully automated approach, the VUS results on INbreast is over 0.7, and the AUC for the 2-class "benign versus malignant" problem is over 0.78, and the AUC for the 2-class breast screening is 0.86.

The 3D segmentation of the left ventricle (LV) in cardiac MRI is a challenging problem, due to the presence of other anatomical structures and artifacts (outliers) around the LV. In this paper, a new formulation of a Robust Active Shape Model (RASM) is presented that is able to deal with those outliers. Instead of using the traditional one-to-one mapping of edge points and model points to compute the shape model parameters, the proposed approach uses a one-to-many mapping strategy and groups these edge points into edge segments (strokes). Then, a probabilistic framework provides a robust estimation of the model parameters, in which the influence in the segmentation of the unreliable outliers is reduced. The proposed method was tested on a public dataset comprising 660 volumes. The results indicate that this methodology provides accurate segmentations that are competitive with other state-of-the art methods.

The vast quantity of visual data generated by the rapid expansion of large scale distributed multicamera networks, makes automated person detection and reidentification (RE-ID) essential components of modern surveillance systems. However, the integration of automated person detection and RE-ID algorithms is not without problems, and the errors arising in this integration must be measured (e.g., detection failures that may hamper the RE-ID performance). In this paper, we present a window-based classifier based on a recently proposed architecture for the integration of pedestrian detectors and RE-ID algorithms, that takes the output of any bounding-box RE-ID classifier and exploits the temporal continuity of persons in video streams. We evaluate our contributions on a recently proposed dataset featuring 13 high-definition cameras and over 80 people, acquired during 30 min at rush hour in an office space scenario. We expect our contributions to drive research in integrated pedestrian detection and RE-ID systems, bringing them closer to practical applications.

Extraction of trajectories of moving objects from video is an important step towards activity classification in a surveillance system. It has been recently shown that multiple motion fields (MMF) estimated from trajectories can efficiently describe the movement of objects, and allow an automatic classification of activities in the scene. However, the object segmentation and tracking may introduce gaps in trajectories due to the sensor failures (i.e. miss detections) or pedestrian occlusions. This may hamper the performance of activity classification from the observed trajectories. In this paper, we propose a predictive motion model based on MMF to describe the trajectory of the object. Precomputed motion fields are used to predict the possible position of an object based on its location and trajectory up to that point and therefore, the method can deal with the case of a missed detection in one or more frames, thus able to perform trajectory completion. Experiments on real data show that the proposed method is remarkable in dealing with up to around 70% of the positions missing in the pedestrian trajectories.

The problem of non-rigid object segmentation is formulated in a two-stage approach in Machine Learning based methodologies. In the first stage, the automatic initialization problem is solved by the estimation of a rigid shape of the object. In the second stage, the non-rigid segmentation is performed. The rational behind this strategy, is that the rigid detection can be performed at lower dimensional space than the original contour space. In this paper, we explore this idea and propose the use of manifolds to reduce even more the dimensionality of the rigid transformation space (first stage) of current state-of-the-art top-down segmentation methodologies. Also, we propose the use of deep belief networks to allow for a training process capable to produce robust appearance models. Experiments in lips segmentation from frontal face images are conducted to testify the performance of the proposed algorithm.

Statistical shape models have been extensively used in a wide range of applications due to their effectiveness in providing prior shape information for object segmentation problems. The most popular method is the Active Shape Model (ASM). However, accurately fitting the shape model to an object boundary under a cluttered environment is a challenging task. Under such assumptions, the model is often attracted towards invalid observations (outliers), leading to meaningless estimates of the object boundary. In this paper, we propose a novel algorithm that improves the robustness of ASM in the presence of outliers. The proposed framework assumes that both type of observations (valid observations and outliers) are detected in the image. A new strategy is devised for treating the data in different ways, depending on the observations being considered as valid or invalid. The proposed algorithm assigns a different weight to each observation. The shape parameters are recursively updated using the Expectation-Maximization method, allowing a correct and robust fit of the shape model to the object boundary in the image. Two estimation criteria are considered: 1) the maximum likelihood criterion; and 2) the maximum a posteriori criterion that uses priors for the unknown parameters. The methods are tested with synthetic and real images, comprising medical images of the heart and image sequences of the lips. The results are promising and show that this approach is robust in the presence of outliers, leading to a significant improvement over the standard ASM and other state of the art methods.

The segmentation of the left ventricle (LV) is an important task to assess the cardiac function in ultrasound images of the heart. This paper presents a novel methodology for the segmentation of the LV in three-dimensional (3-D) echocardiographic images based on the probabilistic data association filter (PDAF). The proposed methodology begins by initializing a 3-D deformable model either semiautomatically, with user input, or automatically, and it comprises the following feature hierarchical approach: 1) edge detection in the vicinity of the surface (low-level features); 2) edge grouping to obtain potential LV surface patches (mid-level features); and 3) patch filtering using a shape-PDAF framework (high-level features). This method provides good performance accuracy in 20 echocardiographic volumes, and compares favorably with the state-of-the-art segmentation methodologies proposed in the recent literature.

In this paper, we propose a new methodology for segmenting non-rigid visual objects, where the search procedure is onducted directly on a sparse low-dimensional manifold, guided by the classification results computed from a deep belief network. Our main contribution is the fact that we do not rely on the typical sub-division of segmentation tasks into rigid detection and non-rigid delineation. Instead, the non-rigid segmentation is performed directly, where points in the sparse low-dimensional can be mapped to an explicit contour representation in image space. Our proposal shows significantly smaller search and training complexities given that the dimensionality of the manifold is much smaller than the dimensionality of the search spaces for rigid detection and non-rigid delineation aforementioned, and that we no longer require a two-stage segmentation process. We focus on the problem of left ventricle endocardial segmentation from ultrasound images, and lip segmentation from frontal facial images using the extended Cohn-Kanade (CK+) database. Our experiments show that the use of sparse low dimensional manifolds reduces the search and training complexities of current segmentation approaches without a significant impact on the segmentation accuracy shown by state-of-the-art approaches.

Active shape models (ASM) have been extensively used in object segmentation problems because they constrain the solution, using shape statistics. However, accurately fitting an ASM to an image prone to outliers is difficult and poor results are often obtained. To overcome this difficulty we propose a robust algorithm based on the Expectation-Maximization framework that assigns different weights (confidence degrees) to the observations extracted from the image. This reduces the influence of outliers since they often receive low weights. We tested the proposed algorithm with synthetic and real images (e.g., lip images and cardiac ultrasound images) achieving promising results. The proposed algorithm performs significantly better than the standard ASM implementation.

This paper proposes an iterative natural gradient algorithm to perform the optimization of switching probabilities in a space-varying hidden Markov model, in the context of human activity recognition in long-range surveillance. The proposed method is a version of the gradient method, developed under an information geometric viewpoint, where the usual Euclidean metric is replaced by a Riemannian metric on the space of transition probabilities. It is shown that the change in metric provides advantages over more traditional approaches, namely: 1) it turns the original constrained optimization into an unconstrained optimization problem; 2) the optimization behaves asymptotically as a Newton method and yields faster convergence than other methods for the same computational complexity; and 3) the natural gradient vector is an actual contravariant vector on the space of probability distributions for which an interpretation as the steepest descent direction is formally correct. Experiments on synthetic and real-world problems, focused on human activity recognition in long-range surveillance settings, show that the proposed methodology compares favorably with the state-of-the-art algorithms developed for the same purpose.

This paper presents a novel manifold learning algorithm for high-dimensional data sets. The scope of the application focuses on the problem of motion tracking in video sequences. The framework presented is twofold. First, it is assumed that the samples are time ordered, providing valuable information that is not presented in the current methodologies. Second, the manifold topology comprises multiple charts, which contrasts to the most current methods that assume one single chart, being overly restrictive. The proposed algorithm, Gaussian process multiple local models (GP-MLM), can deal with arbitrary manifold topology by decomposing the manifold into multiple local models that are probabilistic combined using Gaussian process regression. In addition, the paper presents a multiple filter architecture where standard filtering techniques are integrated within the GP-MLM. The proposed approach exhibits comparable performance of state-of-the-art trackers, namely multiple model data association and deep belief networks, and compares favorably with Gaussian process latent variable models. Extensive experiments are presented using real video data, including a publicly available database of lip sequences and left ventricle ultrasound images, in which the GP-MLM achieves state of the art results.

Estimation of velocity fields from a video sequence is an important step towards activity classification in a surveillance system. It has been recently shown that multiple motion fields estimated from trajectories are an efficient tool to describe the movement of objects, allowing an automatic classification of activities in the scene. However, the trajectory detection in noisy environments is difficult, usually requiring some sort manual editing to complete or correct them. This paper proposes two novel contributions. First, an automatic method for building pedestrian trajectories in far-field surveillance scenarios is presented not requiring user intervention. This basically comprises the detection of multiple moving objects in a video sequence through the detection of the active regions, followed by the estimation of the velocity fields that is accomplished by performing region matching of the above regions at consecutive time instants. This leads to a sequence of centroids and corresponding velocity vectors, describing the local motions presented in the image. A motion correspondence algorithm is then applied to group the centroids in a contiguous sequence of frames into trajectories corresponding to each moving object. The second contribution is a method for automatically finding the trajectories from a library of previously computed ones. Experiments on extensive video sequences from university campuses show that motion fields can be reliably estimated from these automatically detected trajectories, leading to a fully automatic procedure for the estimation of multiple motion fields.

The solution for the top-down segmentation of non rigid visual objects using machine learning techniques is generally regarded as too complex to be solved in its full generality given the large dimensionality of the search space of the explicit representation of the segmentation contour. In order to reduce this complexity, the problem is usually divided into two stages: rigid detection and non-rigid segmentation. The rationale is based on the fact that the rigid detection can be run in a lower dimensionality space (i.e., less complex and faster) than the original contour space, and its result is then used to constrain the non-rigid segmentation. In this paper, we propose the use of sparse manifolds to reduce the dimensionality of the rigid detection search space of current state-of-the-art top-down segmentation methodologies. The main goals targeted by this smaller dimensionality search space are the decrease of the search running time complexity and the reduction of the training complexity of the rigid detector. These goals are attainable given that both the search and training complexities are function of the dimensionality of the rigid search space. We test our approach in the segmentation of the left ventricle from ultrasound images and lips from frontal face images. Compared to the performance of state-of-the-art non-rigid segmentation system, our experiments show that the use of sparse manifolds for the rigid detection leads to the two goals mentioned above.

The segmentation of anatomical structures is a crucial first stage of most medical imaging analysis procedures. A primary example is the segmentation of the left ventricle (LV), from cardiac imagery. Accuracy in the segmentation often requires a considerable amount of expert intervention and guidance which are expensive. Thus, automating the segmentation is welcome, but difficult because of the LV shape variability within and across individuals. To cope with this difficulty, the algorithm should have the skills to interpret the shape of the anatomical structure (i.e. LV shape) using distinct kinds of information, (i.e. different views of the same feature space). These different views will ascribe to the algorithm a more general capability that surely allows for the robustness in the segmentation accuracy. In this paper, we propose an on-line co-training algorithm using a bottom-up and top-down classifiers (each one having a different view of the data) to perform the segmentation of the LV. In particular, we consider a setting in which the LV shape can be partitioned into two distinct views and use a co-training as a way to boost each of the classifiers, thus providing a principled way to use both views together. We testify the usefulness of the approach on a public data base illustrating that the approach compares favorably with other recent proposed methodologies.

The segmentation of the left ventricle (LV) is an important tool to assess the cardiac function in ultrasound images of the heart. This paper presents a methodology for the segmentation of the LV in 3D echocardiography that is based on the probabilistic data association filter (PDAF). The proposed methodology comprises the following feature hierarchical approach: (i) edge detection in the vicinity of the surface (low level features); (ii) edge grouping to obtain potential LV surface patches (mid-level features); and (iii) patch filtering using a shape-PDAF framework (high level features). Also, we propose an automatic procedure to initialize the 3D deformable model. We show that the proposed methodology achieves remarkable accuracy between the obtained contour and expertise medical ground truth and compares favorably with the state-of-the-art segmentation methodologies.

We present a new statistical pattern recognition approach for the problem of left ventricle endocardium tracking in ultrasound data. The problem is formulated as a sequential importance resampling algorithm such that the expected segmentation of the current time step is estimated based on the appearance, shape, and motion models that take into account all previous and current images and previous segmentation contours produced by the method. The new appearance and shape models decouple the affine and nonrigid segmentations of the left ventricle to reduce the running time complexity. The proposed motion model combines the systole and diastole motion patterns and an observation distribution built by a deep neural network. The functionality of our approach is evaluated using a dataset of diseased cases containing 16 sequences and another dataset of normal cases comprised of four sequences, where both sets present long axis views of the left ventricle. Using a training set comprised of diseased and healthy cases, we show that our approach produces more accurate results than current state-of-the-art endocardium tracking methods in two test sequences from healthy subjects. Using three test sequences containing different types of cardiopathies, we show that our method correlates well with interuser statistics produced by four cardiologists.

The analysis of moving objects in image sequences (video) has been one of the major themes in computer vision. In this paper, we focus on video-surveillance tasks; more specifically, we consider pedestrian trajectories and propose modeling them through a small set of motion/vector fields together with a space-varying switching mechanism. Despite the diversity of motion patterns that can occur in a given scene, we show that it is often possible to find a relatively small number of typical behaviors, and model each of these behaviors by a "simple" motion field. We increase the expressiveness of the formulation by allowing the trajectories to switch from one motion field to another, in a space-dependent manner. We present an expectation-maximization algorithm to learn all the parameters of the model, and apply it to trajectory classification tasks. Experiments with both synthetic and real data support the claims about the performance of the proposed approach.

Many approaches to trajectory analysis, such as clustering or classification, use probabilistic generative models, thus not requiring trajectory alignment/registration. Switched linear dynamical models (e.g., HMMs) have been used in this context, due to their ability to describe different motion regimes. However, these models are not suitable for handling space-dependent dynamics that are more naturally captured by nonlinear models. As is well known, these are more difficult to identify. In this paper, we propose a new way of modeling trajectories, based on a mixture of parametric motion vector fields that depend on a small number of parameters. Switching among these fields follows a probabilistic mechanism, characterized by a field of stochastic matrices. This approach allows representing a wide variety of trajectories and modeling space-dependent behaviors without using global nonlinear dynamical models. Experimental evaluation is conducted in both synthetic and real scenarios. The latter concerning with human trajectory modeling for activity classification, a central task in video surveillance.

The use of statistical pattern recognition models to segment the left ventricle of the heart in ultrasound images has gained substantial attention over the last few years. The main obstacle for the wider exploration of this methodology lies in the need for large annotated training sets, which are used for the estimation of the statistical model parameters. In this paper, we present a new on-line co-training methodologythat reduces the need for large training sets for such parameter estimation. Our approach learns the initial parameters of two different models using a small manually annotated training set. Then, given each frame of a test sequence, the methodology not only produces the segmentation of the current frame, but it also uses the results of both classifiers to retrain each other incrementally. This on-line aspect of our approach has the advantages of producing segmentation results and retraining the classifiers on the fly as frames of a test sequence are presented, but it introduces a harder learning setting compared to the usual off-line co-training, where the algorithm has access to the whole set of un-annotated training samples from the beginning. Moreover, we introduce the use of the following new types of classifiers in the co-training framework: deep belief network and multiple model probabilistic data association. We show that our method leads to a fully automatic left ventricle segmentation system that achieves state-of-the-art accuracy on a public database with training sets containing at least twenty annotated images.

The segmentation of the left ventricle (LV) still constitutes an active research topic in medical image processing field. The problem is usually tackled using pattern recognition methodologies. The main difficulty with pattern recognition methods is its dependence of a large manually annotated training sets for a robust learning strategy. However, in medical imaging, it is difficult to obtain such large annotated data. In this paper, we propose an on-line semi-supervised algorithm capable of reducing the need of large training sets. The main difference regarding semi-supervised techniques is that, the proposed framework provides both an on-line retraining and segmentation, instead of on-line retraining and off-line segmentation. Our proposal is applied to a fully automatic LV segmentation with substantially reduced training sets while maintaining good segmentation accuracy.

Video cameras monitoring human activities in public spaces are commonplace in cities worldwide. Such monitoring task is important for safety and security purposes but is also extremely challenging. In this paper, we propose a class of algorithms for far-field human activity recognition, a central task in video surveillance. More specifically, we explore a class of parametric motion vector fields learned from the trajectories described by people in real-world scenarios. The work proposed herein is a space dependent framework, in sense that the vector fields depend on the pedestrian position. Thus, the model is flexible leading to an expressive description of complex trajectories. Also, a model selection strategy is addressed to automatically choose the appropriate number of underlying motion fields presented in the trajectories. Experimental evaluation is conducted in real settings testifying the usefulness of the proposed approach for human activity recognition.

We present a new supervised learning model designed for the automatic segmentation of the left ventricle (LV) of the heart in ultrasound images. We address the following problems inherent to supervised learning models: 1) the need of a large set of training images; 2) robustness to imaging conditions not present in the training data; and 3) complex search process. The innovations of our approach reside in a formulation that decouples the rigid and nonrigid detections, deep learning methods that model the appearance of the LV, and efficient derivative-based search algorithms. The functionality of our approach is evaluated using a data set of diseased cases containing 400 annotated images (from 12 sequences) and another data set of normal cases comprising 80 annotated images (from two sequences), where both sets present long axis views of the LV. Using several error measures to compute the degree of similarity between the manual and automatic segmentations, we show that our method not only has high sensitivity and specificity but also presents variations with respect to a gold standard (computed from the manual annotations of two experts) within interuser variability on a subset of the diseased cases. We also compare the segmentations produced by our approach and by two state-of-the-art LV segmentation models on the data set of normal cases, and the results show that our approach produces segmentations that are comparable to these two approaches using only 20 training images and increasing the training set to 400 images causes our approach to be generally more accurate. Finally, we show that efficient search methods reduce up to tenfold the complexity of the method while still producing competitive segmentations. In the future, we plan to include a dynamical model to improve the performance of the algorithm, to use semisupervised learning methods to reduce even more the dependence on rich and large training sets, and to design a shape model less dependent on the training set.

Recently, there has been an increasing interest in the investigation of statistical pattern recognition models for the fully automatic segmentation of the left ventricle (LV) of the heart from ultrasound data. The main vulnerability of these models resides in the need of large manually annotated training sets for the parameter estimation procedure. The issue is that these training sets need to be annotated by clinicians, which makes this training set acquisition process quite expensive. Therefore, reducing the dependence on large training sets is important for a more extensive exploration of statistical models in the LV segmentation problem. In this paper, we present a novel incremental on-line semi-supervised learning model that reduces the need of large training sets for estimating the parameters of statistical models. Compared to other semi-supervised techniques, our method yields an on-line incremental re-training and segmentation instead of the off-line incremental re-training and segmentation more commonly found in the literature. Another innovation of our approach is that we use a statistical model based on deep learning architectures, which are easily adapted to this on-line incremental learning framework. We show that our fully automatic LV segmentation method achieves state-of-the-art accuracy with training sets containing less than twenty annotated images.

Statistical pattern recognition models are one of the core research topics in the segmentation of the left ventricle of the heart from ultrasound data. The underlying statistical model usually relies on a complex model for the shape and appearance of the left ventricle whose parameters can be learned using a manually segmented data set. Unfortunately, this complex requires a large number of parameters that can be robustly learned only if the training set is sufficiently large. The difficulty in obtaining large training sets is currently a major roadblock for the further exploration of statistical models in medical image analysis. In this paper, we present a novel semi-supervised self-training model that reduces the need of large training sets for estimating the parameters of statistical models. This model is initially trained with a small set of manually segmented images, and for each new test sequence, the system re-estimates the model parameters incrementally without any further manual intervention. We show that state-of-the-art segmentation results can be achieved with training sets containing 50 annotated examples.

In this paper we introduce a novel method to determine the model order of a stochastic model for moving objects. The main assumption is that we make use of the knowledge that the obtained model is going to be used for some task, specifically, for trajectory classification. Particularly, the object motion is described by trajectories performed by the objects (e.g., pedestrians), during their motion, by representing them by a small and meaningful mixtures of vector fields. We present a discriminative method for model selection without resort to computationally expensive cross-validation procedures. The idea is, thus, to select the generative model achieving the best classification performance. Although the topic of application is video surveillance, the proposed method can easily be extended to other practical situations. Experiments with both synthetic and real data concerning pedestrian activities illustrate the performance of the proposed approach.

Many approaches to trajectory analysis tasks (such as clustering or classification) use probabilistic generative models, thus not requiring trajectory alignment/registration. Switched linear dynamical models (e.g., HMMs) have been used in this context, due to their ability to describe different motion regimes. However, this type of models is not suitable for handling space-dependent dynamics, that are more naturally captured by non-linear models. As is well known, these are more difficult to identify. We propose a new way of modeling trajectories, based on a mixture of parametric motion vector fields that depend on a small number of parameters. Switching among these fields follows a probabilistic mechanism, characterized by a field of stochastic matrices. This approach allows representing a wide variety of trajectories and modeling space-dependent behaviors without using global non-linear dynamical models. The proposed model is applied to human trajectory modeling, a central task in video surveillance.

This paper presents a method for tracking non-rigid lip contours on video sequences, when the subject exhibits different emotions. The method is based on multiple dynamic models, which are well suited for the multiple-emotion case. Nonlinear dimensionality reduction is performed using the Gaussian Process Multiple Local Models manifold learning method, taking advantage of the time-order of the samples which provides valuable time information for tracking purposes. The method uses multiple charts which allows arbitrary manifold topology. This is accomplished by decomposing the manifold into multiple local models that are combined in a probabilistic fashion using Gaussian process regression. Furthermore, a multiple filter bank architecture is applied in the reduced-dimensionality manifold domain, based on standard filtering methods (e.g., Kalman and particle filtering). The performance of this approach is illustrated in the extended Cohn-Kanade (CK+) database, where the method achieves remarkable accuracy in lip contour tracking with a wide range of emotions.

The problem of automatic tracking and segmentation of the left ventricle (LV) of the heart from ultrasound images can be formulated with an algorithm that computes the expected segmentation value in the current time step given all previous and current observations using a filtering distribution. This filtering distribution depends on the observation and transition models, and since it is hard to compute the expected value using the whole parameter space of segmentations, one has to resort to Monte Carlo sampling techniques to compute the expected segmentation parameters. Generally, it is straightforward to compute probability values using the filtering distribution, but it is hard to sample from it, which indicates the need to use a proposal distribution to provide an easier sampling method. In order to be useful, this proposal distribution must be carefully designed to represent a reasonable approximation for the filtering distribution. In this paper, we introduce a new LV tracking and segmentation algorithm based on the method described above, where our contributions are focused on a new transition and observation models, and a new proposal distribution. Our tracking and segmentation algorithm achieves better overall results on a previously tested dataset used as a benchmark by the current state-of-the-art tracking algorithms of the left ventricle of the heart from ultrasound images.

Pattern recognition methods have become a powerful tool for segmentation in the sense that they are capable of automatically building a segmentation model from training images. However, they present several difficulties, such as requirement of a large set of training data, robustness to imaging conditions not present in the training set, and complexity of the search process. In this paper we tackle the second problem by using a deep belief network learning architecture, and the third problem by resorting to efficient searching algorithms. As an example, we illustrate the performance of the algorithm in lip segmentation and tracking in video sequences. Quantitative comparison using different strategies for the search process are presented. We also compare our approach to a state-of-the-art segmentation and tracking algorithm. The comparison show that our algorithm produces competitive segmentation results and that efficient search strategies reduce ten times the run-complexity.

The gradient vector flow (GVF) algorithm has been extensively used as a tool to extract the boundary of objects in images. However, its performance significantly degrades when the images have a cluttered background i.e., other structures in the vicinity of the object of interest. In this case, the elastic contour is often attracted towards invalid features. This paper discusses the robustness of GVF and proposes a way to improve its performance. The main difference concerns the use of more robust and informative features (middle level features) which significantly reduce the influence of noise. Experiments are presented to illustrate the performance of the improved GVF algorithm.

We propose a method to classify human trajectories, modeled by a set of motion vector fields, each tailored to describe a specific motion regime. Trajectories are modeled as being composed of segments corresponding to different motion regimes, each generated by one of the underlying motion fields. Switching among the motion fields follows a probabilistic mechanism, described by a field of stochastic matrices. This yields a space-dependent motion model which can be estimated using an expectation-maximization (EM) algorithm. To address the model selection question (how many fields to use?), we adopt a discriminative criterion based on classification accuracy on a held out set. Experiments with real data (human trajectories in a shopping mall) illustrate the ability of the proposed approach to classify complex trajectories into high level classes (client versus non-client).

A central issue in mixture-type models is the determination of a suitable number of components that best suits the observed data. In this paper, we address this issue in the context of trajectory classification based on mixtures of motion vector fields. We adopt a discriminative criterion for choosing among alternative models for each class, based on the classification accuracy on a held out dataset. The key idea is that we make use of the knowledge that the obtained model is going to be used for a specific task: classification. Experiments with both synthetic and real data concerning pedestrian activity classification illustrate the performance of the adopted criterion.

This work introduces a new pattern recognition model for segmenting and tracking lip contours in video sequences. We formulate the problem as a general nonrigid object tracking method, where the computation of the expected segmentation is based on a filtering distribution. This is a difficult task because one has to compute the expected value using the whole parameter space of segmentation. As a result, we compute the expected segmentation using sequential Monte Carlo sampling methods, where the filtering distribution is approximated with a proposal distribution to be used for sampling. The key contribution of this paper is the formulation of this proposal distribution using a new observation model based on deep belief networks and a new transition model. The efficacy of the model is demonstrated in publicly available databases of video sequences of people talking and singing. Our method produces results comparable to state-of-the-art models, but showing potential to be more robust to imaging conditions.

This paper proposes an approach for recognizing human activities (more specifically, pedestrian trajectories) in video sequences, in a surveillance context. A system for automatic processing of video information for surveillance purposes should be capable of detecting, recognizing, and collecting statistics of human activity, reducing human intervention as much as possible. In the method described in this paper, human trajectories are modeled as a concatenation of segments produced by a set of low level dynamical models. These low level models are estimated in an unsupervised fashion, based on a finite mixture formulation, using the expectation-maximization (EM) algorithm; the number of models is automatically obtained using a minimum message length (MML) criterion. This leads to a parsimonious set of models tuned to the complexity of the scene. We describe the switching among the low-level dynamic models by a hidden Markov chain; thus, the complete model is termed a switched dynamical hidden Markov model (SD-HMM). The performance of the proposed method is illustrated with real data from two different scenarios: a shopping center and a university campus. A set of human activities in both scenarios is successfully recognized by the proposed system. These experiments show the ability of our approach to properly describe trajectories with sudden changes.

This work introduces a new approach to modeling object trajectories in image sequences. Trajectories performed by natural objects (e.g., people, animals) typically depend on the position of each object in the scene and can change in an unpredictable way. Despite this diversity, there is often a small number of typical motion patterns based on which it is possible to explain all the observed trajectories. To achieve this goal, we model each of these motion patterns using a motion field and allow objects to switch between fields in a space-varying, possible probabilistic, way. Our approach provides a space-dependent motion model which can be estimated using an expectation-maximization (EM) algorithm. Experiments with both synthetic and real data are presented to illustrate the ability of the proposed approach in modeling different motion patterns.

This paper presents a method for 3D model-based tracking of colored objects using a sampling methodology. The problem is formulated in a Monte Carlo filtering approach, whereby the state of an object is represented by a set of hypotheses. The main originality of this work is an observation model consisting in the comparison of the color information in some sampling points around the target’s hypothetical edges. On the contrary to existing approaches the method does not need to explicitly compute edges in the video stream, thus dealing well with optical or motion blur. The method does not require the projection of the full 3D object on the image, but just of some selected points around the target’s boundaries. This allows a flexible and modular architecture illustrated by experiments performed with different objects (balls and boxes), camera models (perspective, catadioptric, dioptric) and tracking methodologies (particle and Kalman filtering). 1

This work introduces a non-supervised algorithm for learning generative models for classification/recognition of human activities (specifically, pedestrian trajectories) with application to video surveillance. The proposed algorithm comprises two main features: (?) a set of low level dynamical models of the trajectories, estimated in unsupervised manner using the expectation-maximization (EM) algorithm and automatic model selection using the minimum message length (MML) criterion; (ii) a switching dynamical model described by an hidden Markov model (HMM) used to characterize the higher level activities. The hierarchical model with these two levels is herein denoted as switched dynamical hidden Markov model (SD-HMM). We illustrate the performance of the proposed technique for human activity recognition in a university campus.

The segmentation of the heart's left ventricle (LV) chamber in several medical imaging modalities, e.g. Ultrasound (US) said Magnetic Resonance (MRI), is important from a clinical point of view in the diagnosis of certain cardiopathies. Manual segmentation is difficult, not accurate and time consuming. Therefore, automatic segmentation and tracking during cardiac cycles is needed. In this paper an automatic algorithm to segment the LV boundary along a cardiac cycle from ultrasound image sequences is used and a Bayesian despeckling algorithm is proposed. The prior parameter of the Bayesian filter is automatically estimated and an automatic window size selection strategy in used to adapt its dimension to the statistical characteristics of the image in the vicinity of the deformable contour model which segments the LV boundary. Sequences of real ultrasound images are used to illustrate the effectiveness of the approach and a comparison with other state-of-the-art filtering algorithms is provided.

Geometric active contours based on edges perform poorly in the presence of noise or clutter. When the edges have gaps or are indistinct, the contour leaks through the boundary. Furthermore, when spurious edge points that do not belong to the object are present in the image, the contour is stopped by them and either does not converge to the object boundary or there is oversegmentation. This paper addresses the second difficulty. We propose a novel technique which classifies image features as valid or invalid making the curve stop only at valid features and allowing it to bridge the invalid ones. This is incorporated in the stopping force of boundary based level sets, achieving a robust contour estimation. Our algorithm organizes edge points into connected segments (denoted herein as strokes) and classifies each segment as valid or invalid. A confidence degree (weight) is assigned to each stroke and updated during the evolution process. Thus, the proposed stopping force is adaptive. Experimental results with real data will be provided to illustrate the performance of the proposed algorithm.

This paper addresses object tracking in ultrasound images using a robust multiple model tracker. The proposed tracker has the following features: 1) it uses multiple dynamic models to track the evolution of the object boundary, and 2) it models invalid observations (outliers), reducing their influence on the shape estimates. The problem considered in this paper is the tracking of the left ventricle which is known to be a challenging problem. The heart motion presents two phases (diastole and systole) with different dynamics, the multiple models used in this tracker try to solve this difficulty. In addition, ultrasound images are corrupted by strong multiplicative noise which prevents the use of standard deformable models. Robust estimation techniques are used to address this difficulty. The multiple model data association (MMDA) tracker proposed in this paper is based on a bank of nonlinear filters, organized in a tree structure. The algorithm determines which model is active at each instant of time and updates its state by propagating the probability distribution, using robust estimation techniques.

Multiplicative noise is often present in medical and biological imaging, such as magnetic resonance imaging (MRI), Ultrasound, positron emission tomography (PET), single photon emission computed tomography (SPECT), and fluorescence microscopy. Noise reduction in medical images is a difficult task in which linear filtering algorithms usually fail. Bayesian algorithms have been used with success but they are time consuming and computationally demanding. In addition, the increasing importance of the 3-D and 4-D medical image analysis in medical diagnosis procedures increases the amount of data that must be efficiently processed. This paper presents a Bayesian denoising algorithm which copes with additive white Gaussian and multiplicative noise described by Poisson and Rayleigh distributions. The algorithm is based on the maximum a posteriori (MAP) criterion, and edge preserving priors which avoid the distortion of relevant anatomical details. The main contribution of the paper is the unification of a set of Bayesian denoising algorithms for additive and multiplicative noise using a well-known mathematical framework, the Sylvester-Lyapunov equation, developed in the context of the Control theory.

This work introduces a semi-supervised approach for learning generative models for classification/recognition of human trajectories, with application to surveillance. The classifier is based on switched dynamical models, with each model describing a specific motion regime. We present a semi-supervised modified version of the classical Baum-Welch algorithm, which is able to take into account a subset of known model labels. The experimental results reported, using both synthetic and real data, show that the classifier learned with semi-supervision leads to a higher classification accuracy than the fully unsupervised version, thus validating the proposed approach.

We present a model-based 3D tracking system, using wide angle perspective catadioptric sensors. These sensors acquire 360deg views of the environment and the projection from 3D world points to the image plane is approximated by a perspective model. This is a major advantage in structured environments because straight lines on specific surfaces are not deformed by the sensor, allowing the application of standard computer vision algorithms. Objects off the surface are distorted according to a complex projection model, but can be approximated by a simple wide angle perspective mapping. This is exploited here to develop a robust tracking system for autonomous robots using a 3D shape and color-based object model. The use of particle filters allows tracking to be done with 3D realistic motion models and tackling object occlusion, overlap and ambiguities. We show that the use of the perspective model is advantageous over more standard catadioptric projection models, since it renders a very good approximation to the true model, being simpler and more efficient to use, in particular with 3D particle filtering methods.

In this paper, we propose novel methods to evaluate the performance of object detection algorithms in video sequences. This procedure allows us to highlight characteristics (e.g., region splitting or merging) which are specific of the method being used. The proposed framework compares the output of the algorithm with the ground truth and measures the differences according to objective metrics. In this way it is possible to perform a fair comparison among different methods, evaluating their strengths and weaknesses and allowing the user to perform a reliable choice of the best method for a specific application. We apply this methodology to segmentation algorithms recently proposed and describe their performance. These methods were evaluated in order to assess how well they can detect moving regions in an outdoor scene in fixed-camera situations

This paper describes a new algorithm for the recognition of human activities. These activities are modelled using banks of switched dynamical models, each of which is tailored to a specific motion regime. Furthermore, it is assumed that model switching happens according to a space-dependent Markov chain, i.e., some transitions are more probable in specific regions of the image. Space dependence allows the model to represent the interaction between the person and static elements of the scene. The paper describes learning algorithms for space-dependent switched dynamical models and presents experimental results with synthetic and real data.

Deformable models (e.g., snakes) perform poorly in many image analysis problems. The contour model is attracted by edge points detected in the image. However, many edge points do not belong to the object contour, preventing the active contour from converging toward the object boundary. A new algorithm is proposed in this paper to overcome this difficulty. The algorithm is based on two key ideas. First, edge points are associated in strokes. Second, each stroke is classified as valid (inlier) or invalid (outlier) and a confidence degree is associated to each stroke. The expectation maximization algorithm is used to update the confidence degrees and to estimate the object contour. It is shown that this is equivalent to the use of an adaptive potential function which varies during the optimization process. Valid strokes receive high confidence degrees while confidence degrees of invalid strokes tend to zero during the optimization process. Experimental results are presented to illustrate the performance of the proposed algorithm in the presence of clutter, showing a remarkable robustness.

Many object-tracking algorithms are based on low-level features detected in the image. Typically, the object shape and position are estimated to fit the observed features. Unfortunately, image analysis methods often produce invalid features (outliers) which do not belong to the object boundary. These features have a strong influence on the shape estimates, leading to meaningless tracking results. This paper proposes a robust tracking algorithm which is able to deal with outliers, inspired in the probabilistic data association filter proposed in the context of point tracking. The algorithm is based on two key concepts. First, middle level features (strokes) are used instead of low-level ones (edge points). Second, two labels (valid/invalid) are considered for each stroke. Since the stroke labels are unknown all labeling sequences are considered and a probability (confidence degree) is assigned to each of them. In this way, all the strokes contribute to track the moving object but with different weights. This allows a robust performance of the tracker in the presence of outliers. Experimental tests are provided to assess the performance of the proposed algorithm in lip and gesture tracking and surveillance applications.

This paper presents an algorithm for tracking the left ventricle in echocardiographic sequences, using multiple models. The use of multiple dynamic models is appropriate since the heart motion presents two phases (diastole and systole) with different dynamics. The main difficulty concerns the low contrast and speckle noise present in ultrasound images. To overcome this problem a robust multiple model tracker is used, based on a bank of nonlinear filters, organized in a tree structure. This algorithm determines which model is active at each instant of time and updates its state by propagating the probability distribution, using robust estimation techniques. It is shown in the paper that the proposed algorithm simultaneously copes with several dynamic models and with outliers. Furthermore the proposed algorithm provides high level information that is not available when a single model is used.

Abstract This paper describes an algorithm for shape estimation in cluttered scenes. A new image potential is defined based on strokes detected in the image. The motivation is simple. Feature detectors (e.g. edge points detectors) produce many outliers, which hamper the performance of boundary extraction algorithms. To overcome this difficulty we organize edges in strokes and assign a confidence degree (weight) to each stroke. The confidence degrees depend on the distance of the stroke points to the boundary estimates and they are updated during the estimation process. A deformable model is used to estimate the object boundary, based on the minimization of an adaptive potential function which depends on the confidence degree assigned to each stroke. Therefore, the image potential changes during the estimation process. Both steps (weight update, energy minimization) are derived as the solution of a maximum likelihood estimation problem using the EM algorithm. Experimental tests are provided to illustrate the performance of the proposed algorithm.

Kalman filtering has been extensively used in object tracking. However, the tracker performance is severely affected in the presence of multiple objects and cluttered background. The reason is simple. Feature detection produces many outliers and the Kalman filter is not able to discriminate valid data from the clutter. This paper overcome this difficulty and describes a robust algorithm for object tracking denoted as S-PDAF (shape-probabilistic data association filter). Experimental tests show that significant robustness improvement is achieved by the S-PDAF algorithm.

This article addresses the problem of tracking moving objects using deformable models. A Kalman-based algorithm is presented, inspired by a new class of constrained clustering methods, proposed by Abrantes and Marques (1996) in the context of static shape estimation. A set of data centroids is tracked using intra-frame and inter-frame recursions. Centroids are computed as weighted sums of the edge points belonging to the object boundary. The use of centroids introduces competitive learning mechanisms in the tracking algorithm leading to improved robustness with respect to occlusion and contour sliding. Experimental results with traffic sequences are provided.