Semantic segmentation and depth estimation are two important tasks in the area of image processing. Traditionally, these two tasks are addressed in an independent manner. However, for those applications where geometric and semantic information is required, such as robotics or autonomous navigation, depth or semantic segmentation alone are not sufficient. In this paper, depth estimation and semantic segmentation are addressed together from a single input image through a hybrid convolutional network. Different from the state of the art methods where features are extracted by a sole feature extraction network for both tasks, the proposed HybridNet improves the features extraction by separating the relevant features for one task from those which are relevant for both. Experimental results demonstrate that HybridNet results are comparable with the state of the art methods, as well as the single task methods that HybridNet is based on.

Video segmentation is an important building block for high level applications such as scene understanding and interaction analysis. While outstanding results are achieved in this field by state-of-the-art learning and model based methods, they are restricted to certain types of scenes or require a large amount of annotated training data to achieve object segmentation in generic scenes. On the other hand, RGBD data, widely available with the introduction of consumer depth sensors, provides actual world 3D geometry compared to 2D images. The explicit geometry in RGBD data greatly helps in computer vision tasks, but the lack of annotations in this type of data may also hinder the extension of learning based methods to RGBD. In this paper, we present a novel generic segmentation approach for 3D point cloud video (stream data) thoroughly exploiting the explicit geometry in RGBD. Our proposal is only based on low level features, such as connectivity and compactness. We exploit temporal coherence by representing the rough estimation of objects in a single frame with a hierarchical structure, and propagating this hierarchy along time. The hierarchical structure provides an efficient way to establish temporal correspondences at different scales of object-connectivity, and to temporally manage the splits and merges of objects. This allows updating the segmentation according to the evidence observed in the history. The proposed method is evaluated on several challenging datasets, with promising results for the presented approach.

This paper deals with Shape from Silhouette (SfS) volumetric reconstruction in the context of multi-view smart room scenarios. The method that we propose first computes a 2D foreground object segmentation in each one of the views, by using region-based models to model the foreground, and shadow classes, and a pixel-wise model to model the background class. Next, we calculate the reliability maps between foreground and background/shadow classes in each view, by computing the hellinger distance among models. These 2D reliability maps are taken into account finally, in the 3D SfS reconstruction algorithm, to obtain an enhanced final volumetric reconstruction. The advantages of our system rely on the possibility to obtain a volumetric representation which automatically defines the optimal tolerance to errors for each one of the voxels of the volume, with a low rate of false positive and false negative errors. The results obtained by using our proposal improve the traditional SfS reconstruction computed with a fixed tolerance for the overall volume.

We propose a complete multi-view foreground segmentation and 3D reconstruction system that defines a 3-dimensional probabilistic model to model the foreground object in the 3 spatial dimensions, thus gathering the information from all the camera views. This 3D model is projected to each one of the views in order to perform the 2D segmentation with the foreground information shared by all the cameras. Then, for each one of the views, a MAP-MRF classification framework is applied between the projected region-based foreground model, the pixel-wise background model and the region-based shadow model defined for each view. The resultant masks are used to compute the next 3-dimensional reconstruction. This system achieves correct results by reducing the false positive and false negative errors in sequences where some camera sensors can present camouflage situations between foreground and background. Moreover, the use of the 3D model opens possibilities to use it for objects recognition or human activity understanding.

In this paper we present a specific kind of projects that can be used for project-based learning in engineering subjects. The subjects must combine lectures with projects, in order to provide the technical competences together with additional skills such as teamwork learning, oral and written communication skills and application of theory to practice. The projects proposed consist on improving an elemental baseline system. The system is decomposed in modules that correspond to the topics that have been learnt during the lectures. For improving the system, the class is divided in groups and each group has to propose, implement, assess and report a better system. In order to be able to improve the system with a limited amount of time and effort the students need to make a coherent proposal and split the project in several tasks that are usually developed by one or two students. The students within a group cooperate to achieve a better system, but groups compete for the best results. We have already implemented this kind of project in a Speech Processing course and we plan to apply it in a Video Coding course.

In this paper we present a novel foreground segmentation and 3D reconstruction system for multi-view scenarios. The system achieves correct 3D object reconstruction even when foreground segmentation presents critical misses in some of the views. We introduce the spatial redundancy of the multi-view data into the foreground segmentation process by combining segmentation and the 3D reconstruction in a two steps workflow. First, the segmentation of the objects in each view uses a monocular, region-based foreground segmentation in a MAP-MRF framework for foreground, background and shadow classes. Next, we compute an iterative volume reconstruction in a 3D tolerance loop, obtaining an iteratively enhanced SfS volume. Foreground segmentation is improved by updating the foreground model of each view at each iteration. The results presented in this paper show the improved foreground segmentation and the reduction of errors in the reconstruction of the volume.

We present a real-time human body tracking system for a single user in a Smart Room scenario. In this paper we propose a novel system that involves a silhouette-based cost function using variable windows, a hierarchical optimization method, parallel implementations of pixel-based algorithms and efficient usage of a low-cost hardware structure. Results in a Smart Room setup are presented.

In this paper we present a novel foreground segmentation system for monocular static camera sequences and indoor scenarios that achieves a correct shadow removal via global MAP-MRF framework formulation for the foreground, background and shadow classification task. We propose to combine a region-based spatial-color foreground model and a pixel-wise background model in the RGB domain with an spatial-Brightness Distortion (BD) and Color Distortion (CD) shadow model which present specific features to classify potential shadow regions. The results presented in the paper show the improvement of the system avoiding the necessity of thresholds for shadow detection task and reducing false positive and false negative detections originated by the shadow effects that other methods of the state of the art present.

This paper presents a low cost real-time alternative to available commercial human motion capture systems. First, a set of distinguishable markers are placed on several human body landmarks and the scene is captured by a number of calibrated and synchronized cameras. In order to establish a physical relation among markers, a human body model (HBM) is defined. Markers are detected on all camera views and delivered as the input of an annealed particle filter scheme where every particle encodes an instance of the pose of the HBM to be estimated. Likelihood between particles and input data is performed through the generalized symmetric epipolar distance and kinematic constrains are enforced in the propagation step towards avoiding impossible poses. Tests over the HumanEva annotated dataset yield quantitative results showing the effectiveness of the proposed algorithm. Results over sequences involving fast and complex motions are also presented.

In this paper we present a segmentation system for monocular video sequences with static camera that aims at foreground/background separation and tracking. We propose to combine a simple pixel-wise model for the background with a general purpose region based model for the foreground. The background is modeled using one Gaussian per pixel, thus achieving a precise and easy to update model. The foreground is modeled using a Gaussian mixture model with feature vectors consisting of the spatial (x, y) and colour (r, g, b) components. The spatial components of this model are updated using the expectation maximization algorithm after the classification of each frame. The background model is formulated in the 5 dimensional feature space in order to be able to apply a maximum a posteriori framework for the classification. The classification is done using a graph cut algorithm that allows taking into account neighborhood information. The results presented in the paper show the improvement of the system in situations where the foreground objects have similar colors to those of the background.

This paper presents the trajectory tree as a hierarchical region-based representation for video sequences. Motion, as well as spatial features from multiple frames are used to generate a set of temporal regions structured within a hierarchy of scale and motion coherency. The resulting representations offer a global description of the entire video sequence and enhance semantic analysis potential. A multiscale segmentation strategy is proposed whereby region-merging criteria of progressively greater complexity are used to define partition layers of increasing aptitude for object detection. A novel data structure, called the trajectory adjacency graph, is defined for the long-term analysis of partition sequences. Furthermore, mechanisms for assessing connectivity, verifying temporal continuity, and proposing merging operations based on color, affine, and translational motion homogeneity characteristics over the entire sequence are also introduced. Finally, as demonstrated through experimental results, the trajectory tree offers a concise yet versatile support for video object segmentation, description and retrieval tasks.

This paper presents a new approach to the problem of simultaneous tracking of several people in low resolution sequences from multiple calibrated cameras. Redundancy among cameras is exploited to generate a discrete 3D colored representation of the scene. Two Monte Carlo based schemes adapted to the incoming 3D discrete data are introduced. First, a particle filtering technique is proposed relying on a volume likelihood function taking into account both occupancy and color information. Sparse sampling is presented as an alternative based on a sampling of the surface voxels in order to estimate the centroid of the tracked people. In this case, the likelihood function is based on local neighborhoods computations thus decreasing the computational load of the algorithm. A discrete 3D re-sampling procedure is introduced to drive these samples along time. Multiple targets are tracked by means of multiple filters and interaction among them is modeled through a 3D blocking scheme. Tests over annotated databases yield quantitative results showing the effectiveness of the proposed algorithms in indoor scenarios.

In this paper we present a real-time object tracking system for monocular video sequences with static camera. The workflow is based on a pixel-based foreground detection system followed by foreground object tracking. The foreground detection method performs the segmentation in three levels: Moving Foreground, Static Foreground and Background level. The tracking uses the foreground segmentation for identifying the tracked objects, but minimizes the reliance on the foreground segmentation, using a modified Mean Shift tracking algorithm. Combining this tracking system with the Multi-Level foreground segmentation, we have improved the tracking results using the classification in static or moving objects. The system solves successfully a high percentage of the moving objects occlusions, and most of the occlusions between static and moving objects.

In this paper we present an efficient image-based rendering algorithm that obtains novel images from a set of views of the scene of interest. The approach described uses silhouette image data to compute the Visual Hull, the largest volume that is compatible with the silhouettes that delimit the objects of interest. Since the Visual Hull is not explicitly computed, this approach does not suffer from the quantization artifacts of volumetric approaches. In contrast to previous works, we explore how detection errors in the silhouettes affect the novel rendered view and propose a method to detect errors in the original silhouettes based on the consistency principle that states that the projection of the Visual Hull should exactly correspond with original silhouettes.

This paper addresses 2-D and 3-D active entity detection in video scenes. Active entities are the foreground parts in a stationary background scene and they typically correspond to the regions of interest in many applications such as video surveillance, object and person tracking, and suspicious object detection, among others. We present a novel framework that permits obtaining 2-D and 3-D active entities as an inter-dependent probabilistic procedure. In the process of creating this framework, a study has been conducted to explore ways to generalize existing activity detection techniques to a Bayesian form. A new Bayesian 3-D activity detection technique has been developed. The Bayesian framework gives a unified manner to interact between the planar and the volumetric detection tasks and helps to prevent the propagation of noisy pixel observations to the 3-D space. However, when large systematic errors occur in the 2-D detection level, a different approach has to be taken to correct them. We use a new 3-D foreground detection scheme that is able to correct errors in 2-D planar detections by checking the consistency between 3-D foreground detections and the set of corresponding 2-D foreground regions.

This paper describes a hierarchical analysis framework for image sequences. Region merging schemes traditionally used in the construction of partition hierarchies are extended to multiple frames using trajectory merging criteria. The merging criteria assess homogeneity among features throughout the entire sequence to recursively create partitions in the spatio-temporal domain. We propose similarity measures using long-term affine and translational motion features. Furthermore, the analysis of connectivity relations and the algorithm implementation over trajectory adjacency graphs allow the generation of partition sets containing temporally consistent objects characterized by coherent motion. Lastly, we introduce the novel trajectory tree as a single, hierarchical representation of the partitions generated for the complete sequence. Experimental results are provided, illustrating the usefulness of the approach.

This paper describes a technique for generating partition sequences of regions presenting long-term homogeneity in color and motion coherency in terms of affine models. The technique is based on region merging schemes compatible with hierarchical representation frameworks and can be divided into two stages: partition tracking and partition sequence analysis. Partition tracking is a recursive algorithm whereby regions are constructed according to short-term spatio-temporal features, namely color and motion. Partition sequence analysis proposes the trajectory adjacency graph (TAG) to exploit the long-term connectivity relations of tracked regions. A novel trajectory merging strategy using color homogeneity criteria over multiple frames is introduced. Algorithm performance is assessed and comparisons to other proposals are drawn by means of established evaluation metrics.

The visual hull is defined as the intersection of the visual cones formed by the back-projection of C 2D silhouettes into the 3D space. The set of 2D silhouettes is consistent if there exists at least one volume which exactly explains them. Shape from silhouette (SfS) is the general term used to refer to the techniques employed to obtain a volume from silhouettes, which are considered to be consistent. In this paper we extend the idea of SfS to be used with sets of inconsistent silhouettes resulting from inaccurate calibration and erroneous 2D silhouette extraction techniques. The method presented detects and corrects errors in the silhouettes based on the consistency principle, implying an unbiased treatment of false alarms and misses in 2D.

Object segmentation and tracking are two key issues in the analysis of scenes for video surveillance or scene understanding applications. The paper presents a new algorithm in the context of object segmentation. The proposed method combines an adaptive background learning technique with a hierarchical segmentation method based on binary partition trees. The result is a region-based dynamic scene description, where each active region is characterized by a temporal feature, reflecting on the time it remains in the same position of the scene. This description is then used to classify the background and foreground objects of the scene and can also be used as an additional feature for region tracking and scene understanding.

Dealing with shadows and highlights is essential in object detection and tracking applications such as automated video surveillance systems. This is especially true for outdoor scenarios subject to variable lighting and weather conditions. In this paper, we present a novel scheme for effective shadows (highlights) detection using both color and texture cues. Since in any such algorithm, misclassifications often occur, resulting in distorted object shapes, the core of this scheme is the introduction of a technique capable of correcting these errors. The technique is based on morphological reconstruction of the shadow-removed blobs conditioned on the blobs prior to a shadow-removal process, assuming that the object shapes are properly defined along most part of their contours after the initial detection. Experiments on a variety of real-world video data demonstrate the favorable performance and robustness of the proposed scheme.

This paper presents a novel approach to the problem of determining head pose estimation and face 3D orientation of several people in low resolution sequences from multiple calibrated cameras. Spatial redundancy is exploited and the heads of people in the scene are detected and geometrically approximated by an ellipsoid using a voxel reconstruction and a moment analysis method. Skin patches from each detected head are located in each camera view. Data fusion is performed by back-projecting skin patches from single images onto the estimated 3D head model, thus providing a synthetic reconstruction of the head appearance. Finally, these data are processed in a pattern analysis framework thus giving a reliable and robust estimation of face orientation. Tracking over time is performed by Kalman filtering. Results are provided showing the effectiveness of the proposed algorithm in a Smart-Room scenario.

This paper describes an approach for generating binary partition tree [P. Salembier and L. Garrido, 2000] representations and video object segmentations using a novel region merging strategy based on motion similarity measures of multiple frames of an image sequence. The system operates over color-homogeneous regions, tracked across frames of a shot, representing an over-segmentation of the objects. A long-term motion similarity measure is introduced for region merging, offering accurate segmentation of objects and extending temporal consistency between the tracked partitions to hierarchical representations of every frame within the shot. Experimental results are presented, illustrating the usefulness of the approach.

This paper presents an object recognition method that exploits the representation of the images obtained by means of a binary partition tree (BPT). The shape matching technique in which it is based was first presented in F. Marques et al., (2002). This method compares a transformed version of an object shape model (reference contour) to the contours of a partition of the image. The comparison is based on a distance map that measures the Euclidean distance between any points in the image to the partition contours. In F. Marques et al., (2002), this algorithm was applied using a colour-based segmentation of the image and a full-search was performed to find the best match between the searched object and the contours of this segmentation. Here, the information of the binary partition tree is used both to obtain the segmentation and to guide and reduce the search for the optimum match between the shape and the objects of the image.

The paper discusses the application of a facial expression recognition system in unrestrained video intervals. The system is based on the modeling of expressions by means of hidden Markov models. The observations used to create the models are the MPEG-4 standardized facial animation parameters (FAPs). The FAPs of a video sequence are first extracted and then analyzed using a semi-continuous HMM. The basic recognizer (Pardas, M. et al., Proc. 2002 IEEE Conf. Acoustics, Speech and Signal Processing, vol.4, p.3624-7, 2002) shows good performance in distinguishing entire expressions, previously marked, in video sequences. We now describe the adaptation of the technique to deal with unrestrained expression intervals in video sequences, that is, expressions whose boundaries have not been previously marked in the scene. We have taken advantage of the symmetry of the expressions to extract a new HMM topology and we have trained the models without requiring to record and analyse a new database. We have also used a discarding process with the aim of eliminating expressions not included in the models, together with intervals without any expressions at all. The combination of the presented techniques is suitable for temporal block sampling with later expression classification or discarding.

In this paper a facial expression recognition system is presented. The system is based on the modelling of the expressions by means of Hidden Markov Models. The observations used to create the models are the MPEG-4 standardized Facial Animation Parameters (FAPs). The FAPs of a video sequence are first extracted and then analyzed using semi-continuous HMM. The system shows good performance for distinguishing isolated expressions and can also be used, with lower accuracy, to extract the expressions in long video sequences where speech is mixed with silence frames.

This paper presents a new technique for object matching that exploits the information about transitions in the image obtained by means of a segmentation approach. Object matching is performed by comparing a transformed version of an object shape model (reference contour) to the contours in the image partition. The comparison is based on a distance map that measures the Euclidean distance between any point in the image to the partition contours. Examples using parametric and non-parametric reference contours are provided to assess the quality of the proposed technique.

This paper addresses the application of active contours or snakes for location and tracking of facial features. Conventional snake approaches find the position of the snake by finding a minimum of its energy, composed of internal and external forces. The external forces pull the contours toward features such as lines and edges. However, in many applications this minimization leads to contours that do not represent correctly the feature we are looking for. We propose to introduce some higher level information by a statistical characterization of the snaxels that should represent the contour. This higher level information is introduced in the selection of candidates in a dynamic programming implementation of the active contours algorithm, as well as in the external energy. Furthermore, the same approach is used for tracking the contours using motion estimation.

Facial analysis is used in a number of applications like face recognition systems, human-computer interaction through head movements or facial expressions, or model based coding. In all these applications a very precise extraction of the feature points corresponding to the eyes is necessary. In this paper we present a method for automatic extraction and tracking of the eyelids. First the images are filtered in order to enhance the interesting features. Then a dynamic programming algorithm is applied to extract the precise shape of the eyelids. Although restrictions are applied on the possible shapes, they are not modeled as ellipse arcs. This allows extracting the corner of the eyes with higher precision. Once the eyelids are extracted for the first image, a new robust tracking algorithm based on snakes is applied to track them along the sequence, which allows large motion of the eyes and rotation and zooming of the head.

This paper addresses the application of active contours or snakes for robust tracking of contours. Conventional snake approaches to tracking initialize the current frame snake with the snake obtained in the previous frame and then optimize this result considering only the current frame information. In our approach motion estimation is embedded in the energy minimization process of the snake. This is possible using a dynamic programming approach for this minimization and introducing a new term into the energy of the snake. In this way, larger displacements of the contour are allowed, and the tracking is more robust. This technique is applied for tracking facial features such as the contour of the face, the mouth, the nostrils and the eyebrows.

We present a method to estimate the relative depth between objects in scenes of video sequences. The information for the estimation of the relative depth is obtained from the overlapping produced between objects when there is relative motion as well as from motion coherence between neighbouring regions. A relaxation labelling algorithm is used to solve conflicts and assign every region to a depth level. The depth estimation is used in a segmentation scheme which uses grey level information to produce a first segmentation. Regions of this partition are merged on the basis of their depth level.

This paper presents a generic video coding algorithm allowing the content-based manipulation of objects. This manipulation is possible thanks to the definition of a spatiotemporal segmentation of the sequences. The coding strategy relies on a joint optimization in the rate-distortion sense of the partition definition and of the coding techniques to be used within each region. This optimization creates the link between the analysis and synthesis parts of the coder. The analysis defines the time evolution of the partition, as well as the elimination or the appearance of regions that are homogeneous either spatially or in motion. The coding of the texture as well as of the partition relies on region-based motion compensation techniques. The algorithm offers a good compromise between the ability to track and manipulate objects and the coding efficiency.

In this paper we describe a very low bit rate video coding system that belongs to the class of object based coding systems. The system takes advantage of homogeneity in both gray level and motion, trying to use the most adequate criterion for defining a partition in order to obtain the best compromise between cost and quality. In order to do so, multiple partitions are generated for every image in a hierarchical way, from the coarsest one to the finest one. Then, a decision step will chose which parts of the image need to be coded with regions from one partition or from another. The aim of the paper is to study the features that this multiple partition must have, and propose a way to create it.

We present a coding scheme that achieves, for each image in the sequence, the best segmentation in terms of rate-distortion theory. It is obtained from a set of initial regions and a set of available coding techniques. The segmentation combines spatial and motion criteria. It selects at each area of the image the most adequate criterion for defining a partition in order to obtain the best compromise between cost and quality. In addition, the proposed scheme is very suitable for addressing content-based functionalities.

This paper presents a video coding scheme for very low bit rate applications. The coding approach relies on active meshes and can be viewed as a particular case of region-based coding. The active mesh is used to efficiently represent and code the various regions of the scene and the motion information. The variation of the mesh topology as well as the strategy for coding the synthesis error are defined by an optimization technique following the rate-distortion criterion.

This paper deals with the use of some morphological tools for image and video coding. Mathematical morphology can be considered as a shape-oriented approach to signal processing, and some of its features make it very useful for compression. Rather than describing a coding algorithm, the purpose of this paper is to describe some morphological tools that have proved attractive for compression. Four sets of morphological transformations are presented: connected operators, the region-growing version of the watershed, the geodesic skeleton, and a morphological interpolation technique. The authors discuss their implementation, and show how they can be used for image and video segmentation, contour coding, and texture coding.

In object-based video coding systems, the scenes are described in terms of three dimensional objects, which can be coded as textures and contours. However, in order to achieve high compression ratios, the redundancy in the temporal dimension must be exploited using motion compensation. In this context, motion compensation can take advantage of the description of the scene as three dimensional regions. In this paper we present a motion estimation method which finds, in a fast and computationally simple manner, the affine parameters for every region of the scene. These parameters correspond to the motion of the texture. An ordering between regions is established in order to have the right compensation of contours.>

This paper describes a segmentation-based coding algorithm for very low bit rate. The algorithm involves a segmentation relying on a time recursive morphological segmentation technique. It extracts both large and contrasted regions. Then, a simple region-based motion estimation is performed. Finally, the sequence is coded by contour-texture motion compensation. The proposed method leads to very low bit rates while being able to represent the meaningful regions of the scene. Moreover, it makes no assumption about the scene content and can be used for any sequence.

This paper deals with a hierarchical morphological segmentation algorithm for image sequence coding. Mathematical morphology is very attractive for this purpose because it efficiently deals with geometrical features such as size, shape, contrast, or connectivity that can be considered as segmentation-oriented features. The algorithm follows a top-down procedure. It first takes into account the global information and produces a coarse segmentation, that is, with a small number of regions. Then, the segmentation quality is improved by introducing regions corresponding to more local information. The algorithm, considering sequences as being functions on a 3-D space, directly segments 3-D regions. A 3-D approach is used to get a segmentation that is stable in time and to directly solve the region correspondence problem. Each segmentation stage relies on four basic steps: simplification, marker extraction, decision, and quality estimation. The simplification removes information from the sequence to make it easier to segment. Morphological filters based on partial reconstruction are proven to be very efficient for this purpose, especially in the case of sequences. The marker extraction identifies the presence of homogeneous 3-D regions. It is based on constrained flat region labeling and morphological contrast extraction. The goal of the decision is to precisely locate the contours of regions detected by the marker extraction. This decision is performed by a modified watershed algorithm. Finally, the quality estimation concentrates on the coding residue, all the information about the 3-D regions that have not been properly segmented and therefore coded. The procedure allows the introduction of the texture and contour coding schemes within the segmentation algorithm. The coding residue is transmitted to the next segmentation stage to improve the segmentation and coding quality. Finally, segmentation and coding examples are presented to show the validity and interest of the coding approach.