Semantic segmentation and depth estimation are two important tasks in the area of image processing. Traditionally, these two tasks are addressed in an independent manner. However, for those applications where geometric and semantic information is required, such as robotics or autonomous navigation, depth or semantic segmentation alone are not sufficient. In this paper, depth estimation and semantic segmentation are addressed together from a single input image through a hybrid convolutional network. Different from the state of the art methods where features are extracted by a sole feature extraction network for both tasks, the proposed HybridNet improves the features extraction by separating the relevant features for one task from those which are relevant for both. Experimental results demonstrate that HybridNet results are comparable with the state of the art methods, as well as the single task methods that HybridNet is based on.

Video segmentation is an important building block for high level applications such as scene understanding and interaction analysis. While outstanding results are achieved in this field by state-of-the-art learning and model based methods, they are restricted to certain types of scenes or require a large amount of annotated training data to achieve object segmentation in generic scenes. On the other hand, RGBD data, widely available with the introduction of consumer depth sensors, provides actual world 3D geometry compared to 2D images. The explicit geometry in RGBD data greatly helps in computer vision tasks, but the lack of annotations in this type of data may also hinder the extension of learning based methods to RGBD. In this paper, we present a novel generic segmentation approach for 3D point cloud video (stream data) thoroughly exploiting the explicit geometry in RGBD. Our proposal is only based on low level features, such as connectivity and compactness. We exploit temporal coherence by representing the rough estimation of objects in a single frame with a hierarchical structure, and propagating this hierarchy along time. The hierarchical structure provides an efficient way to establish temporal correspondences at different scales of object-connectivity, and to temporally manage the splits and merges of objects. This allows updating the segmentation according to the evidence observed in the history. The proposed method is evaluated on several challenging datasets, with promising results for the presented approach.

Human body analysis raises special interest because it enables a wide range of interactive applications. In this paper we present a gesture estimator that discriminates body poses in depth images. A novel collaborative method is proposed to learn 3D features of the human body and, later, to estimate specific gestures. The collaborative estimation framework is inspired by decision forests, where each selected point (anchor point) contributes to the estimation by casting votes. The main idea is to detect a body part by accumulating the inference of other trained body parts. The collaborative voting encodes the global context of human pose, while 3D features represent local appearance. Body parts contributing to the detection are interpreted as a voting process. Experimental results for different 3D features prove the validity of the proposed algorithm.

This paper provides an alternative solution to the costly representation of multi-view video data, which can be used for both rendering and scene analyses. Initially, a new efficient Monte Carlo discrete surface reconstruction method for foreground objects with static background is presented, which outperforms volumetric techniques and is suitable for GPU environments. Some extensions are also presented, which allow a speeding up of the reconstruction by exploiting multi-resolution and temporal correlations. Then, a fast meshing algorithm is applied, which allows interpolating a continuous surface from the discrete reconstructed points. As shown by the experimental results, the original video frames can be approximated with high accuracy by projecting the reconstructed foreground objects onto the original viewpoints. Furthermore, the reconstructed scene can be easily projected onto any desired virtual viewpoint, thus simplifying the design of free-viewpoint video applications. In our experimental results, we show that our techniques for reconstruction and meshing compare favorably with the state-of-the-art, and we also introduce a rule-of-thumb for effective application of the method with a good quality versus representation cost trade-off.

Temporal clustering of human motion into semantically meaningful behaviors is a challenging task. While unsupervised methods do well to some extent, the obtained clusters often lack a semantic interpretation. In this paper, we propose to learn what makes a sequence of human poses different from others such that it should be annotated as an action. To this end, we formulate the problem as weakly supervised temporal clustering for an unknown number of clusters. Weak supervision is attained by learning a metric from the implicit semantic distances derived from already annotated databases. Such a metric contains some low-level semantic information that can be used to effectively segment a human motion sequence into distinct actions or behaviors. The main advantage of our approach is that metrics can be successfully used across datasets, making our method a compelling alternative to unsupervised methods. Experiments on publicly available mocap datasets show the effectiveness of our approach.

End-effectors are considered to be the main topological extremities of a given 3D body. Even if the nature of such body is not restricted, this paper focuses on the human body case. Detection of human extremities is a key issue in the human motion capture domain, being needed to initialize and update the tracker. Therefore, the effectiveness of human motion capture systems usually depends on the reliability of the obtained end-effectors. The increasing accuracy, low cost and easy installation of depth cameras has opened the door to new strategies to overcome the body pose estimation problem. With the objective of detecting the head, hands and feet of a human body, we propose a new local feature computed from depth data, which gives an idea of its curvature and prominence. Such feature is weighted depending on recent detections, providing also a temporal dimension. Based on this feature, some end-effector candidate blobs are obtained and classified into head, hands and feet according to three probabilistic descriptors.

A novel real-time algorithm for head and hand tracking is proposed in this paper. This approach is based on data from a range camera, which is exploited to resolve ambiguities and overlaps. The position of the head is estimated with a depth-based template matching, its robustness being reinforced with an adaptive search zone. Hands are detected in a bounding box attached to the head estimate, so that the user may move freely in the scene. A simple method to decide whether the hands are open or closed is also included in the proposal. Experimental results show high robustness against partial occlusions and fast movements. Accurate hand trajectories may be extracted from the estimated hand positions, and may be used for interactive applications as well as for gesture classification purposes.

In this paper we present a novel foreground segmentation and 3D reconstruction system for multi-view scenarios. The system achieves correct 3D object reconstruction even when foreground segmentation presents critical misses in some of the views. We introduce the spatial redundancy of the multi-view data into the foreground segmentation process by combining segmentation and the 3D reconstruction in a two steps workflow. First, the segmentation of the objects in each view uses a monocular, region-based foreground segmentation in a MAP-MRF framework for foreground, background and shadow classes. Next, we compute an iterative volume reconstruction in a 3D tolerance loop, obtaining an iteratively enhanced SfS volume. Foreground segmentation is improved by updating the foreground model of each view at each iteration. The results presented in this paper show the improved foreground segmentation and the reduction of errors in the reconstruction of the volume.

We present a real-time human body tracking system for a single user in a Smart Room scenario. In this paper we propose a novel system that involves a silhouette-based cost function using variable windows, a hierarchical optimization method, parallel implementations of pixel-based algorithms and efficient usage of a low-cost hardware structure. Results in a Smart Room setup are presented.

A novel real-time algorithm for head and hand tracking is proposed in this paper. This approach is based on 2.5D data from a range camera, which is exploited to resolve ambiguities and overlaps. Experimental results show high robustness against partial occlusions and fast movements. The estimated positions are fairly stable, allowing the extraction of accurate trajectories which may be used for gesture classification purposes.

We address the problem of reconstructing 3D shapes from color data available in a sparse set of views from all directions of a scene. As an advantage when compared to multiview stereo approaches, our method is able to reconstruct object surfaces from a small number of views in wide-baseline setups. This introduces a trade-off between reconstruction accuracy and spatial coverage. The proposed algorithm obtains candidate surface points using a photo-consistency test and restricting the analysis to foreground pixels. The final surface points are extracted by iteratively carving away candidate points that are not photo-consistent with the complete multiview set. Finally, a surface patch is obtained from each colored surface point by estimating its orientation. Experimental results reflect the validity of the approach by comparing it to a voxelized implementation.

This paper presents a fast algorithm for recovering shape and motion of dynamic scenes from sequences of silhouettes in calibrated multi-camera settings in wide-baseline configurations. The proposed method captures the static shape on the first frame by deforming an arbitrary set of points located on an initial surface enclosing the actual scene. Then, the initial 3D shape is propagated in subsequent frames by quickly deforming the initial static shape in order to fit the available silhouettes. The resulting points lie on the surface of the Visual Hull. Shape and motion descriptions are jointly obtained by the dynamic procedure that adapts existing surfaces to the set of silhouettes at each new time instant. Experimental results show how the proposed dynamic method obtains silhouette-consistent shape estimates -with the same accuracy than a static approach- and dense motion estimates with very low usage of computational resources, by exploiting both spatial and temporal redundancies.

This paper considers the problem of reconstructing a surface from a point set. More specifically, we propose a method which focuses on obtaining fast surface reconstructions for visual purposes. The proposed scheme is based on propagation in a voxelized space, which is performed in the directions defined by a propagation pattern, during an optimal number of iterations. Real-time applications are conceivable thanks to a low execution time and computational cost, keeping an acceptable visual quality of the reconstruction.

This paper presents a low cost real-time alternative to available commercial human motion capture systems. First, a set of distinguishable markers are placed on several human body landmarks and the scene is captured by a number of calibrated and synchronized cameras. In order to establish a physical relation among markers, a human body model (HBM) is defined. Markers are detected on all camera views and delivered as the input of an annealed particle filter scheme where every particle encodes an instance of the pose of the HBM to be estimated. Likelihood between particles and input data is performed through the generalized symmetric epipolar distance and kinematic constrains are enforced in the propagation step towards avoiding impossible poses. Tests over the HumanEva annotated dataset yield quantitative results showing the effectiveness of the proposed algorithm. Results over sequences involving fast and complex motions are also presented.

Illumination changes may lead to false foreground (FG) segmentation and tracking results. Most of the existing FG extraction algorithms obtain a background (BG) estimation from temporal statistical parameters. Such algorithms consider a quasi-static BG which does not change but slowly. Therefore, fast illumination changes are not taken into account by the BG estimator and they are considered as FG. The aim of the proposed algorithm is to reduce illumination effects in video sequences in order to improve foreground segmentation performances.

The detection of the acoustic events (AEs) that are naturally \nproduced in a meeting room may help to describe the human \nand social activity that takes place in it. When applied to \nspontaneous recordings, the detection of AEs from only audio \ninformation shows a large amount of errors, which are mostly due to temporal overlapping of sounds. In this paper, a system to detect and recognize AEs using both audio and video information is presented. A feature-level fusion strategy is used, and the structure of the HMM-GMM based system considers each class separately and uses a one-against-all strategy for training. Experime ntal AED results with a new and rather spontaneous dataset are presented which show the advantage of the proposed approach.

This paper presents a new approach to the problem of simultaneous tracking of several people in low resolution sequences from multiple calibrated cameras. Redundancy among cameras is exploited to generate a discrete 3D colored representation of the scene. Two Monte Carlo based schemes adapted to the incoming 3D discrete data are introduced. First, a particle filtering technique is proposed relying on a volume likelihood function taking into account both occupancy and color information. Sparse sampling is presented as an alternative based on a sampling of the surface voxels in order to estimate the centroid of the tracked people. In this case, the likelihood function is based on local neighborhoods computations thus decreasing the computational load of the algorithm. A discrete 3D re-sampling procedure is introduced to drive these samples along time. Multiple targets are tracked by means of multiple filters and interaction among them is modeled through a 3D blocking scheme. Tests over annotated databases yield quantitative results showing the effectiveness of the proposed algorithms in indoor scenarios.

This paper presents a new approach to the problem of simultaneous tracking of several people in low resolution sequences from multiple calibrated cameras. Spatial redundancy is exploited to generate a discrete 3D binary representation of the scene. A particle filtering scheme adapted to the incoming 3D discrete data is proposed. A volume likelihood function and a discrete 3D re-sampling procedure are introduced to evaluate and drive particles. Multiple targets are tracked by means of multiple particle filters and interaction among them is modeled through a 3D blocking scheme. Test over annotated databases yield quantitative results showing the effectiveness of the proposed algorithm in the indoor scenarios.

This paper presents a multimodal approach to head pose estimation and 3D gaze orientation of individuals in a SmartRoom environment equipped with multiple cameras and microphones. We first introduce the two monomodal approaches as reference. In video, we estimate head orientation from color information by exploiting spatial redundancy among cameras. Audio information is processed to estimate the direction of the voice produced by a speaker making use of the directivity characteristics of the head radiation pattern. Two multimodal information fusion schemes working at data and decision levels are analyzed in terms of accuracy and robustness of the estimation. Experimental results conducted over the CLEAR evaluation database are reported and the comparison of the proposed multimodal head pose estimation algorithms with the reference monomodal approaches proves the effectiveness of the proposed approach.

The visual hull is defined as the intersection of the visual cones formed by the back-projection of C 2D silhouettes into the 3D space. The set of 2D silhouettes is consistent if there exists at least one volume which exactly explains them. Shape from silhouette (SfS) is the general term used to refer to the techniques employed to obtain a volume from silhouettes, which are considered to be consistent. In this paper we extend the idea of SfS to be used with sets of inconsistent silhouettes resulting from inaccurate calibration and erroneous 2D silhouette extraction techniques. The method presented detects and corrects errors in the silhouettes based on the consistency principle, implying an unbiased treatment of false alarms and misses in 2D.

This paper presents a new procedure to explore 3D for visual analysis in multicamera environments. We propose a 3D geometry to sample (voxelize) the space when checking the consistency of the analyzed features in the multiple views. Contrary to regular voxelization, the proposed geometry is irregular in 3D, but becomes regular once projected onto camera images. The aim is to better exploit the redundancy and the enriched information provided in multiview frameworks at the analysis stage. This is accomplished by balancing the usage of the available data (i.e. captured pixels) across the multiple cameras, instead of focusing in a regular sampling of the 3D space from which we do not have direct data. An efficient recursive algorithm that uses the proposed geometry is outlined, and the experimental results reflect higher accuracy than regular voxelization with equivalent restrictions for the chosen multiview analysis applications

Binary partition trees (BPTs) are a well known technique used for region-based image representation and analysis. BPTs are usually created as a result of a merging process based on homogeneity properties, such as colour or motion. In this paper, we present a BPT creation technique based on a general merging algorithm, where the homogeneity criteria are neither low-level (pixel oriented, intra-region), nor high-level features (object oriented, semantics), but a combination of several criteria including region-based and structural features such as shape and partial-inclusion. We are thus combining intra-region homogeneity (e.g colour-based) with inter-region homogeneity (structural), with the long term aim of bridging the gap in region-based image analysis from low-level features to a higher level interpretation of the image by the intermediate description of the image structure (which we call "syntactic visual features"). Syntactic visual features are geometric relationships among regions based on shapes and the spatial configuration of neighboring regions in the image and can be found by structure analysis (or syntactic analysis). The aim of this work is to present how the addition of combined pixel-region and structural features leads to better object BPT representation.

This paper presents a novel approach to the problem of determining head pose estimation and face 3D orientation of several people in low resolution sequences from multiple calibrated cameras. Spatial redundancy is exploited and the heads of people in the scene are detected and geometrically approximated by an ellipsoid using a voxel reconstruction and a moment analysis method. Skin patches from each detected head are located in each camera view. Data fusion is performed by back-projecting skin patches from single images onto the estimated 3D head model, thus providing a synthetic reconstruction of the head appearance. Finally, these data are processed in a pattern analysis framework thus giving a reliable and robust estimation of face orientation. Tracking over time is performed by Kalman filtering. Results are provided showing the effectiveness of the proposed algorithm in a Smart-Room scenario.

The aim of This work is to present how structural relationships among regions can be used in the context of a general merging algorithm based on texture. In Garrido and Salembier (1998) and Salembier and Garrido (2000) the notions of merging order, merging model and merging criterion were introduced for the creation of binary-partition-trees (BPTs). We extend this framework to syntactic features, representing geometric relationships between image regions, which are taken into account for the creation of the BPT. In particular, we perform structure analysis on the shapes and the spatial configuration of image regions. In order to test the syntactic features approach, a proof of concept has been carried out following a quasi-inclusion criterion.

This paper discusses the interest of Binary Partition Trees for image and region representation in the context of indexing and similarity based retrieval. Binary Partition Trees concentrate in a compact and structured way the set of regions that compose an image. Since the tree is able to represent images in a multiresolution way, only simple descriptors need to be attached to the nodes. Moreover, this representation is used for similarity based region retrieval.

A feature-based subband coding scheme is presented. It is intended for the coding of the texture component of a two-component image model. The primary component contains the strong edges, individually described by their amplitudes and positions. A morphological interpolation technique, presented by Casas, Salembier and Torres (see IEEE International Conference on Image Processing, vol.I, p.526-9, 1995), reconstructs the smooth areas of the image from the coded transitions. The residual texture component is coded by subband decomposition. Information about the edge structure, available from the primary component, is used for texture coding by employing adaptive quantizers and bit allocations over arbitrarily shaped regions. This feature-based subband coding technique produces subjective improvements over conventional subband schemes. The enhanced rendition of the strong edges is especially noticeable at very-low bit-rates. The coding of the primary component, based on morphological tools, is also more efficient and of lower complexity than in previously reported schemes for two-component image coding models.

The objective of this paper is to provide an efficient and yet simple method to encode the prediction error image of video sequences, based on a stochastic vector quantization (SVQ) approach that has been modified to cope with the intrinsic decorrelated nature of the prediction error image of video signals. In the SVQ scheme, the codewords are generated by stochastic techniques instead of being generated by a training set representative of the expected input image as is normal use in VQ. The performance of the scheme is shown for the particular case of segmentation-based video coding although the technique can be also applied to motion-compensated hybrid coding schemes.

This paper deals with the use of some morphological tools for image and video coding. Mathematical morphology can be considered as a shape-oriented approach to signal processing, and some of its features make it very useful for compression. Rather than describing a coding algorithm, the purpose of this paper is to describe some morphological tools that have proved attractive for compression. Four sets of morphological transformations are presented: connected operators, the region-growing version of the watershed, the geodesic skeleton, and a morphological interpolation technique. The authors discuss their implementation, and show how they can be used for image and video segmentation, contour coding, and texture coding.

In this paper a new morphological interpolation technique is presented. It is applied to the coding of the smooth (primary) component in a sketch-based image compression approach for very low bit-rates. The interpolation technique is intended to perform two dimensional interpolation from any set of initial pixels and, in particular, from sketch data. It makes intensive use of geodesic dilation, a morphological operator that may be implemented by means of FIFO queues. This results in a very efficient process compared to those that perform interpolation by linear filtering on the initial image. For the application of this method to interpolative image coding, the sketch data is extracted as a set of maximum curvature lines by means of the watershed algorithm. From such information, the interpolation technique obtains a fair reconstruction of both the smooth texture component and the main transitions of the image signal at low bit-rate cost.

In segmentation-based image coding techniques the image to be compressed is first segmented. Then, the information is coded describing the shape and the interior of the regions. A new method to encode the texture obtained in segmentation-based coding schemes is presented. The approach combines 2-D linear prediction and stochastic vector quantization. To encode a texture, a linear predictor is computed first. Next, a codebook following the prediction error model is generated and the prediction error is encoded with VQ. In the decoder, the error image is decoded first and then filtered as a whole, using the prediction filter. Hence, correlation between pixels is not lost from one block to another and a good reproduction quality can be achieved.

A new method for the extraction, selection and coding of perceptually significant image components in a video sequence is presented. It has been applied to the coding residues of a segmentation-based multilevel coding scheme in order to show the need for a careful perceptual selection in very low bit-rate video coding applications. Mathematical morphology operators are used for the extraction and selection steps of the proposed algorithm. The locations of the selected image components are then efficiently coded with a positional coder using prediction in the temporal dimension. The results show that for coding rates lower than 10 kbit/s, perceptual selection becomes a critical issue in order to achieve a good coding rendition. At higher rates, the fact of considering perceptual factors also leads to higher compression for similar subjective image quality.>

In second generation image compression techniques the image to be compressed is first segmented. The pixels are divided into mutually exclusive spatial regions based on some criteria. After segmentation, the image consists of regions separated by contours. Then, the information is coded describing the shapes and interiors of the regions. The interiors of the regions are usually encoded using polynomials. The objective of this paper is to encode the interior of the regions by stochastic vector quantization techniques. If the segmentation process has been well defined and the obtained regions are homogeneous, then it is possible to design a specific codebook suited to the statistics of each region. The approach is to design the codebook according to some previously defined model for the regions of the image found in the segmentation process. If the approach is combined with efficient contour coding techniques, good visual results for high compression rates are obtained.>

We present a morphological filter for lossless image subsampling for a given downsampling-upsampling strategy. This filter is applied in a multiresolution decomposition and results in a more efficient scheme for image coding purposes than other lossy sampling schemes. Its main advantage is a greatly reduced computational load compared to multiresolution schemes performed with linear filters.>

A new method for texture coding which combines 2-D linear prediction and stochastic vector quantization is presented in this paper. To encode a texture, a linear predictor is computed first. Next, a codebook following the prediction error model is generated and the prediction error is encoded with VQ, using an algorithm which takes into account the pixels surrounding the block being encoded. In the decoder, the error image is decoded first and then filtered as a whole, using the prediction filter. Hence, correlation between pixels is not lost from one block to another and a good reproduction quality can be achieved.>

In this paper, the importance of including small image features at the initial levels of a progressive second generation video coding scheme is presented. It is shown that a number of meaningful small features called details should be coded, even at very low data bit-rates, in order to match their perceptual significance to the human visual system. We propose a method for extracting, perceptually selecting and coding of visual details in a video sequence using morphological techniques. Its application in the framework of a multiresolution segmentation-based coding algorithm yields better results than pure segmentation techniques at higher compression ratios, if the selection step fits some main subjective requirements. Details are extracted and coded separately from the region structure and included in the reconstructed images in a later stage. The bet of considering the local background of a given detail for its perceptual selection breaks the concept of "partition" in the segmentation scheme. As long as details are not considered as adjacent regions but isolated features spread over the image, "detail coding" can be seen as one step towards the so called feature-based video coding techniques.>

In progressive image coding, the cost in bits of the successive residual images being encoded does not always correspond with the subjective importance that such components may have. The original idea of this paper arose for increasing the efficiency when coding the last Laplacian levels in linear decompositions for pyramidal coding. The same principle has been applied to the non-linear image decomposition presented in which a segmentation-based progressive scheme is used for coding purposes. The `post-it' method for extracting details based on mathematical morphology proposed by Meyer has been modified in order to improve the rendition of the posterior coding, and a suitable technique for coding the extracted details from an extension of run-length coding is then applied. In both linear and non-linear cases the results of this method for `detail-coding' are compared against the conventional progressive coding technique, i.e., pyramidal coding or pure segmentation-based contour/texture coding.