The latest developments in 3D capturing, processing, and rendering provide means to unlock novel 3D application pathways. The main elements of an integrated platform, which target tele-immersion and future 3D applications, are described in this paper, addressing the tasks of real-time capturing, robust 3D human shape/appearance reconstruction, and skeleton-based motion tracking. More specifically, initially, the details of a multiple RGB-depth (RGB-D) capturing system are given, along with a novel sensors' calibration method. A robust, fast reconstruction method from multiple RGB-D streams is then proposed, based on an enhanced variation of the volumetric Fourier transform-based method, parallelized on the Graphics Processing Unit, and accompanied with an appropriate texture-mapping algorithm. On top of that, given the lack of relevant objective evaluation methods, a novel framework is proposed for the quantitative evaluation of real-time 3D reconstruction systems. Finally, a generic, multiple depth stream-based method for accurate real-time human skeleton tracking is proposed. Detailed experimental results with multi-Kinect2 data sets verify the validity of our arguments and the effectiveness of the proposed system and methodologies.

In this paper, we propose a novel, full-body, real-time 3D reconstruction framework that makes use of pre-scanned body parts (more precisely pre-scanned 3D heads) so as to provide a more detailed 3D reconstruction mainly in the semantically important head area. Our framework deals with 3 major challenges: (a) multiple depth sensors collaboration, (b) pre-scanned head positioning and (c) reconstruction and texturing. In all the above challenges, we propose novel solutions so as to cope with time and space complexity, synchronization and 3D mesh quality. Experimental evaluation provides evidence that superior 3D mesh quality can be achieved compared to simple (not enhanced) use of depth cameras' data.

Tele-immersion (TI) related technologies can change the way people interact and bridge the gap between the physical and digital worlds. However, while the technology itself advances, most developed platforms have complex setups and require large investments. In this work, a low-cost platform is introduced, integrating multiple TI-related advances. Focusing on ease of use and rapid deployment, a fast and fully automatic calibration method is proposed. The platform enables real-time 3D reconstruction of users and their placement into a pre-authored 3D environment. Moreover, interaction is achieved through the user's body posture, removing the need for additional equipment and enabling natural control while immersed. Developing a real-time TI platform requires the efficient integration of several multidisciplinary elements. An elegant, minimal solution to these challenges is proposed and validated in a prototype TI multiplayer game, SpaceWars.

This paper presents a novel approach, which relies on content-based guided image filtering and weighted semi-global optimization for fast and accurate disparity estimation. The approach uses a pixel-based cost term that combines gradient, Gabor-Feature, and color information. The pixel-based matching costs are filtered by applying guided image filtering, which relies on rectangular support windows of two different sizes. In this way, two filtered costs are estimated for each pixel. Among the two filtered costs, the one that will be finally assigned to each pixel depends on the local image content around this pixel. The filtered cost volume is further refined by exploiting weighted semi-global optimization, which improves the disparity estimation accuracy. Finally, the disparity refinement in outlier regions relies on a straightforward and time-efficient outliers handling scheme and on a simple approach which deals with the disparity outliers at depth discontinuities. Experimental results on the Middlebury online stereo evaluation benchmark and 27 additional Middlebury stereo pairs prove that our method is able to generate disparity maps with high accuracy while keeping the computational cost low.

In this work, we explore the potential of exploiting activity-related global features in order to improve the performance of an existing human Time-Varying Mesh (TVM) compression scheme. The TVM compression scheme used, employs two kinds of frames, namely Intra(I)-Fames and Enhanced Predicted(EP) Frames. In this scheme, I-Frames are used as a reference to encode EP-Frames. The paper introduces a strategy for selecting the most appropriate I-Frame that will serve as a reference frame for the encoding of EP-Frames, exploiting activity-related characteristics. Two different strategies are presented, using a skeleton-matching criterion and a periodicity measurement metric based on human skeleton. Evaluation is conducted on two sequences of the MPEG-3DGC database [1]. Results show that the concept is sound, but they also reveal the sensitivity of the proposed methods to the skeleton quality, thus the need for more robust skeleton tracking techniques.

In this work, the 3D flow estimation problem is formulated in the 4D spatiotemporal frequency domain, and it is shown that 3D motion manifests itself as energy concentration along hyper-planes in that domain. Based on this, the construction and use of appropriate directional multidimensional “steerable” filters, which can extract directional energy in spacetime, is proposed. Steerable filters have been constructed for up to 3 dimensions. We extend the relevant mathematical definitions to multiple dimensions and formulate filter-based algorithms for 3D flow estimation. Experimental results on simulated and real data verify the efficiency of the algorithms.

In this paper the scanline optimization used for stereo matching, is revisited. In order to improve the performance of this semi-global technique, a new criterion to check depth discontinuity, is introduced. This criterion is defined according to the mean-shift-based image segmentation result. Additionally, this work proposes the employment of a pixel dissimilarity metric for the computation of the cost term, which is then provided to the guided image filter approach to estimate the initial cost volume. The algorithm is tested on the four images of the online Middlebury stereo evaluation benchmark. Moreover, it is tested on 27 additional Middlebury stereo pairs for assessing thoroughly its performance. The extended comparison verifies the efficiency of this work.

The primary objective of this paper is to present and analyze key aspects related to next-generation tele-immersion applications, studying the end-to-end chain from 3D capturing of remote users to rendering. The key modules for 3D reconstruction of moving humans and their mesh compression, are presented and discussed. The chain performance is evaluated in terms of frame-rates, delay, and visual quality.

In this paper, the problem of real-time, full 3D reconstruction of foreground moving objects, an important task for Tele-Immersion applications, is addressed. More specifically, the proposed reconstruction method receives input from multiple consumer RGB-Depth cameras. A fast and efficient method to calibrate the sensors in initially described. More importantly, an efficient method to smoothly fuse the captured raw point sets is then presented, followed by a volumetric method to produce watertight and manifold meshes. Given the implementation details, the proposed method can operate at high frame rates. The experimental results, with respect to reconstruction quality and rates, verify the effectiveness of the proposed methodology.

In this paper, a novel skeleton-based approach to human time-varying mesh (H-TVM) compression is presented. The topic of TVM compression is new and has many challenges, such as handling the lack of obvious mapping of vertices across frames and handling the variable connectivity across frames, while maintaining efficiency, which are the most important ones. Very few works exist in the literature, while not all of the challenges have been addressed yet. In addition, developing an efficient and real-time solution, handling the above, obviously is a difficult task. We attempt to address the H-TVM compression problem inspired from video coding using different types of frames and trying to efficiently remove inter-frame geometric redundancy utilizing the recent advances in human skeleton tracking. The overall approach focuses on compression efficiency, low distortion, and low computation time enabling for real-time transmission of H-TVMs. It efficiently compresses geometry and vertex attributes of TVMs. In addition, this paper is the first to provide an efficient method for connectivity coding of TVMs, by introducing a modification to the state-of-the-art MPEG-4 TFAN algorithm. Experiments are conducted in the MPEG-3DGC TVM database. The method outperforms the state-of-the-art standardized static mesh coder MPEG-4 TFAN at low bit-rates, while remaining competent at high bit-rates. It gives a practical proof of concept that in the combined problem of geometry, connectivity, and vertex attribute coding of TVMs, efficient inter-frame redundancy removal is possible, establishing ground for further improvements. Finally, this paper proposes a method for motion-based coding of H-TVMs that can further enhance the overall experience when H-TVM compression is used in a tele-immersion scenario.

In this paper, the problem of automatic dance performance evaluation from human Motion Capture (MoCap) data is addressed. A novel framework is presented, using data captured by Kinect-based human skeleton tracking, where the evaluation of user's performance is achieved against a gold-standard performance of a teacher. The framework addresses several technical challenges, including global and local temporal synchronization, spatial alignment and comparison of two “dance motion signals.” Towards the solution of these technical challenges, a set of appropriate quaternionic vector-signal processing methodologies is proposed, where the 4D (spatiotemporal) human motion data are represented as sequences of pure quaternions. Such a quaternionic representation offers several advantages, including the facts that joint angles and rotations are inherently encoded in the phase of quaternions and the three coordinates variables ( X,Y,Z) are treated jointly, with their intra-correlations being taken into account. Based on the theory of quaternions, a number of advantageous algorithms are formulated. Initially, global temporal synchronization of dance MoCap data is achieved by the use of quaternionic cross-correlations, which are invariant to rigid spatial transformations between the users. Secondly, a quaternions-based algorithm is proposed for the fast spatial alignment of dance MoCap data. Thirdly, the MoCap data can be temporally synchronized in a local fashion, using Dynamic Time Warping techniques adapted to the specific problem. Finally, a set of quaternionic correlation-based measures (scores) are proposed for evaluating and ranking the performance of a dancer. These quaternions-based scores are invariant to rigid transformations, as proved and demonstrated. A total score metric, through a weighted combination of three different metrics is proposed, where the weights are optimized using Particle Swarm Optimization (PSO). The presented experimental results using the Huawei/3DLife/EMC2dataset are promising and verify the effectiveness of the proposed methods.

The problem of robust, realistic and especially fast 3-D reconstruction of objects, although extensively studied, is still a challenging research task. Most of the state-of-the-art approaches that target real-time applications, such as immersive reality, address mainly the problem of synthesizing intermediate views for given view-points, rather than generating a single complete 3-D surface. In this paper, we present a multiple-Kinect capturing system and a novel methodology for the creation of accurate, realistic, full 3-D reconstructions of moving foreground objects, e.g., humans, to be exploited in real-time applications. The proposed method generates multiple textured meshes from multiple RGB-Depth streams, applies a coarse-to-fine registration algorithm and finally merges the separate meshes into a single 3-D surface. Although the Kinect sensor has attracted the attention of many researchers and home enthusiasts and has already appeared in many applications over the Internet, none of the already presented works can produce full 3-D models of moving objects from multiple Kinect streams in real-time. We present the capturing setup, the methodology for its calibration and the details of the proposed algorithm for real-time fusion of multiple meshes. The presented experimental results verify the effectiveness of the approach with respect to the 3-D reconstruction quality, as well as the achieved frame rates.

The ever increasing availability of high speed Internet access has led to a leap in technologies that support real-time realistic interaction between humans in online virtual environments. In the context of this work, we wish to realise the vision of an online dance studio where a dance class is to be provided by an expert dance teacher and to be delivered to online students via the web. In this paper we study some of the technical issues that need to be addressed in this challenging scenario. In particular, we describe an automatic dance analysis tool that would be used to evaluate a student's performance and provide him/her with meaningful feedback to aid improvement.

This work addresses the Huawei/3DLife Grand Challenge, presenting a novel method for the analysis of dance movements. The approach focuses on the decomposition of the dance movements into elementary motions. Placing this problem into a probabilistic framework, we propose to exploit Gaussian processes to accurately model the different components of the decomposition. The preliminary results, presented in this paper, are very promising. In particular, two applications are considered, illustrating the relevance of the proposed approach, namely the correction of tracking errors and the smoothing of some movements of the teacher to help toward the dance learning.

In this work, we describe a novel system that automatically evaluates dance performances against a gold-standard performance and provides visual feedback to the performer in a 3D virtual environment. The system acquires the motion of a performer via Kinect-based human skeleton tracking, making the approach viable for a large range of users, including home enthusiasts. Unlike traditional gaming scenarios, when the motion of a user must by kept in synch with a pre-recorded avatar that is displayed on screen, the technique described in this paper targets online interactive scenarios where dance choreographies can be set, altered, practiced and refined by users. In this work, we have addressed some areas of this application scenario. In particular, a set of appropriate signal processing and soft computing methodologies is proposed for temporally aligning dance movements from two different users and quantitatively evaluating one performance against another.

We present a novel approach that handles the motion estimation/tracking problem in a phase-frequency tracking framework. Motion estimation in the spectral domain has been extensively studied. However, the real-time time-varying velocities tracking is still a challenging task. Our approach is based on a proposed phase-locked loop (PLL)-based system, which is hardware implementable and can be used for true real-time motion estimation of objects in image sequences. We describe all the necessary details of the PLL-based system which can track a single or multiple time-varying motions. We highlight the advantages and limitations of our methodology and we present promising simulation results on both synthetic and natural image sequences.

Although the motion estimation problem has been extensively studied, most of the proposed estimation approaches deal mainly with monochrome videos. The most usual way to apply them also in color image sequences is to process each color channel separately. A different, more sophisticated approach is to process the color channels in a "holistic" manner using quaternions, as proposed by Ell and Sangwine. In this paper, we extend standard spatiotemporal Fourier-based approaches to handle color image sequences, using the hypercomplex Fourier transform. We show that translational motions are manifested as energy concentration along planes in the hypercomplex 3-D Fourier domain and we describe a methodology to estimate the motions, based on this property. Furthermore, we compare the three-channels-separately approach with our approach and we show that the computational effort can be reduced by a factor of 1/3, using the hypercomplex Fourier transform. Also, we propose a simple, accompanying method to extract the moving objects in the hypercomplex Fourier domain. Our experimental results on synthetic and natural images verify our arguments throughout the paper.

We extend existing spatiotemporal approaches to handle time-varying motions estimation of multiple objects. It is shown that multiple, time-varying motions estimation is equivalent to the instantaneous frequency estimation of superpositioned FM sinusoids. Therefore, we apply established signal processing tools, such as time-frequency representations to show that for each time instant, the energy is concentrated along planes in the 3-D space: spatial frequencies-instantaneous frequency. Using fuzzy C-planes, we estimate indirectly the instantaneous velocities. Furthermore, adapting existing approaches to our problem, we attain the identification of the moving objects. The experimental results verify the effectiveness of our methodology.

Motion estimation in the spatiotemporal domain has been extensively studied and many methodologies have been proposed, which, however, cannot handle both time-varying and multiple motions. Extending previously published ideas, we present an efficient method for estimating multiple, linearly time-varying motions. It is shown that the estimation of accelerated motions is equivalent to the parameter estimation of superpositioned chirp signals. From this viewpoint, one can exploit established signal processing tools such as the chirp-Fourier transform. It is shown that accelerated motion results in energy concentration along planes in the 4-D space: spatial frequencies-temporal frequency-chirp rate. Using fuzzy c-planes clustering, we estimate the plane/motion parameters. The effectiveness of our method is verified on both synthetic as well as real sequences and its advantages are highlighted.