In this paper, we tackle the challenge of three-dimensional estimation of expressive, animatable, and textured human avatars from a single frontal image. Leveraging a Skinned Multi-Person Linear (SMPL) parametric body model, we adjust the model parameters to faithfully reflect the shape and pose of the individual, relying on the mesh generated by a Pixel-aligned Implicit Function (PIFu) model. To robustly infer the SMPL parameters, we deploy a multi-step optimization process. Initially, we recover the position of 2D joints using an existing pose estimation tool. Subsequently, we utilize the 3D PIFu mesh together with the 2D pose to estimate the 3D position of joints. In the subsequent step, we adapt the body’s parametric model to the 3D joints through rigid alignment, optimizing for global translation and rotation. This step provides a robust initialization for further refinement of shape and pose parameters. The next step involves optimizing the pose and the first component of the SMPL shape parameters while imposing constraints to enhance model robustness. We then refine the SMPL model pose and shape parameters by adding two new registration loss terms to the optimization cost function: a point-to-surface distance and a Chamfer distance. Finally, we introduce a refinement process utilizing a deformation vector field applied to the SMPL mesh, enabling more faithful modeling of tight to loose clothing geometry. As most other works, we optimize based on images of people wearing shoes, resulting in artifacts in the toes region of SMPL. We thus introduce a new shoe-like mesh topology which greatly improves the quality of the reconstructed feet. A notable advantage of our approach is the ability to generate detailed avatars with fewer vertices compared to previous research, enhancing computational efficiency while maintaining high fidelity. We also demonstrate how to gain even more details, while maintaining the advantages of SMPL. To complete our model, we design a texture extraction and completion approach. Our entirely automated approach was evaluated against recognized benchmarks, X-Avatar and PeopleSnapshot, showcasing competitive performance against state-of-the-art methods. This approach contributes to advancing 3D modeling techniques, particularly in the realms of interactive applications, animation, and video games. We will make our code and our improved SMPL mesh topology available to the community: https://github.com/ETS-BodyModeling/ImplicitParametricAvatar .

Virtual reality applications possess significant societal potential, capable of revolutionizing user experiences and generating substantial revenue. However, their high demand for bit rates poses significant challenges. The MPEG Immersive Video (MIV) standard, an integral component of MPEG-I, is designed to efficiently compress visual content from multiple cameras by pruning redundant information. This article proposes a new method to enhance the compression efficiency of MIV by grouping and preserving small clusters of non-pruned pixels that would otherwise be discarded in the default configuration of the Test Model for Immersive Video (TMIV). Experimental results demonstrate that the proposed method attains an average Bjøntegaard-Delta bitrate (BD-BR) reduction of $3.35 \%$ across six tested sequences when compared to TMIV with the default configuration. Notably, one of them exhibits a reduction reaching $5.12 \%$.

In this paper, we tackle the challenge of three-dimensional estimation of expressive, animatable, and textured human avatars from a single frontal image. Leveraging a Skinned Multi-Person Linear (SMPL) parametric body, we adjust the model parameters to faithfully reflect the shape and pose of the individual, relying on the mesh generated by a Pixel-aligned Implicit Function (PIFu) model. To robustly infer the SMPL parameters, we deploy a multi-step optimization process. Initially, we recover the position of 2D joints using an existing pose estimation tool. Subsequently, we utilize the 3D PIFu mesh together with the 2D pose to estimate the 3D position of joints. In the subsequent step, we adapt the body’s parametric model to the 3D joints through rigid alignment, optimizing for global translation and rotation. This step provides a robust initialization for further refinement of shape and pose parameters. The next step involves optimizing the pose and the first component of the SMPL shape parameters while imposing constraints to enhance model robustness. We then refine the SMPL model pose and shape parameters by adding two new registration loss terms to the optimization cost function: a point-to-surface distance and a Chamfer distance. Finally, we introduce a refinement process utilizing a deformation vector field applied to the SMPL mesh, enabling more faithful modeling of tight to loose clothing geometry. A notable advantage of our approach is the ability to generate detailed avatars with fewer vertices compared to previous research, enhancing computational efficiency while maintaining high fidelity. To complete our model, we design a texture extraction and completion approach. Our entirely automated approach was evaluated against recognized benchmarks, X-Avatar and PeopleSnapshot, showcasing competitive performance against state-of-the-art methods. This approach contributes to advancing 3D modeling techniques, particularly in the realms of interactive applications, animation, and video games. We will make the code accompanying our paper publicly available upon its acceptance.

Utilizing frame-compatible (FC) formats is a common strategy for leveraging the existing single-view video transmission infrastructure to stream stereoscopic videos. However, using this method often comes with challenges, as stereoscopic video requires higher transmission bandwidth and larger memory buffers on the decoder compared to single-view videos. When it comes to stereoscopic 360∘videos, these requirements become more challenging since they ask for ultra-high-resolution formats with high frame rates (e.g., 6K, 8K, or 12K at 100 frames per second) to provide an acceptable quality of experience (QoE) to the users (4K on a 120∘viewport). To address these challenges, sub-sampled versions of the left and right views are usually used to form the spatial FC format, leading to a loss of visual quality. In this paper, we first analyze the amount of distortion due to hor-izontal and vertical downsampling in equirectangular projection (ERP). Then, we propose a novel region-dependent downsampling packing (RDDP) method using horizontal downsampling which exploits the uneven sampling characteristic of the ERP for stereoscopic 360∘videos. Experimental results, using the VVC (VVenC) encoder, show that compared with the standard side-by-side (SbS) format, the proposed RDDP method for both views provides an average of around 8.7% and 7% of Bjentegaard-Delta bitrate (BD-BR) reduction for Random Access (RA) and Low Delay B (LDB) configurations, respectively with almost the same encoding time of SbS on average.

The Affine motion estimation (AME) in Versatile Video Coding (VVC) can predict complex non-translational motions such as rotation, zoom, or shearing more effectively than the translational motion estimation (TME) tools, at the cost of greatly increased computational complexity. In this paper, to reduce encoding complexity, we propose a novel adjustable fast decision method for AME in VVC. Our method skips the AME process for blocks with low TME rate-distortion (RD) cost as we observed that skipping them reduces the encoding time without significantly affecting the compression performance. A distinctive feature of the proposed method is that it can be progressively adjusted to provide different compromises between speed-up and compression performance. First, a default TME RD cost threshold is estimated using a Multiple Linear Regression (MLR) model and then, adjusted to achieve the desired trade-off between speed-up and coding performance. Experimental results show that, with the default threshold, the proposed method can reduce the VTM encoding time by 8% on average, on classes B, C, and D, with a Bjøntegaard-Delta bitrate (BD-BR) of 0.44%. For the same classes and using 1.5 times the default threshold, it can reach 11% with a BD-BR of 0.82%.

The robustness and accuracy of the intensity-based 3D/2D registration of a 3D model on planar X-ray image(s) is related to the quality of the image correspondences between the digitally reconstructed radiographs (DRR) generated from the 3D models (varying image) and the X-ray images (fixed target). While much effort may be devoted to generating realistic DRR that are similar to real X-rays (using complex X-ray simulation, adding densities information in 3D models, etc.), significant differences still remain between DRR and real X-ray images. Differences such as the presence of adjacent or superimposed soft tissue and bony or foreign structures lead to image matching difficulties and decrease the 3D/2D registration performance. In the proposed method, the X-ray images were converted into DRR images using a GAN-based cross-modality image-to-images translation. With this added prior step of XRAY-to-DRR translation, standard similarity measures become efficient even when using simple and fast DRR projection. For both images to match, they must belong to the same image domain and essentially contain the same kind of information. The XRAY-to-DRR translation also addresses the well-known issue of registering an object in a scene composed of multiple objects by separating the superimposed or/and adjacent objects to avoid mismatching across similar structures. We applied the proposed method to the 3D/2D fine registration of vertebra deformable models to biplanar radiographs of the spine. We showed that the XRAY-to-DRR translation enhances the registration results, by increasing the capture range and decreasing dependence on the similarity measure choice since the multi-modal registration becomes mono-modal.

We address 3D human pose and shape estimations from multi-view images. We use the SMPL body model, and regress the model parameters that best fit the shape and pose. To solve for the parameters, we first compute 3Djoint positions from 2D joint estimations on images by using a linear algebraic triangulation. Then, we fit the 3D parametric body model to the 3Djoints while imposing a bone orientation constraint between the 3D model and the corresponding body parts detected in the images. We do so by minimizing a new set of objective functions through a two-step optimization process that provides a good initialization for the refinement of the shape and pose parameters. Our approach is evaluated on the Human3.6M and HumanEva benchmarks, showing superior results with respect to state-of-the-art methods.

In this article we propose a method to estimate the distance to the camera of a human subject from a monocular image without restricting the orientation of the head. The inter pupillary distance of the subject is used as anchor to estimate the size of pixels in the plane of the subject and the distance to the camera is recovered from this measurement. The inter pupillary distance in the image is estimated by using a learning-based approach that detects the head of the subject and estimates the orientation of the head with respect to the optical axis of the camera. It also proposes a method to estimate the pupillary distance from 2 images. The proposed approaches are validated by comparing the recovered distance to the real distance obtained with an OptiTrack motion-capture system. It is demonstrated that using the proposed algorithm allows a more robust estimation of the distance regardless of the head rotation.

Virtual reality (VR) streaming is impaired by the large amount of data required to deliver 360-degree video resulting in low-quality end user experience when network bandwidth is limited, or latency is high. To address these challenges, proposed in this paper is a novel method for viewpoint prediction for long-term horizons in VR streaming. This method uses a long short-term memory (LSTM) encoder-decoder network to carry out a sequence-to-sequence prediction. To enhance the results obtained by this network, experiments are performed using viewpoint information from users on low-latency networks. By applying an effective tile-based quality assignment after viewpoint prediction, a 61% average bandwidth reduction, with respect to the transmission of the whole ERP frame, is achieved along with a high-quality viewport rendered to the end user.

This paper presents an efficient method for encoding common projection formats in 360° video coding, in which we exploit inactive regions. These regions are ignored in the reconstruction of the equirectangular format or the viewport in virtual reality applications. As the content of these pixels is irrelevant, we neglect the corresponding pixel values in rate-distortion optimization, residual transformation, as well as in-loop filtering and achieve bitrate savings of up to 10%.

To date, 3D spine reconstruction from biplanar radiographs involves intensive user supervision and semi-automated methods that are time-consuming and not effective in clinical routine. This paper proposes a new, fast, and automated 3D spine reconstruction method through which a realistic statistical shape model of the spine is fitted to images using convolutional neural networks (CNN). The CNNs automatically detect the anatomical landmarks controlling the spine model deformation through a hierarchical and gradual iterative process. The performance assessment used a set of 68 biplanar radiographs, composed of both asymptomatic subjects and adolescent idiopathic scoliosis patients, in order to compare automated reconstructions with ground truths build using multiple experts-supervised reconstructions. The mean (SD) errors of landmark locations (3D Euclidean distances) were 1.6 (1.3) mm, 1.8 (1.3) mm, and 2.3 (1.4) mm for the vertebral body center, endplate centers, and pedicle centers, respectively. The clinical parameters extracted from the automated 3D reconstruction (reconstruction time is less than one minute) presented an absolute mean error between 2.8° and 4.7° for the main spinal parameters and between 1° and 2.1° for pelvic parameters. Automated and expert's agreement analysis reported that, on average, 89% of automated measurements were inside the expert's confidence intervals. The proposed automated 3D spine reconstruction method provides an important step that should help the dissemination and adoption of 3D measurements in clinical routine.

Recovering of the depth structure of a scene from monocular video content provides an important advantage in applications such as AR (placing and removing of objects) or 3D-TV and 3D cinema (2D-to-3D video conversion). In this paper, we present an automatic method to generate relative depth maps from monocular video sequences. It relies on the dynamic occlusion depth cue to recover the depth order of objects in the scene. The forward and backward motion analysis between each two consecutive frames allows the calculation of their dynamic occlusions. We estimate the motion using a modified version of the EpicFlow. Our modifications to this optical flow method made it coherent in forward-backward directions without compromising its performance. Thanks to this new feature, occlusions are simpler to calculate than the approaches used in the relevant literature. The obtained occlusions allow order deduction of the objects contained in the image. These objects are obtained using a segmentation approach which considers both color and motion. Ours results show a small improvement to the quality of the optical flow while adding the forward/backward coherence. With respect to the depth ordering our approach obtains slightly better results than the reference method while removing a computationally costly step from the processing.

Rate-constrained motion estimation (RCME) is the most computationally intensive task of H.265/HEVC encoding. Massively parallel architectures, such as graphics processing units (GPUs), used in combination with a multi-core central processing unit (CPU), provide a promising computing platform to achieve fast encoding. However, the dependencies in deriving motion vector predictors (MVPs) prevent the parallelization of prediction units (PUs) processing at a frame level. Moreover, the conditional execution structure of typical fast search algorithms is not suitable for GPUs designed for data-intensive parallel problems. In this paper, we propose a novel highly parallel RCME method based on multiple temporal motion vector (MV) predictors and a new fast nested diamond search (NDS) algorithm well-suited for a GPU. The proposed framework provides fine-grained encoding parallelism. Experimental results show that our approach provides reduced GPU load with better BD-Rate compared to prior full search parallel methods based on a single MV predictor.

Rate-constrained motion estimation (RCME) is considered to be the most time-consuming process of H.265/HEVC encoding. Massively parallel architectures, such as graphics processing units (GPUs), used in combination with a multi-core central processing unit (CPU), provide a promising computing platform to achieve fast encoding. However, the inherent dependencies in the process for deriving motion vector predictors (MVPs) prevent the parallelization of prediction units (PUs) processing. In this paper, we present a framework for performing a two-stage parallel RCME, in which the RCME of all the PUs of a frame can be calculated in parallel. A novel method is introduced to overcome the dependencies inherent to the derivation of MVPs. Multiple temporal predictors (MTPs) within the two-stage parallel RCME framework provide fine-grained parallelism encoding without significant BD-Rate penalty, compared to serial encoding. Experimental results show that our proposed approach achieves a BD-Rate improvement of over 1% as compared to state-of-the-art parallel methods providing similar time reductions.

Bony structure segmentation in radiographic images is an important tool for the diagnosis and treatment of orthopedic conditions. Current methods rely on the detection of single edges which often fail to correctly recover a structure's boundary when other nearby structures are present, as is the case in the hip joint. The use of minimal paths to detect separately two adjacent edges may lead to leakage of the femoral head contour into the acetabulum's edge due to small intensity variations and/or edge discontinuity. This article presents a new method for simultaneously detecting two adjacent edges in a radiographic image by using a novel 3D minimal path algorithm where interrelation constraints are incorporated. We apply this technique on radiographic images of hip joint in order to simultaneously extract adjacent bony contours of femoral head and acetabulum. We prove that the new algorithm improves the extraction of both contours in the region.

A novel data structure is proposed for magnitude-ordering 4D wavelet coefficients of wavelet-based multiview video coding. This data structure consists of temporal-view 2D zerotree data structures followed by spatial 2D zerotree data structures. Based on this 4D data structure, an algorithm is developed for the coding of 4D wavelet coefficients. Experiment results confirm that the proposed coding algorithm outperforms conventional algorithms that do not use the 4D zerotree data structure.

This paper proposes a method for the realtime detection of vandalism in video sequences. The proposed method detects vandalism through the robust extraction of a sequence of high-level events leading to it without resorting to object recognition and using a single camera. Vandalism is declared when an object enters the scene and causes an unauthorized change inside a predefined vandalisable area in the scene such as a pay phone or a sign. The proposed method was tested offline and on-line and our results show that it is robust in detecting vandalism or graffiti in surveillance video sequences.

This paper proposes a novel algorithm for the real-time detection and correction of occlusion and split in feature-based tracking of objects for surveillance applications. The proposed algorithm detects sudden variations of spatio-temporal features of objects in order to identify possible occlusion or split events. The detection is followed by a validation stage that uses past tracking information to prevent false detection of occlusion or split. Special care is taken in case of heavy occlusion, when there is a large superposition of objects. In this case the system relies on long-term temporal behavior of objects to avoid updating the video object features with unreliable (e.g. shape and motion) information. Occlusion is corrected by separating occluded objects. For the detection of splits, in addition to the analysis of spatio-temporal changes in objects features, our algorithm analyzes the temporal behavior of split objects to discriminate between errors in segmentation and real separation of objects, such as in the deposit of an object. Split is corrected by physically merging the objects detected to be split. To validate the proposed approach, objective and visual results are presented. Experimental results show the ability of the proposed algorithm to detect and correct, both, split and occlusion of objects. The proposed algorithm is most suitable in video surveillance applications due to: its good performance in multiple, heavy, and total occlusion; its distinction between real object separation and faulty object split; its handling of simultaneous occlusion and split events; and its low computational complexity.

The purpose of this study is to investigate a variational method for joint segmentation and parametric estimation of image motion by basis function representation of motion and level set evolution. The functional contains three terms. One term is of classic regularization to bias the solution toward a segmentation with smooth boundaries. A second term biases the solution toward a segmentation with boundaries which coincide with motion discontinuities, following a description of motion discontinuities by a function of the image spatio-temporal variations. The third term refers to region information and measures conformity of the parametric representation of the motion of each region of segmentation to the image spatio-temporal variations. The components of motion in each region of segmentation are represented as functions in a space generated by a set of basis functions. The coefficients of the motion components considered combinations of the basis functions are the parameters of representation. The necessary conditions for a minimum of the functional, which are derived taking into consideration the dependence of the motion parameters on segmentation, lead to an algorithm which condenses to concurrent curve evolution, implemented via level sets, and estimation of the parameters by least squares within each region of segmentation. The algorithm and its implementation are verified on synthetic and real images using a basis of cosine transforms.

Motion-compensated temporal filtering implemented using lifting is an effective and efficient temporal decomposition tool that facilitates video compression competitive with the current standards. As recently shown, however, in order that a lifting-based motion-compensated discrete wavelet transform indeed implements the intended filtering along motion trajectories, motion transformation must be invertible and motion composition between frames must be well-defined. A departure from these conditions results in the application of sub-optimal subband decomposition filters which, in turn, degrades coding performance, even if prediction-step energy is minimized during motion estimation. We study the impact of motion field invertibility error on the coding performance of an MCTF/DWT video coder. We propose two new motion field inversion methods and compare them to previously reported inversion techniques. We also compare coding results for all inversion algorithms with those of coding based on triangular meshes that are inherently invertible. Our results show that a significant improvement in coding performance is possible with more accurate motion field inversion.

This paper presents a novel approach to the reconstruction of images from nonuniformly spaced samples. This problem is often encountered in digital image processing applications. Nonrecursive video coding with motion compensation, spatiotemporal interpolation of video sequences, and generation of new views in multicamera systems are three possible applications. We propose a new reconstruction algorithm based on a spline model for images. We use regularization, since this is an ill-posed inverse problem. We minimize a cost function composed of two terms: one related to the approximation error and the other related to the smoothness of the modeling function. All the processing is carried out in the space of spline coefficients; this space is discrete, although the problem itself is of a continuous nature. The coefficients of regularization and approximation filters are computed exactly by using the explicit expressions of B-spline functions in the time domain. The regularization is carried out locally, while the computation of the regularization factor accounts for the structure of the nonuniform sampling grid. The linear system of equations obtained is solved iteratively. Our results show a very good performance in motion-compensated interpolation applications.

The purpose of this study is to investigate image segmentation from the viewpoint of image data regularized clustering. From this viewpoint, segmentation into a fixed but arbitrary number N of regions is stated as the simultaneous minimization of N - 1 energy functional, each involving a single region and its complement. The resulting Euler-Lagrange curve evolution equations yield a partition at convergence provided the curves are initialized so as to define an arbitrary partition of the image domain. The method is implemented via level sets, and results are shown on synthetic and natural vectorial images.

Active contours and level sets provide a solid formal framework for image segmentation. The problem, stated as the minimization of a functional containing terms of conformity to data and regularization, is solved by curve evolution implemented via level set partial differential equations (PDE). The purpose of this study is to investigate approximation by basis functions as a model for image representation in segmentation by level set PDE. This model is mathematically yielding, affords more generality than current piecewise constant and Gaussian models, and can be just as efficient as the most general piecewise smooth model. We state the problem using this model to measure conformity of segmentation to data. The resulting functional is minimized via level set evolution PDE. Experimental results are shown to demonstrate the formulation.

We are concerned with the reconstruction of a regularly-sampled image based on irregularly-spaced samples thereof. We propose a new iterative method based on a cubic spline representation of the image. An objective function taking into account the similarity to the known samples and the regularity of the function is minimized in order to obtain a good approximation. We apply the developed algorithm to motion-compensated image interpolation. Under motion compensation, the resulting sampling grids are irregular and require irregular/regular interpolation. We show experimental results on real-world images and we compare our results with other methods proposed in the literature.

We are concerned with the reconstruction of a regularly-sampled image based on irregularly-spaced samples thereof. We propose a new iterative method based on a wavelet representation of the image. For this representation we use a biorthogonal spline wavelet basis implemented on an oversampled grid. We apply the developed algorithm to disparity-compensated stereoscopic image interpolation. Under disparity compensation, the resulting sampling grids are irregular and require the irregular/regular interpolation. We show the experimental results on real-world images and we compare our results with other methods proposed in the literature.