Abstract Portrait synthesis creates realistic digital avatars which enable users to interact with others in a compelling way. Recent advances in StyleGAN and its extensions have shown promising results in synthesizing photorealistic and accurate reconstruction of human faces. However, previous methods often focus on frontal face synthesis and most methods are not able to handle large head rotations due to the training data distribution of StyleGAN. In this work, our goal is to take as input a monocular video of a face, and create an editable dynamic portrait able to handle extreme head poses. The user can create novel viewpoints, edit the appearance, and animate the face. Our method utilizes pivotal tuning inversion (PTI) to learn a personalized video prior from a monocular video sequence. Then we can input pose and expression coefficients to MLPs and manipulate the latent vectors to synthesize different viewpoints and expressions of the subject. We also propose novel loss functions to further disentangle pose and expression in the latent space. Our algorithm shows much better performance over previous approaches on monocular video datasets, and it is also capable of running in real‐time at 54 FPS on an RTX 3080.

Most face relighting methods are able to handle diffuse shadows, but struggle to handle hard shadows, such as those cast by the nose. Methods that propose techniques for handling hard shadows often do not produce geometrically consistent shadows since they do not directly leverage the estimated face geometry while synthesizing them. We propose a novel differentiable algorithm for synthesizing hard shadows based on ray tracing, which we incorporate into training our face relighting model. Our proposed algorithm directly utilizes the estimated face geometry to synthesize geometrically consistent hard shadows. We demonstrate through quantitative and qualitative experiments on Multi-PIE and FFHQ that our method produces more geometrically consistent shadows than previous face relighting methods while also achieving state-of-the-art face relighting performance under directional lighting. In addition, we demonstrate that our differentiable hard shadow modeling improves the quality of the estimated face geometry over diffuse shading models.

Single image 3D face reconstruction with accurate geometric details is a critical and challenging task due to the similar appearance on the face surface and fine details in organs. In this work, we introduce a self-supervised 3D face reconstruction approach from a single image that can recover detailed textures under different camera settings. The proposed network learns high-quality disparity maps from stereo face images during the training stage, while just a single face image is required to generate the 3D model in real applications. To recover fine details of each organ and facial surface, the framework introduces facial landmark spatial consistency to constrain the face recovering learning process in local point level and segmentation scheme on facial organs to constrain the correspondences at the organ level. The face shape and textures will further be refined by establishing holistic constraints based on the varying light illumination and shading information. The proposed learning framework can recover more accurate 3D facial details both quantitatively and qualitatively compared with state-of-the-art 3DMM and geometry-based reconstruction algorithms based on a single image.

Existing face relighting methods often struggle with two problems: maintaining the local facial details of the subject and accurately removing and synthesizing shadows in the relit image, especially hard shadows. We propose a novel deep face relighting method that addresses both problems. Our method learns to predict the ratio (quotient) image between a source image and the target image with the desired lighting, allowing us to relight the image while maintaining the local facial details. During training, our model also learns to accurately modify shadows by using estimated shadow masks to emphasize on the high-contrast shadow borders. Furthermore, we introduce a method to use the shadow mask to estimate the ambient light intensity in an image, and are thus able to leverage multiple datasets during training with different global lighting intensities. With quantitative and qualitative evaluations on the Multi-PIE and FFHQ datasets, we demonstrate that our proposed method faithfully maintains the local facial details of the subject and can accurately handle hard shadows while achieving state-of-the-art face relighting performance.

We propose a novel real-time selfie video stabilization method. Our method is completely automatic and runs at 26 fps. We use a 1D linear convolutional network to directly infer the rigid moving least squares warping which implicitly balances between the global rigidity and local flexibility. Our network structure is specifically designed to stabilize the background and foreground at the same time, while providing optional control of stabilization focus (relative importance of foreground vs. background) to the users. To train our network, we collect a selfie video dataset with 1005 videos, which is significantly larger than previous selfie video datasets. We also propose a grid approximation to the rigid moving least squares that enables the real-time frame warping. Our method is fully automatic and produces visually and quantitatively better results than previous real-time general video stabilization methods. Compared to previous offline selfie video methods, our approach produces comparable quality with a speed improvement of orders of magnitude. Our code and selfie video dataset is available at https://github.com/jiy173/selfievideostabilization.

Deep neural networks have significantly enhanced the performance of various computer vision tasks, including single image depth estimation and image segmentation. However, most existing approaches handle them in supervised manners and require a large number of ground truth labels that consume extensive human efforts and are not always available in real scenarios. In this paper, we propose a novel framework to estimate disparity maps and segment images simultaneously by jointly training an encoder-decoder-based interactive convolutional neural network (CNN) for single image depth estimation and a multiple class CNN for image segmentation. Learning the neural network for one task can be beneficial from simultaneously learning from another one under a multi-task learning framework. We show that our proposed model can learn per-pixel depth regression and segmentation from just a single image input. Extensive experiments on available public datasets, including KITTI, Cityscapes urban, and PASCAL-VOC demonstrate the effectiveness of our model compared with other state-of-the-art methods for both tasks.

We propose an automatic parametric human body reconstruction algorithm which can efficiently construct a model using a single Kinect sensor. A user needs to stand still in front of the sensor for a couple of seconds to measure the range data. The user's body shape and pose will then be automatically constructed in several seconds. Traditional methods optimize dense correspondences between range data and meshes. In contrast, our proposed scheme relies on sparse key points for the reconstruction. It employs regression to find the corresponding key points between the scanned range data and some annotated training data. We design two kinds of feature descriptors as well as corresponding regression stages to make the regression robust and accurate. Our scheme follows with dense refinement where a pre-factorization method is applied to improve the computational efficiency. Compared with other methods, our scheme achieves similar reconstruction accuracy but significantly reduces runtime.

We present in this paper a method to reconstruct passive millimeter-wave images. We formulate the Graph Cuts concept to estimate the image by performing a global optimization through all the samples of the passive millimeter-wave radiometer. Our scheme assigns labels for each pixel to be reconstructed by defining a cost function consisting of two terms. The first term reflects the cost that each pixel can have by taking a certain label value. The second term ensures that the image is piecewise smooth. For the latter, we assess the performance of several discontinuity cost models to find out which model is best suited to our problem. Compared to state of the art schemes, results show that our method leads to images with enhanced contrast and the shapes of the objects are better preserved. Compared to optimization schemes like belief propagation, the graph cuts scheme leads to improved temperature resolution and is more robust against label assignment.

We present a robust method to reconstruct a millimeter-wave image from a passive sensor. The method operates directly on the raw samples from the radiometer. It allocates for each pixel to be estimated a patch in the space formed by all the raw samples of the image. It then estimates the noise in the patch by measuring some distances that reflect how far the samples are from forming a piecewise smooth surface. It then allocates a weight for each sample that defines its contribution in the pixel reconstruction. This is done via a smoothing Kernel that enforces the distances to have a piecewise smooth variation inside the patch. Results on real datasets show that our scheme leads to more contrast and less noise and the shape of an object is better preserved in a constructed image compared to state-of-the-art schemes. The proposed scheme produces better results even with low integration times, i.e., 10% of the total integration time used in our experiments.

Determining the pose of a moving camera is an important task in computer vision. In this paper, we derive a projective Newton algorithm on the manifold to refine the pose estimate of a camera. The main idea is to benefit from the fact that the 3-D rigid motion is described by the special Euclidean group, which is a Riemannian manifold. The latter is equipped with a tangent space defined by the corresponding Lie algebra. This enables us to compute the optimization direction, i.e., the gradient and the Hessian, at each iteration of the projective Newton scheme on the tangent space of the manifold. Then, the motion is updated by projecting back the variables on the manifold itself. We also derive another version of the algorithm that employs homeomorphic parameterization to the special Euclidean group. We test the algorithm on several simulated and real image data sets. Compared with the standard Newton minimization scheme, we are now able to obtain the full numerical formula of the Hessian with a 60% decrease in computational complexity. Compared with Levenberg-Marquardt, the results obtained are more accurate while having a rather similar complexity.

We present a new technique to reconstruct passive millimeter-wave images. To do that, we formulate the reconstruction as a global optimization through all the raw samples of the radiometer. The method assigns some labels for each pixel to be reconstructed. We then define a cost function consisting of two terms. The first one reflects the cost that each pixel assumes by taking a certain label value. The second one preserves the property of the radiometric image of being piecewise smooth. We apply belief propagation after that to minimize the error function and find the globally optimal value for each pixel. Our scheme leads to images with enhanced contrast and preserves the shape of the objects. We confirm these points via a comparison with several state of the art methods on real radiometric data.

We present in this work a robust method to reconstruct millimeter-wave images from passive sensors. The method estimates each pixel of the image by applying a locally adaptive smoothing filter to the samples of the sensor. The filter assigns for each sample in the neighborhood of the pixel under consideration a weighting factor that determines its contribution. The filter preserves the piecewise smoothness of the image since it applies the multi-quadric splines combined with some adaptive distances to smooth the samples while preserving the edges. Results show that the our scheme leads to more contrast, less noise and the shape of the objects are better preserved in the constructed images when compared to state of the art methods.

The future of video coding for 3DTV lies in the combination of depth maps and corresponding textures. Most current video coding standards, however, are only optimized for visual quality and are not able to efficiently compress depth maps. We present in this work a content adaptive depth map meshing with tritree and entropy encoding for 3D videos. We show that this approach outperforms the intra frame prediction of AVC/H.264 for the coding of depth maps of still images. We also demonstrate by combining AVC/H.264 with our algorithm that we are able to increase the visual quality of the encoded texture on average by 6 dB. This work is currently limited to still images but an extension to intra coding of 3D video is straightforward.

We propose in this paper a new scheme based on compressed sensing to compress a depth map. We first subsample the entity in the frequency domain to take advantage of its compressibility. We then derive a reconstruction scheme to recover the original map from the subsamples using a non-linear conjugate gradient minimization scheme. We preserve the discontinuities of the depth map at the edges and ensure its smoothness elsewhere by incorporating the Total Variation constraint in the minimization. The results we obtained on various test depth maps show that the proposed method leads to lower error rate at high compression ratio when compared to standard image compression techniques like JPEG and JPEG 2000.

In several telepresence applications the transmitting end is resource constrained, hence multi-view video has to be source coded and transmitted to a location where stereo matching is performed. The effect of video source coding on stereo matching techniques is presently not well understood. In this paper, we present an analysis which quantitatively demonstrates the impact of standard video coding techniques on the quality of well-known stereo matching algorithms. We propose a quantitative evaluation methodology and framework. Evaluation is also presented for the state-of-the-art Annex H of H.264/AVC (multi-view coding). The results shed a light on the tradeoff between video coding parameters and the quality of dense stereo matching. This enables us to select suitable operating regions for both stereo matching and video coding, for a diverse range of applications. In addition, various system configurations with varying performance-complexity tradeoff are also compared, giving insight on selecting the configuration suitable for a variety of telepresence applications.

The application of zoom camera lenses in machine vision has gained a lot of attention lately. The main difficulty in their employment lies in the accurate estimation of their intrinsic parameters. In this paper, we propose novel approaches to determine these parameters by estimating continuous models of their variations as the focus and the zoom change. The first method is based on the moving least squares (MLS) multiple regression scheme which determines from a predefined number of samples, the complete model of the intrinsic parameters. MLS fits a polynomial function at each focus and zoom setting by using the measured neighboring points. In order to reduce the computational complexity of MLS, we propose another algorithm in which the MLS generated curves are clustered. Then, each cluster is modeled with a single polynomial function. This decreases the complexity of computations for the applications where delay is critical, e.g., telepresence, to the evaluation of simple polynomials. Compared to previous techniques, the proposed algorithms lead to a noticeable increase in the estimation accuracy of the intrinsic parameters. In addition, they are able to generate accurate models of these parameters with only a few measurement points.

The interest in content adaptive mesh generation of images has been arising lately due to its wide area of applications in image processing. The major issue is to represent an image with a low number of pixels while preserving its content. These pixels or the nonuniform samples are then used to generate a mesh that approximates the corresponding image. This work presents a novel method based on Binary Space Partitions in combination with three clustering schemes to approximate an image with a mesh. The algorithm has the ability to simultaneously reduce the number of pixels and generate the mesh approximation. The idea is to assume each triangle of the mesh as a plane. Consequently, it will be possible to reconstruct the inlying pixels with planar equations defined from the three nodes of each triangle. If a triangle's equation does not have the ability to reconstruct the pixels lying within up to a predefined error, it is split into two new triangles. Tested on several real images, the proposed method leads to reduced size meshes in a fast manner while retaining the visual quality of the reconstructed images. In addition, it is parallelizable due to the property of Binary Space Partitions which facilitates its application in real-time scenarios.

Content adaptive mesh generation is an important research area with many applications in image processing and computer vision. The main issue is to represent an image with the pixels that preserve most of the amount of its information. The obtained pixels are then used to generate a mesh that approximates the original image. This work presents a novel iterative method that simultaneously reduces the number of the pixels and generates the mesh approximation of an image. The main idea is to incorporate binary space partitions along with singular value decomposition to cluster the pixels into planes and thus the nodes of the mesh are nothing but the pixels that define each plane. Compared to previous techniques, the proposed method leads to a 30% reduction in the size of the approximating mesh. In addition, the method minimizes the artifacts obtained from the reconstruction of the original image from the approximating mesh.

Depth from stereo is an important research field in computer vision due to the wide range of its applications. In this work, we present a stereo matching algorithm based on belief propagation (BP). The algorithm is designed to work on sparse images originating from image content adaptive mesh representation techniques. There, an image is approximated with a mesh. The nodes of the mesh are the non-uniform samples which are the ones that form the sparse image. The key issue in the proposed method is to formulate BP such that it matches a sparse left stereo image with a dense right image to obtain a sparse depth map. Moreover, we propose a simple method that recovers the dense disparity map of the scene from the sparse one using the approximating mesh of the image. The results obtained show that the proposed method leads to an average of 40% improvement in the quality of depth maps when compared to existing sparse stereo matching techniques.

The video shows a rich multi-modal multi-user telepresence system, which was developed within the SFB453 funded by the German Research Foundation (www.sfb453.de). As a complex application scenario, the remote repairing of a broken pipe is presented in this paper. The system basically consists of two operator-teleoperator- pairs. While one of the operators interacts with a stationary human- system-interface, the other operator uses a mobile one. Both systems provide visual, auditory, and haptic feedback and enable to control the motion of head, arms, and grippers as well as the locomotion of the corresponding teleoperator.

The aim of genetic mapping is to locate the loci responsible for specific traits such as complex diseases. These traits are normally caused by mutations at multiple loci of unknown locations and interactions. In this work, we model the biological system that relates DNA polymorphisms with complex traits as a linear mixing process. Given this model, we propose a new fine-scale genetic mapping method based on independent component analysis. The proposed method outputs both independent associated groups of SNPs in addition to specific associated SNPs with the phenotype. It is applied to a clinical data set for the Schizophrenia disease with 368 individuals and 42 SNPs. It is also applied to a simulation study to investigate in more depth its performance. The obtained results demonstrate the novel characteristics of the proposed method compared to other genetic mapping methods. Finally, we study the robustness of the proposed method with missing genotype values and limited sample sizes.

Extracting the motion parameters of a moving camera is an important issue in computer vision. This is due to the need of numerous emerging applications like telepresence and robot navigation. The key issue is to determine a robust estimate of the (3×3) essential matrix with its five degrees of freedom. In this work, a robust technique to compute the essential matrix is suggested under the assumption that the images are calibrated. The algorithm is a combination of the five-point relative pose problem using an optimization technique on a manifold, with the random sample consensus. The results show that the proposed method delivers faster and more accurate results than the standard techniques.

The accuracy of computer vision systems is highly dependent on the correct estimates of the camera intrinsic parameters. This accuracy is important in numerous applications like telepresence and robot navigation. In this work, a novel technique is proposed to model the variation of the camera's intrinsic parameters as a function of the focus and the zoom. The proposed method computes the complete surfaces of the intrinsic parameters from a predefined number of focus/zoom measurements using a moving least-squares (MLS) regression technique. Then, it approximates the generated MLS surfaces by employing adaptive Delaunay meshes. Compared to a previous technique using bivariate polynomial functions, the new method results in a 94% enhancement of the mean estimation error. In addition, the new method leads to the same accuracy of the results as compared to a previous version of the MLS technique while requiring a less amount of computations.

Determining the model-order of a given data set is an important task in signal analysis. Principal component analysis (PCA) can be used for this purpose if there is a criterion upon which the correct order can be chosen. In this work, we propose a new and simple technique to determine automatically the rank of a PCA model. Tested with simulated data, the algorithm is able to determine the correct model order efficiently. Applied to video sequences, this method is able to estimate the necessary subspaces that capture the motion and illuminance changes within the different frames. This helps in reducing the storage need/requirements of video sequences and improves the efficiency of context based search and retrieval techniques.

Identifying the causal genetic markers responsible for complex diseases is a main aim in human genetics. In the context of complex diseases, which are believed to have multiple causal loci of largely unknown effect and position, there is a need to develop advanced methods for gene mapping. In this work, we propose a novel algorithm based on independent component analysis for gene mapping. To apply the algorithm, we model the intra-cellular interactions as a mixing process of multiple sources. Results prove the superiority of the proposed algorithm over conventional statistical based methods, and demonstrate yet another successful application of a well known signal processing technique to an important problem in the field of human genetics.