Denoising Diffusion Models (DDMs) have emerged as a strong competitor to Generative Adversarial Networks (GANs). However, despite their widespread use in image synthesis and editing applications, their latent space is still not as well understood. Recently, a semantic latent space for DDMs, coined ‘$h$-space’, was shown to facilitate semantic image editing in a way reminiscent of GANs. The$h$-space is comprised of the bottleneck activations in the DDM's denoiser across all timesteps of the diffusion process. In this paper, we explore the properties of$h$-space and propose several novel methods for finding meaningful semantic directions within it. We start by studying unsupervised methods for revealing interpretable semantic directions in pretrained DDMs. Specifically, we show that interpretable directions emerge as the principal components in the latent space. Additionally, we provide a novel method for discovering image-specific semantic directions by spectral analysis of the Jacobian of the denoiser w.r.t. the latent code. Next, we extend the analysis by finding directions in a supervised fashion in unconditional DDMs. We demonstrate how such directions can be found by annotating generated samples with a domain-specific attribute classifier. We further show how to semantically disentangle the found directions by simple linear projection. Our approaches are applicable without requiring any architectural modifications, text-based guidance, CLIP-based optimization, or model fine-tuning.

Recording real life human motion as a skinned mesh animation with an acceptable quality is usually difficult. Even though recent advances in pose estimation have enabled motion capture from off-the-shelf webcams, the low quality makes it infeasible for use in production quality animation. This work proposes to use recent advances in the prediction of human motion through neural networks to augment low quality human motion, in an effort to bridge the gap between cheap recording methods and high quality recording. First, a model, competitive with prior work in short-term human motion prediction, is constructed. Then, the model is trained to clean up motion from two low quality input sources, mimicking a real world scenario of recording human motion through two webcams. Experiments on simulated data show that the model is capable of significantly reducing noise, and it opens the way for future work to test the model on annotated data.

In this work we present a method that combines tensor-based face modelling and analysis and non-rigid structure-from-motion (NRSFM). The core idea is to see that the conventional tensor formulation for the face structure and expression analysis can be utilised while the structure component can be directly analysed as the non-rigid structure-from-motion problem. To the NRSFM problem part we further present a novel prior-free approach that factorises the 2D input shapes into affine projection matrices, rank-one 3D affine basis shapes, and the basis shape coefficients. The linear combination of the basis shapes thus yields the recovered 3D shapes upto an affine transformation. In contrast to most works in literature, no calibration information of the cameras or structure prior is required. Experiments on challenging face datasets show that our method, with and without the metric upgrade, is accurate and fast when compared to the state-of-the-art and is well suitable for dense reconstruction and face editing.

In this paper, we propose TGAN a tensor-based method for modeling the latent space of generative models. The objective is to identify semantic directions in latent space. To this end, we propose to fit a multilinear tensor model on a structured facial expression database, which is initially embedded into latent space. We validate our approach on StyleGAN trained on FFHQ using BU-3DFE as a structured facial expression database. We show how the parameters of the multilinear tensor model can be approximated by Alternating Least Squares. Further, we introduce a stacked style-separated tensor model, defined as an ensemble of style-specific models to integrate our approach with the extended latent space of StyleGAN. We show that taking the individual styles of the extended latent space into account leads to higher model flexibility and lower reconstruction error. Finally, we do several experiments comparing our approach to former work on both GANs and multilinear models. Concretely, we analyze the expression subspace and find that the expression trajectories meet at an apathetic face that is consistent with earlier work. We also show that by changing the pose of a person, the generated image from our approach is closer to the ground truth than results from two competing approaches.

In this work, we present a new versatile 3D multilinear statistical face model, based on a tensor factorisation of 3D face scans, that decomposes the shapes into person and expression subspaces. Investigation of the expression subspace reveals an inherent low-dimensional substructure, and further, a star-shaped structure. This is due to two novel findings. (1) Increasing the strength of one emotion approximately forms a linear trajectory in the subspace. (2) All these trajectories intersect at a single point - not at the neutral expression as assumed by almost all prior works-but at an apathetic expression. We utilise these structural findings by reparameterising the expression subspace by the fourth-order moment tensor centred at the point of apathy. We propose a 3D face reconstruction method from single or multiple 2D projections by assuming an uncalibrated projective camera model. The non-linearity caused by the perspective projection can be neatly included into the model. The proposed algorithm separates person and expression subspaces convincingly, and enables flexible, natural modelling of expressions for a wide variety of human faces. Applying the method on independent faces showed that morphing between different persons and expressions can be performed without strong deformations.

Indexing and retrieving music based on emotion is a powerful retrieval paradigm with many applications. Traditionally, studies in the field of music emotion recognition have focused on training and testing supervised machine learning models using a single music dataset. To be useful for today's vast music libraries, however, such machine learning models must be widely applicable beyond the dataset for which they were created. In this work, we analyze to what extent models trained on one music dataset can predict emotion in another dataset constructed using a different methodology, by conducting cross-dataset experiments with three publicly available datasets. Our results suggest that training a prediction model on a homogeneous dataset with carefully collected emotion annotations yields a better foundation than prediction models learned on a larger, more varied dataset, with less reliable annotations.

In this paper, we present a new statistical model for human faces. Our approach is built upon a tensor factorisation model that allows controlled estimation, morphing and transfer of new facial shapes and expressions. We propose a direct parametrisation and regularisation for person and expression related terms so that the training database is well utilised. In contrast to existing works we are the first to reveal that the expression subspace is star shaped. This stems from the fact that increasing the strength of an expression approximately forms a linear trajectory in the expression subspace, and all these linear trajectories intersect in a single point which corresponds to the point of no expression or the point of apathy. After centring our analysis to this point, we then demonstrate how the dimensionality of the expression subspace can be further reduced by projection pursuit with the help of the fourth-order moment tensor. The results show that our method is able to achieve convincing separation of the person specific and expression subspaces as well as flexible, natural modelling of facial expressions for wide variety of human faces. By the proposed approach, one can morph between different persons and different expressions even if they do not exist in the database. In contrast to the state-of-the-art, the morphing works without causing strong deformations. In the application of expression classification, the results are also better.

In this paper, we address the problem of imaging membrane proteins for single-particle cryo-electron microscopy reconstruction of the isolated protein structure. More precisely, we propose a method for learning and removing the interfering vesicle signals from the micrograph, prior to reconstruction. In our approach, we estimate the subspace of the vesicle structures and project the micrographs onto the orthogonal complement of this subspace. We construct a 2D statistical model of the vesicle structure, based on higher order singular value decomposition (HOSVD), by considering the structural symmetries of the vesicles in the polar coordinate plane. We then propose to lift the HOSVD model to a novel hierarchical model by summarizing the multidimensional HOSVD coefficients by their principal components. Along with the model, a solid vesicle normalization scheme and model selection criterion are proposed to make a compact and general model. The results show that the vesicle structures are accurately separated from the background by the HOSVD model that is also able to adapt to the asymmetries of the vesicles. This is a promising result and suggests even wider applicability of the proposed approach in learning and removal of statistical structures.

In this study, the authors propose a multi‐view stereo reconstruction method which creates a three‐dimensional point cloud of a scene from multiple calibrated images captured from different viewpoints. The method is based on a prioritised match expansion technique, which starts from a sparse set of seed points, and iteratively expands them into neighbouring areas by using multiple expansion stages. Each seed point represents a surface patch and has a position and a surface normal vector. The location and surface normal of the seeds are optimised using a homography‐based local image alignment. The propagation of seeds is performed in a prioritised order in which the most promising seeds are expanded first and removed from the list of seeds. The first expansion stage proceeds until the list of seeds is empty. In the following expansion stages, the current reconstruction may be further expanded by finding new seeds near the boundaries of the current reconstruction. The prioritised expansion strategy allows efficient generation of accurate point clouds and their experiments show its benefits compared with non‐prioritised expansion. In addition, a comparison to the widely used patch‐based multi‐view stereo software shows that their method is significantly faster and produces more accurate and complete reconstructions.

We present a fully automated framework for scoring a patient's risk of cardiovascular disease (CVD) and mortality from a standard lateral radiograph of the lumbar aorta. The framework segments abdominal aortic calcifications for computing a CVD risk score and performs a survival analysis to validate the score. Since the aorta is invisible on X-ray images, its position is reasoned from 1) the shape and location of the lumbar vertebrae and 2) the location, shape, and orientation of potential calcifications. The proposed framework follows the principle of Bayesian inference, which has several advantages in the complex task of segmenting aortic calcifications. Bayesian modeling allows us to compute CVD risk scores conditioned on the seen calcifications by formulating distributions, dependencies, and constraints on the unknown parameters. We evaluate the framework on two datasets consisting of 351 and 462 standard lumbar radiographs, respectively. Promising results indicate that the framework has potential applications in diagnosis, treatment planning, and the study of drug effects related to CVD.

We have developed a breast coordinate system that is based on breast anatomy to register female breasts into a common coordinate frame in 2-D mediolateral (ML) or mediolateral oblique (MLO) view mammograms. The breasts are registered according to the location of the pectoral muscle and the nipple and the shape of the breast boundary because these are the most robust features independent of the breast size and shape. On the basis of these landmarks, we have constructed a nonlinear mapping between the parameter frame and the breast region in the mammogram. This mapping makes it possible to identify the corresponding positions and orientations among all of the ML or MLO mammograms, which facilitates an implicit use of the registration, i.e., no explicit image warping is needed. We additionally show how the coordinate transform can be used to extract Gaussian derivative features so that the feature positions and orientations are registered and extracted without nonlinearly deforming the images. We use the proposed breast coordinate transform in a cross-sectional breast cancer risk assessment study of 490 women, in which we attempt to learn breast cancer risk factors from mammograms that were taken prior to when the breast cancer became visible to a radiologist. The coordinate system provides both the relative position and orientation information on the breast region from which the features are derived. In addition, the coordinate system can be used in temporal studies to pinpoint anatomically equivalent locations between the mammograms of each woman and among the mammograms of all of the women in the study. The results of the cross-sectional study show that the classification into cancer and control groups can be improved by using the new coordinate system, compared to other systems evaluated. Comparisons were performed using the area-under-the-receiver-operating-characteristic-curve score. In general, the new coordinate system makes an accurate anatomical registration of breasts possible, which suggests its wide applicability wherever 2-D mammogram registration is required.

This paper proposes a method for computing a quasi-dense set of matching points between three views of a scene. The method takes a sparse set of seed matches between pairs of views as input and then propagates the seeds to neighboring regions. The proposed method is based on the best-first match propagation strategy, which is here extended from two-view matching to the case of three views. The results show that utilizing the three-view constraint during the correspondence growing improves the accuracy of matching and reduces the occurrence of outliers. In particular, compared with two-view stereo, our method is more robust for repeating texture. Since the proposed approach is able to produce high quality depth maps from only three images, it could be used in multi-view stereo systems that fuse depth maps from multiple views.

In this paper, we propose how the parameter distributions of multilinear geometric entities can be dualised. The dualisation concern, for example, the parameter distributions of conics, multiple view tensors, homographies, or as simple entities as points, lines, and planes. The dual distributions are related to Triggs' joint feature distributions but our approach is different in certain fundamental aspects. Our starting point is in the assumption that the maximum likelihood estimate, or the corresponding robust estimate, and the covariance matrix of the parameters of the geometric entity are available. We then use the asymptotic normality property of the MLE which allows us to transform the parameter uncertainty distribution in a dual form. The dualisation of the parameter distribution allows us, for instance, to look at the uncertainty distributions in feature distributions, which are essentially tied to the distribution of training data, and helps us to derive conditional distributions for point or line transfer and characterise confidence intervals of the estimates. Applications of the proposed approach are thus uncertainty analysis, statistical prediction, probabilistic transfer, etc.

In this paper, we present a non-rigid quasi-dense matching method and its application to object recognition and segmentation. The matching method is based on the match propagation algorithm which is here extended by using local image gradients for adapting the propagation to smooth non-rigid deformations of the imaged surfaces. The adaptation is based entirely on the local properties of the images and the method can be hence used in non-rigid image registration where global geometric constraints are not available. Our approach for object recognition and segmentation is directly built on the quasi-dense matching. The quasi-dense pixel matches between the model and test images are grouped into geometrically consistent groups using a method which utilizes the local affine transformation estimates obtained during the propagation. The number and quality of geometrically consistent matches is used as a recognition criterion and the location of the matching pixels directly provides the segmentation. The experiments demonstrate that our approach is able to deal with extensive background clutter, partial occlusion, large scale and viewpoint changes, and notable geometric deformations.

This paper shows how the random sampling, M-estimators, random walk can be combined to create a consistent sampler for generic models in problems that are difficult due to outliers and multimodality of the solution. Our method contains three major steps: (1) finding the local peaks of the selected robust cost function (M-estimator) using seeds from random sampling of minimal configurations, (2) constructing an approximation of the posterior density by using the local Hessian approximations of the cost function, and (3) sampling by the Metropolis-Hastings selection rule with a mixture proposal distribution containing both draws from the approximated posterior density and random walk to achieve consistence of the samples. The experiments verify that the sampler has much better mixing properties than a conventional random walk sampler. Thus, the approach is promising MCMC method for Bayesian inference that need to numerically evaluate integrals over the posterior density.

This paper presents an interesting observation that epipolar geometry and log-polar transform can be naturally combined by setting the center of the log-polar transform into the epipoles. This choice preserves the linearity of the epipolar lines. Moreover, the setting is especially advantageous, when camera moves towards the optical axis of the camera as the log-polar transform compensates the large scale changes in the scene. This practically implies that conventional matching techniques can be used with wide baseline images. We discuss the approach with both calibrated and uncalibrated cameras and show some dense wide baseline reconstruction examples where the epipoles are close to the image centers.

In this paper we propose extensions to the match propagation algorithm which is a technique for computing quasi-dense point correspondences between two views. The extensions make the match propagation applicable for wide baseline matching, i.e., for cases where the camera pose can vary a lot between the views. Our first extension is to use a local affine model for the geometric transformation between the images. The estimate of the local transformation is obtained from affine covariant interest regions which are used as seed matches. The second extension is to use the second order intensity moments to adapt the current estimate of the local affine transformation during the propagation. This allows a single seed match to propagate into regions where the local transformation between the views differs from the initial one. The experiments with real data show that the proposed techniques improve both the quality and coverage of the quasi-dense disparity map.

In conventional tomography, the interior of an object is reconstructed from tomographic projections such as X-ray or transmission electron microscope images. All the current reconstruction methods assume that projection geometry of the imaging device is either known or solved in advance by using e.g., fiducial or nonfiducial feature points in the images. In this paper, we propose a novel approach where the imaging geometry is solved simultaneously with the volume reconstruction problem while no correspondence information is needed. Our approach is a direct application of Bayesian inversion theory and produces the maximum likelihood or maximum a posteriori estimates for the motion parameters under the selected noise and prior distributions. In this paper, the method is implemented for a two-dimensional model problem with one-dimensional affine projection data. The performance of the method is tested with simulated and measured X-ray projection data.

Fish-eye lenses are convenient in such applications where a very wide angle of view is needed, but their use for measurement purposes has been limited by the lack of an accurate, generic, and easy-to-use calibration procedure. We hence propose a generic camera model, which is suitable for fish-eye lens cameras as well as for conventional and wide-angle lens cameras, and a calibration method for estimating the parameters of the model. The achieved level of calibration accuracy is comparable to the previously reported state-of-the-art.

In this paper, we are going to answer the following question: assuming that we have estimates for the epipolar geometry and its uncertainty between two views, how probable is it that a new, independent point pair will satisfy the true epipolar geometry and be, in this sense, a feasible candidate correspondence pair? If we knew the true fundamental matrix, the answer would be trivial but in reality we do not know it because of estimation errors. So, as an independent point in the first view is given, we will show we may compute the point-probability-density function, termed as the epipolar pdf, for the epipolar line points in the second view that describes the current level of knowledge of the epipolar geometry between the views. This point-point-probability-density relation is a probabilistic form of the epipolar constraint that also approaches the true point-line relation as the number of training correspondences tends to infinity. In this paper, we will also show that the eigenvectors of the epipolar line covariance matrix have certain interpretations on the image plane, of which one is the previously observed, narrowest point of the epipolar envelope. The results of this paper are important since the uncertainty of the epipolar constraint can be now taken into account in a sound way in applications.

We propose, in maximum likelihood sense, an optimal method for the affine fundamental matrix estimation in the presence of both Gaussian noise and outliers. It is based on weighting the squared residuals by the iteratively completed, residual posterior probabilities to be relevant. The proposed principle is also used for the covariance matrix estimation of the affine F-matrix where the novelty is in the fact that all data is used rather than the (erroneously) relevant classified matching points. The experiments on both synthetic and real data verify the optimality of the method in the sense of both false matches and Gaussian noise in data.

We have developed a simple and efficient wavelet-based technique for matching points in uncalibrated images. The results show that with the proposed method the probability for obtaining a false match practically vanishes when natural images are used. We also show how the uncertainty of the fundamental matrix can be interpreted from image matching viewpoint and explain how the disparity information incorporated in the fundamental matrix covariance can be utilized.

In this article the use of shape features in content-based image retrieval is studied. The emphasis is on techniques which do not demand object segmentation. PicSOM, the image retrieval system used in the experiments, requires that features are represented by constant-sized feature vectors for which the Euclidean distance can be used as a similarity measure. The shape features suggested here are edge histograms and Fourier transform based features computed for an edge image in Cartesian and polar coordinate planes. The results show that both local and global shape features are important clues of shapes in an image.