High-quality depth and normal maps play a crucial role in various image processing and computer vision applications. However, the noisy data from sensors requires meticulous pre-processing. Denoising depth and normals separately, can lead to inconsistent geometric information, neg. atively impacting the performance of applications relying on this data. While some recent studies have suggested joint denoising approaches that aim to achieve coherent geometric representations, these methods are based on simple geometric assumptions, such as piecewise planar regions, and thus suffer from lower accuracy for non-planar geometry. In this paper, we present a versatile non-planar model specifically designed to handle both planar and non-planar geometry. To achieve this, we formulate the underlying problem as a partitioning of our non-planar model parameters on weighted finite graphs. Solving this problem involves utilizing a modified version of the Cut Pursuit algorithm, which efficiently divides input data into regions of similar geometric models. Finally, we leverage these resulting partitions and their associated model parameters to compute a denoised and infilled depth image, along with its coherent normal map.

Partitioning of images is a fundamental image processing task, which is closely related to various problems and applications in computer vision. Due to the hard nature of the underlying problem, existing algorithms are very compute intense. In this work we present a stochastic algorithm to efficiently approximate solutions of the image partitioning problem. Our contributions lie in the novel convex reformulation of the underlying graph cut problem, along with the application of a stochastic solver. These changes allow a faster convergence compared to other graph cut based methods, which is confirmed by our experiments.

Recently deep generative models have achieved impressive progress in modeling the distribution of training data. In this work, we present for the first time a generative model for 4D light field patches using variational autoencoders to capture the data distribution of light field patches. We develop a generative model conditioned on the central view of the light field and incorporate this as a prior in an energy minimization framework to address diverse light field reconstruction tasks. While pure learning-based approaches do achieve excellent results on each instance of such a problem, their applicability is limited to the specific observation model they have been trained on. On the contrary, our trained light field generative model can be incorporated as a prior into any model-based optimization approach and therefore extend to diverse reconstruction tasks including light field view synthesis, spatial-angular super resolution and reconstruction from coded projections. Our proposed method demonstrates good reconstruction, with performance approaching end-to-end trained networks, while outperforming traditional model-based approaches on both synthetic and real scenes. Furthermore, we show that our approach enables reliable light field recovery despite distortions in the input.

In this paper, we introduce a novel variational approach to estimate the scene flow from RGB-D images. We regularize the ill-conditioned problem of scene flow estimation in a unified framework by enforcing piecewise rigid motion through decomposition into rotational and translational motion parts. Our model crucially regularizes these components by an L0“norm”, thereby facilitating implicit motion segmentation in a joint energy minimization problem. Yet, we also show that this energy can be efficiently minimized by a proximal primal-dual algorithm. By implementing this approximate L0rigid motion regularization, our scene flow estimation approach implicitly segments the observed scene of into regions of nearly constant rigid motion. We evaluate our joint scene flow and segmentation estimation approach on a variety of test scenarios, with and without ground truth data, and demonstrate that we outperform current scene flow techniques.

Abstract The advent of affordable consumer grade RGB‐D cameras has brought about a profound advancement of visual scene reconstruction methods. Both computer graphics and computer vision researchers spend significant effort to develop entirely new algorithms to capture comprehensive shape models of static and dynamic scenes with RGB‐D cameras. This led to significant advances of the state of the art along several dimensions. Some methods achieve very high reconstruction detail, despite limited sensor resolution. Others even achieve real‐time performance, yet possibly at lower quality. New concepts were developed to capture scenes at larger spatial and temporal extent. Other recent algorithms flank shape reconstruction with concurrent material and lighting estimation, even in general scenes and unconstrained conditions. In this state‐of‐the‐art report, we analyze these recent developments in RGB‐D scene reconstruction in detail and review essential related work. We explain, compare, and critically analyze the common underlying algorithmic concepts that enabled these recent advancements. Furthermore, we show how algorithms are designed to best exploit the benefits of RGB‐D data while suppressing their often non‐trivial data distortions. In addition, this report identifies and discusses important open research questions and suggests relevant directions for future work.