We investigate the capabilities of neural inverse procedural modeling to infer high-quality procedural yarn models with fiber-level details from single images of depicted yarn samples. While directly inferring all parameters of the underlying yarn model based on a single neural network may seem an intuitive choice, we show that the complexity of yarn structures in terms of twisting and migration characteristics of the involved fibers can be better encountered in terms of ensembles of networks that focus on individual characteristics. We analyze the effect of different loss functions including a parameter loss to penalize the deviation of inferred parameters to ground truth annotations, a reconstruction loss to enforce similar statistics of the image generated for the estimated parameters in comparison to training images as well as an additional regularization term to explicitly penalize deviations between latent codes of synthetic images and the average latent code of real images in the encoder’s latent space. We demonstrate that the combination of a carefully designed parametric, procedural yarn model with respective network ensembles as well as loss functions even allows robust parameter inference when solely trained on synthetic data. Since our approach relies on the availability of a yarn database with parameter annotations and we are not aware of such a respectively available dataset, we additionally provide, to the best of our knowledge, the first dataset of yarn images with annotations regarding the respective yarn parameters. For this purpose, we use a novel yarn generator that improves the realism of the produced results over previous approaches.

Photo-realistic 3D content is crucial for creating convincing digital images. For certain classes of everyday objects, such as food, the human perception is sensitive to even small inconsistencies, making the creation of such content difficult and time-consuming, even for experts. In this sketch, we will explore the use of an automated pipeline for capturing 3D shape and Bidirectional Texture Function of food. The acquired data is used to render photo-realistic images of purely virtual objects under arbitrary lighting and with full global illumination.

Abstract State‐of‐the‐art car paint shows not only interesting and subtle angular dependency but also significant spatial variation. Especially in sunlight these variations remain visible even for distances up to a few meters and give the coating a strong impression of depth which cannot be reproduced by a single BRDF model and the kind of procedural noise textures typically used. Instead of explicitly modeling the responsible effect particles we propose to use image‐based reflectance measurements of real paint samples and represent their spatial varying part by Bidirectional Texture Functions (BTF). We use classical BRDF models like Cook‐Torrance to represent the reflection behavior of the base paint and the highly specular finish and demonstrate how the parameters of these models can be derived from the BTF measurements. For rendering, the image‐based spatially varying part is compressed and efficiently synthesized. This paper introduces the first hybrid analytical and image‐based representation for car paint and enables the photo‐realistic rendering of all significant effects of highly complex coatings.

Abstract In this paper we present a novel computer vision based hand‐tracking technique, which is capable of robustly tracking 6+4DOF of the human hand in real‐time (at least 25 frames per second) with the help of 3 (or more) off‐the‐shelf consumer cameras. ‘6+4DOF’ means that the system can track the global pose (6 continuous parameters for translation and rotation) of 4 different gestures. A key feature of our system is its fully automatic real‐time initialization procedure, which, along with a sound tracking‐lost detector, makes the system fit for real‐world applications. Because of this, our method acts as an enabling technology for uncumbersome hand‐based 3D Human‐Computer‐Interaction (HCI). Previously, using the hand as an at least 6DOF input device involved the use of either datagloves or markers. Using our tracking we evaluated the use of the hand as an input device for two prevalent Virtual Reality applications: fly‐through exploration of a virtual world and a simple digital assembly simulation.

Abstract Image‐based representations of an object profit from known geometry. The more accurate this geometry is known, the better corresponding pixels in the different images can be aligned, which leads to less artifacts and better compression performance. For opaque objects the per‐pixel data can then be interpreted as a sampling of the BRDF at the respective surface point. In order to parameterize this sampled data a coordinate frame has to be defined. In previous work this coordinate frame was either the global frame or a local frame derived from the base geometry. Both approaches lead to misalignments between sample vectors: Features of basically very similar BRDFs will be shifted to different regions in the sample vector leading to poor compression performance. In order to improve alignment between the sampled BRDFs in image‐based rendering, we propose an optimization algorithm which determines consistent coordinate frames for every sample point on the object surface. This way we efficiently align the features even of anisotropic reflection functions and reconstruct approximate local coordinate frames without performing an explicit 3D‐reconstruction. The optimization is calculated efficiently by exploiting the Fourier‐shift theorem for spherical harmonics. In order to deal with different materials in a scene, the technique is combined with a clustering algorithm. We demonstrate the utility of our method by applying it to BTFs and 6D surface reflectance fields. Categories and Subject Descriptors (according to ACM CCS): I.3.3 [Picture/Image Generation]: Digitizing and scanning I.3.7 [Three‐Dimensional Graphics and Realism]: Color, shading, shadowing, and texture

Abstract One of the main challenges in computer graphics is still the realistic rendering of complex materials such as fabric or skin. The difficulty arises from the complex meso structure and reflectance behavior defining the unique look‐and‐feel of a material. A wide class of such realistic materials can be described as 2D‐texture under varying light‐ and view direction, namely, the Bidirectional Texture Function (BTF). Since an easy and general method for modeling BTFs is not available, current research concentrates on image‐based methods, which rely on measured BTFs (acquired real‐world data) in combination with appropriate synthesis methods. Recent results have shown that this approach greatly improves the visual quality of rendered surfaces and therefore the quality of applications such as virtual prototyping. This state‐of‐the‐art report (STAR) will present the techniques for the main tasks involved in producing photo‐realistic renderings using measured BTFs in details.

Creating long animation sequences with nontrivial repetitions is a time consuming and often difficult task. This is true for 2D images and even more true for 3D sequences. Based upon the idea of video textures we propose a simple algorithm to create new user controlled animation sequences based only on a few key frames by the analysis of velocity and position coherence. The simplicity of the method is achieved by carrying out the calculations on the main principal components of the reference animation, hence reducing the dimensionality of the input data. This also leads to significant compression. Smooth animations are ensured, using one of the proposed blending schemes.