We introduce SynthLight, a diffusion model for portrait relighting. We frame image relighting as a re-rendering problem, where pixels are transformed in response to changes in environmental lighting. Using a physically-based rendering engine, we create a dataset to simulate this lighting-conditioned transformation with 3D head assets under varying lighting. We propose training and inference strategies to bridge the gap between the synthetic and real image domains: (1) multi-task training leveraging real human portraits without lighting labels; (2) an inference time diffusion sampling scheme based on classifier-free guidance leveraging the input portrait to better preserve details. Our method generalizes to diverse real portraits and produces realistic illumination effects, including specular highlights and cast shadows, while preserving identity. Our quantitative experiments on light stage data demonstrate results comparable to state-of-the-art relighting methods. Our qualitative results on in-the-wild images showcase rich and unprecedented illumination effects.

Generating 3D scenes is still a challenging task due to the lack of readily available scene data. Most existing methods only produce partial scenes and provide limited navigational freedom. We introduce a practical and scalable solution that uses 360° video as an intermediate scene representation, capturing the full-scene context and ensuring consistent visual content throughout the generation. We propose WorldPrompter, a generative pipeline that synthesizes traversable 3D scenes from text prompts. WorldPrompter incorporates a conditional 360° panoramic video generator, capable of producing a 128-frame video that simulates a person walking through and capturing a virtual environment. The resulting video is then reconstructed as Gaussian splats by a fast feedforward 3D reconstructor, enabling a true walkable experience within the 3D scene. Experiments demonstrate that our panoramic video generation model, trained with a mix of image and video data, achieves convincing spatial and temporal consistency for static scenes. This is validated by an average COLMAP matching rate of 94.6%, allowing for high-quality panoramic Gaussian splat reconstruction and improved navigation throughout the scene. Qualitative and quantitative results also show it outperforms the state-of-the-art 360° video generators and 3D scene generation models.

The fashion industry has a real need to preview fabric designs using the actual threads they intend to use, ensuring that the designs they envisage can be physically realized. Unfortunately, today's fabric rendering relies on either hand-tuned parameters or parameters acquired from already fabricated cloth. Furthermore, existing curve-based scattering models are not suitable for this problem: they are either not naturally differentiable due to discrete fiber count parameters, or require a more detailed geometry representation, introducing extra complexity. In this work, we bridge this gap by presenting a novel pipeline that captures and digitizes physical threads and predicts the appearance of the fabric based on the weaving pattern. We develop a practical thread scattering model based on simulations of multiple fiber scattering within a thread. Using a cost-efficient multi-view setup, we capture threads of diverse colors and materials. We apply differentiable rendering to digitize threads, demonstrating that our model significantly improves the reconstruction accuracy compared to existing models, matching both reflection and transmission. We leverage a two-scale rendering technique to efficiently render woven cloth. We validate that our digital threads, combined with simulated woven yarn geometry, can accurately predict the fabric appearance by comparing to real samples. We show how our work can aid designs using diverse thread profiles, woven patterns, and textured design patterns.

We propose a method for depth estimation under different illumination conditions, i.e., day and night time. As photometry is uninformative in regions under low-illumination, we tackle the problem through a multi-sensor fusion approach, where we take as input an additional synchronized sparse point cloud (i.e., from a LiDAR) projected onto the image plane as a sparse depth map, along with a camera image. The crux of our method lies in the use of the abundantly available synthetic data to first approximate the 3D scene structure by learning a mapping from sparse to (coarse) dense depth maps along with their predictive uncertainty – we term this, SpaDe. In poorly illuminated regions where photometric intensities do not afford the inference of local shape, the coarse approximation of scene depth serves as a prior; the uncertainty map is then used with the image to guide refinement through an uncertainty-driven residual learning (URL) scheme. The resulting depth completion network leverages complementary strengths from both modalities – depth is sparse but insensitive to illumination and in metric scale, and image is dense but sensitive with scale ambiguity. SpaDe can be used in a plug-and-play fashion, which allows for 24% improvement when augmented onto existing methods to preprocess sparse depth. We demonstrate URL on the nuScenes dataset where we improve over all baselines by an average 12.39% in all-day scenarios, 12.02% when tested specifically for daytime, and 14.95% for nighttime scenes. Code available at : https://github.com/ezhovv/all-day-depth

We present PointShopAR, a novel tablet-based system for AR environmental design using point clouds as the underlying representation. It integrates point cloud capture and editing in a single AR workflow to help users quickly prototype design ideas in their spatial context. We hypothesize that point clouds are well suited for prototyping, as they can be captured more rapidly than textured meshes and then edited immediately in situ on the capturing device. We based the design of PointShopAR on the practical needs of six architects in a formative study. Our system supports a variety of point cloud editing operations in AR, including selection, transformation, hole filling, drawing, morphing, and animation. We evaluate PointShopAR through a remote study on usability and an in-person study on environmental design support. Participants were able to iterate design rapidly, showing the merits of an integrated capture and editing workflow with point clouds in AR environmental design.

Procedural modeling is now the de facto standard of material modeling in industry. Procedural models can be edited and are easily extended, unlike pixel-based representations of captured materials. In this article, we present a semi-automatic pipeline for general material proceduralization. Given Spatially Varying Bidirectional Reflectance Distribution Functions (SVBRDFs) represented as sets of pixel maps, our pipeline decomposes them into a tree of sub-materials whose spatial distributions are encoded by their associated mask maps. This semi-automatic decomposition of material maps progresses hierarchically, driven by our new spectrum-aware material matting and instance-based decomposition methods. Each decomposed sub-material is proceduralized by a novel multi-layer noise model to capture local variations at different scales. Spatial distributions of these sub-materials are modeled either by a by-example inverse synthesis method recovering Point Process Texture Basis Functions (PPTBF) [ 30 ] or via random sampling. To reconstruct procedural material maps, we propose a differentiable rendering-based optimization that recomposes all generated procedures together to maximize the similarity between our procedural models and the input material pixel maps. We evaluate our pipeline on a variety of synthetic and real materials. We demonstrate our method’s capacity to process a wide range of material types, eliminating the need for artist designed material graphs required in previous work [ 38 , 53 ]. As fully procedural models, our results expand to arbitrary resolution and enable high-level user control of appearance.

Relating small-scale structures to large-scale appearance is a key element in material appearance design. Bi-scale material design requires finding small-scale structures - meso-scale geometry and micro-scale BRDFs - that produce a desired large-scale appearance expressed as a macro-scale BRDF. The adjustment of small-scale geometry and reflectances to achieve a desired appearance can become a tedious trial-and-error process. We present a learning-based solution to fit a target macro-scale BRDF with a combination of a meso-scale geometry and micro-scale BRDF. We confront challenges in representation at both scales. At the large scale we need macro-scale BRDFs that are both compact and expressive. At the small scale we need diverse combinations of geometric patterns and potentially spatially varying micro-BRDFs. For large-scale macro-BRDFs, we propose a novel 2D subset of a tabular BRDF representation that well preserves important appearance features for learning. For small-scale details, we represent geometries and BRDFs in different categories with different physical parameters to define multiple independent continuous search spaces. To build the mapping between large-scale macro-BRDFs and small-scale details, we propose an end-to-end model that takes the subset BRDF as input and performs classification and parameter estimation on small-scale details to find an accurate reconstruction. Compared with other fitting methods, our learning-based solution provides higher reconstruction accuracy and covers a wider gamut of appearance.

Most augmented reality (AR) authoring tools only support the author’s current environment, but designers often need to create site-specific experiences for a different environment. We propose DistanciAR, a novel tablet-based workflow for remote AR authoring. Our baseline solution involves three steps. A remote environment is captured by a camera with LiDAR; then, the author creates an AR experience from a different location using AR interactions; finally, a remote viewer consumes the AR content on site. A formative study revealed understanding and navigating the remote space as key challenges with this solution. We improved the authoring interface by adding two novel modes: Dollhouse, which renders a bird’s-eye view, and Peek, which creates photorealistic composite images using captured images. A second study compared this improved system with the baseline, and participants reported that the new modes made it easier to understand and navigate the remote scene.

Non-photorealistic rendering (NPR) and image processing algorithms are widely assumed as a proxy for drawing. However, this assumption is not well assessed due to the difficulty in collecting and registering freehand drawings. Alternatively, tracings are easier to collect and register, but there is no quantitative evaluation of tracing as a proxy for freehand drawing. In this paper, we compare tracing, freehand drawing, and computer-generated drawing approximation (CGDA) to understand their similarities and differences. We collected a dataset of 1,498 tracings and freehand drawings by 110 participants for 100 image prompts. Our drawings are registered to the prompts and include vector-based timestamped strokes collected via stylus input. Comparing tracing and freehand drawing, we found a high degree of similarity in stroke placement and types of strokes used over time. We show that tracing can serve as a viable proxy for freehand drawing because of similar correlations between spatio-temporal stroke features and labeled stroke types. Comparing hand-drawn content and current CGDA output, we found that 60% of drawn pixels corresponded to computer-generated pixels on average. The overlap tended to be commonly drawn content, but people's artistic choices and temporal tendencies remained largely uncaptured. We present an initial analysis to inform new CGDA algorithms and drawing applications, and provide the dataset for use by the community.

Procedural textures are powerful tools that have been used in graphics for decades. In contrast to the alternative exemplar-based texture synthesis techniques, procedural textures provide user control and fast texture generation with low-storage cost and unlimited texture resolution. However, creating procedural models for complex textures requires a time-consuming process of selecting a combination of procedures and parameters. We present an example-based framework to automatically select procedural models and estimate parameters. In our framework, we consider textures categorized by commonly used high level classes. For each high level class we build a data-driven inverse modeling system based on an extensive collection of real-world textures and procedural texture models in the form of node graphs. We use unsupervised learning on collected real-world images in a texture class to learn sub-classes. We then classify the output of each of the collected procedural models into these sub-classes. For each of the collected models we train a convolutional neural network (CNN) to learn the parameters to produce a specific output texture. To use our framework, a user provides an exemplar texture image within a high level class. The system first classifies the texture into a sub-class, and selects the procedural models that produce output in that sub-class. The pre-trained CNNs of the selected models are used to estimate the parameters of the texture example. With the predicted parameters, the system can generate appropriate procedural textures for the user. The user can easily edit the textures by adjusting the node graph parameters. In a last optional step, style transfer augmentation can be applied to the fitted procedural textures to recover details lost in the procedural modeling process. We demonstrate our framework for four high level classes and show that our inverse modeling system can produce high-quality procedural textures for both structural and non-structural textures.

A 3D modeling system with all-inclusive functionality is too demanding for a casual 3D modeler to learn. There has been a shift towards more approachable systems, with easy-to-learn, intuitive interfaces. However, most modeling systems still employ mouse and keyboard interfaces, despite the ubiquity of tablet devices and the benefits of multi-touch interfaces. We introduce an alternative 3D modeling and fabrication paradigm using developable surfaces, inspired by traditional papercrafting, and we implement it as a complete system designed for a multi-touch tablet, allowing a user to fabricate 3D scenes. We demonstrate the modeling and fabrication process of assembling complex 3D scenes from a collection of simpler models, in turn shaped through operations applied to virtual paper. Our fabrication method facilitates the assembly of the scene with real paper by automatically converting scenes into a series of cutouts with appropriately added fiducial markers and supporting structures. Our system assists users in creating occluded supporting structures to help maintain the spatial and rigid properties of a scene without compromising its aesthetic qualities. We demonstrate several 3D scenes modeled and fabricated in our system, and evaluate the faithfulness of our fabrications relative to their virtual counterparts and 3D-printed fabrications.

Modeling virtual textiles has long been an appealing topic in computer graphics. To date, considerable effort has been devoted to their distinctive appearance and physically-based simulation. The apperance of staining patterns, commonly seen on textiles, has received comparatively little attention. This paper introduces techniques for simulating staining effects on fabric. Based on the microstructure of yarn, we propose a triple-layer model (TLM) to handle the liquid-yarn interaction for the wetting and wicking computation, and we formalize the liquid spreading in woven cloth into two typical actions, the in-yarn diffusion and the cross-yarn diffusion. The dye diffusion is driven by the liquid diffusion and the concentration distribution of pigments. The warp-weft anisotropy is handled by simulation of the yarn's structure in the two directions. Experimental results demonstrate that a wide range of fabric stain phenomenon on different textile materials, such as the water ring effect, the high saturate stain contour, and the dynamic wash away effect, can be simulated effectively without loss of visual realism. The realism of our simulation results is comparable to effects shown in photographs of real-world examples.

We introduce the problem of computing a human-perceived softness measure for virtual 3D objects. As the virtual objects do not exist in the real world, we do not directly consider their physical properties but instead compute the human-perceived softness of the geometric shapes. In an initial experiment, we find that humans are highly consistent in their responses when given a pair of vertices on a 3D model and asked to select the vertex that they perceive to be more soft. This motivates us to take a crowdsourcing and machine learning framework. We collect crowdsourced data for such pairs of vertices. We then combine a learning-to-rank approach and a multi-layer neural network to learn a non-linear softness measure mapping any vertex to a softness value. For a new 3D shape, we can use the learned measure to compute the relative softness of every vertex on its surface. We demonstrate the robustness of our framework with a variety of 3D shapes and compare our non-linear learning approach with a linear method from previous work. Finally, we demonstrate the accuracy of our learned measure with user studies comparing our measure with the human-perceived softness of both virtual and real objects, and we show the usefulness of our measure with some applications.

We introduce the problem of computing a human-perceived softness measure for virtual 3D objects. As the virtual objects do not exist in the real world, we do not directly consider their physical properties but instead compute the human-perceived softness of the geometric shapes. We collect crowdsourced data where humans rank their perception of the softness of vertex pairs on virtual 3D models. We then compute shape descriptors and use a learning-to-rank approach to learn a softness measure mapping any vertex to a softness value. Finally, we demonstrate our framework with a variety of 3D shapes.

Abstract In early or preparatory design stages, an architect or designer sketches out rough ideas, not only about the object or structure being considered, but its relation to its spatial context. This is an iterative process, where the sketches are not only the primary means for testing and refining ideas, but also for communicating among a design team and to clients. Hence, sketching is the preferred media for artists and designers during the early stages of design, albeit with a major drawback: sketches are 2D and effects such as view perturbations or object movement are not supported, thereby inhibiting the design process. We present an interactive system that allows for the creation of a 3D abstraction of a designed space, built primarily by sketching in 2D within the context of an anchoring design or photograph. The system is progressive in the sense that the interpretations are refined as the user continues sketching. As a key technical enabler, we reformulate the sketch interpretation process as a selection optimization from a set of context‐generated canvas planes in order to retrieve a regular arrangement of planes. We demonstrate our system (available at http:/geometry.cs.ucl.ac.uk/projects/2016/smartcanvas/ ) with a wide range of sketches and design studies.

While the concept of visual saliency has been previously explored in the areas of mesh and image processing, saliency detection also applies to other sensory stimuli. In this paper, we explore the problem of tactile mesh saliency, where we define salient points on a virtual mesh as those that a human is more likely to grasp, press, or touch if the mesh were a real-world object. We solve the problem of taking as input a 3D mesh and computing the relative tactile saliency of every mesh vertex. Since it is difficult to manually define a tactile saliency measure, we introduce a crowdsourcing and learning framework. It is typically easy for humans to provide relative rankings of saliency between vertices rather than absolute values. We thereby collect crowdsourced data of such relative rankings and take a learning-to-rank approach. We develop a new formulation to combine deep learning and learning-to-rank methods to compute a tactile saliency measure. We demonstrate our framework with a variety of 3D meshes and various applications including material suggestion for rendering and fabrication.

Texture synthesis is a well-established area, with many important applications in computer graphics and vision. However, despite their success, synthesis techniques are not used widely in practice because the creation of good exemplars remains challenging and extremely tedious. In this paper, we introduce an unsupervised method for analyzing texture content across multiple scales that automatically extracts good exemplars from natural images. Unlike existing methods, which require extensive manual tuning, our method is fully automatic. This allows the user to focus on using texture palettes derived from their own images, rather than on manual interactions dictated by the needs of an underlying algorithm. Most natural textures exhibit patterns at multiple scales that may vary according to the location (non-stationarity). To handle such textures many synthesis algorithms rely on an analysis of the input and a guidance of the synthesis. Our new analysis is based on a labeling of texture patterns that is both (i) multi-scale and (ii) unsupervised -- that is, patterns are labeled at multiple scales, and the scales and the number of labeled clusters are selected automatically. Our method works in two stages. The first builds a hierarchical extension of superpixels and the second labels the superpixels based on random walk in a graph of similarity between superpixels and a nonnegative matrix factorization. Our label-maps provide descriptors for pixels and regions that benefit state-of-the-art texture synthesis algorithms. We show several applications including guidance of non-stationary synthesis, content selection and texture painting. Our method is designed to treat large inputs and can scale to many megapixels. In addition to traditional exemplar inputs, our method can also handle natural images containing different textured regions.

This paper examines the following question: given a collection of man-made shapes, e.g., chairs, can we effectively explore and rank the shapes with respect to a given human body-in terms of how well a candidate shape fits the specified human body? Answering this question requires identifying which shapes are more suitable for a prescribed body, and how to alter the input geometry to better fit the shapes to a given human body. The problem links physical proportions of the human body and its interaction with object geometry, which is often expressed as ergonomics guidelines. We present an interactive system that allows users to explore shapes using different avatar poses, while, at the same time providing interactive previews of how to alter the shapes to fit the user-specified body and pose. We achieve this by first constructing a fuzzy shape-to-body map from the ergonomic guidelines to multi-contacts geometric constraints; and then, proposing a novel contact-preserving deformation paradigm to realize a reshaping to adapt the input shape. We evaluate our method on collections of models from different categories and validate the results through a user study.

A 3D modeling system that provides all-inclusive functionality is generally too demanding for a casual 3D modeler to learn. In recent years, there has been a shift towards developing more approachable systems, with easy-to-learn, intuitive interfaces. However, most modeling systems still employ mouse and keyboard interfaces, despite the ubiquity of tablet devices, and the benefits of multi-touch interfaces applied to 3D modeling. In this paper, we introduce an alternative 3D modeling paradigm for creating developable surfaces, inspired by traditional papercrafting, and implemented as a system designed from the start for a multi-touch tablet. We demonstrate the process of assembling complex 3D scenes from a collection of simpler models, in turn shaped through operations applied to sheets of virtual paper. The modeling and assembling operations mimic familiar, real-world operations performed on paper, allowing users to quickly learn our system with very little guidance. We outline key design decisions made throughout the development process, based on feedback obtained through collaboration with target users. Finally, we include a range of models created in our system.

The BTF data structure was a breakthrough for appearance modeling in computer graphics. More research is needed though to make BTFs practical in rendering applications. We present the first systematic study of the effects of Approximate filtering on the appearance of BTFs, by exploring the spatial, angular and temporal domains over a varied set of stimuli. We perform our initial experiments on simple geometry and lighting, and verify our observations on more complex settings. We consider multi-dimensional filtering versus conventional mipmapping, and find that multi-dimensional filtering produces superior results. We examine the tradeoff between under- and oversampling, and find that different filtering strategies can be applied in each domain, while maintaining visual equivalence with respect to a ground truth. For example, we find that preserving contrast is more important in static than dynamic images, indicating greater levels of spatial filtering are possible for animations. We find that filtering can be performed more aggressively in the angular domain than in the spatial. Additionally, we find that high-level visual descriptors of the BTF are linked to the perceptual performance of pre-filtered approximations. In turn, some of these high-level descriptors correlate with low level statistics of the BTF. We show six different practical applications of applying our findings to improving filtering, rendering and compression strategies.

Texture is an essential feature in modeling the appearance of objects and is instrumental in making virtual objects appear interesting and/or realistic. Unfortunately, obtaining textures is a labor intensive task requiring parameter tuning for procedural methods or careful photography and post-processing for natural images. Many texture synthesis techniques have been developed to generate textures of arbitrary spatial extent, but these techniques require the user to first produce an exemplar consisting solely of the desired texture. We present a fast method using diffusion manifolds to locate textures in unconstrained photographs, and extract exemplar tiles. The method requires the user to only specify a single point within the image containing the desired texture and the scale of the desired texture. The user may tune the result using simple interactions. The method is non-local, in the sense that the desired texture does not have to appear in a single contiguous region in the source image. This document is a summary of our work and system. A full account is available online.

Abstract Directors employ a process called “color grading” to add color styles to feature films. Color grading is used for a number of reasons, such as accentuating a certain emotion or expressing the signature look of a director. We collect a database of feature film clips and label them with tags such as director, emotion, and genre. We then learn a model that maps from the low‐level color and tone properties of film clips to the associated labels. This model allows us to examine a number of common hypotheses on the use of color to achieve goals, such as specific emotions. We also describe a method to apply our learned color styles to new images and videos. Along with our analysis of color grading techniques, we demonstrate a number of images and videos that are automatically filtered to resemble certain film styles.

One major shortcoming of existing bi-scale material design systems is the lack of support for inverse design: there is no way to directly edit the large-scale appearance and then rapidly solve for the small-scale details that approximate that look. Prior work is either too slow to provide quick feedback, or limited in the types of small-scale details that can be handled. We present a novel computational framework for inverse bi-scale material design. The key idea is to convert the challenging inverse appearance computation into efficient search in two precomputed large libraries: one including a wide range of measured and analytical materials, and the other procedurally generated and height-map-based geometries. We demonstrate a variety of editing operations, including finding visually equivalent details that produce similar large-scale appearance, which can be useful in applications such as physical fabrication of materials.

Sophisticated methods for true spectral rendering have been developed in computer graphics to produce highly accurate images. In addition to traditional applications in visualizing appearance, such methods have potential applications in many areas of scientific study. In particular, we are motivated by the application of studying avian vision and appearance. An obstacle to using graphics in this application is the lack of reliable input data. We introduce an end-to-end measurement system for capturing spectral data on 3D objects. We present the modification of a recently developed hyperspectral imager to make it suitable for acquiring such data in a wide spectral range at high spectral and spatial resolution. We capture four megapixel images, with data at each pixel from the near-ultraviolet (359 nm) to near-infrared (1,003 nm) at 12 nm spectral resolution. We fully characterize the imaging system, and document its accuracy. This imager is integrated into a 3D scanning system to enable the measurement of the diffuse spectral reflectance and fluorescence of specimens. We demonstrate the use of this measurement system in the study of the interplay between the visual capabilities and appearance of birds. We show further the use of the system in gaining insight into artifacts from geology and cultural heritage.

Compositing is one of the most commonly performed operations in computer graphics. A realistic composite requires adjusting the appearance of the foreground and background so that they appear compatible; unfortunately, this task is challenging and poorly understood. We use statistical and visual perception experiments to study the realism of image composites. First, we evaluate a number of standard 2D image statistical measures, and identify those that are most significant in determining the realism of a composite. Then, we perform a human subjects experiment to determine how the changes in these key statistics influence human judgements of composite realism. Finally, we describe a data-driven algorithm that automatically adjusts these statistical measures in a foreground to make it more compatible with its background in a composite. We show a number of compositing results, and evaluate the performance of both our algorithm and previous work with a human subjects study.

Abstract Bidirectional texture functions (BTFs) represent the appearance of complex materials. Three major shortcomings with BTFs are the bulky storage, the difficulty in editing and the lack of efficient rendering methods. To reduce storage, many compression techniques have been applied to BTFs, but the results are difficult to edit. To facilitate editing, analytical models have been fit, but at the cost of accuracy of representation for many materials. It becomes even more challenging if efficient rendering is also needed. We introduce a high‐quality general representation that is, at once, compact, easily editable, and can be efficiently rendered. The representation is computed by adopting the stagewise Lasso algorithm to search for a sparse set of analytical functions, whose weighted sum approximates the input appearance data. We achieve compression rates comparable to a state‐of‐the‐art BTF compression method. We also demonstrate results in BTF editing and rendering.

Abstract The appearance of weathering effects on stone is important for creating outdoor scenes in computer graphics. To achieve them, previous research has built upon physical simulation, which, while yielding a degree of realism, is computationally expensive and inapplicable to the situation when the object geometry is unknown. Also, physical simulation requires specific knowledge of the stone properties and environmental processes. In this paper, we present a simple visual simulation pipeline for creating weathering effects on stone within a single image. Two primary effects of stone weathering, i.e., smoothing and roughening, are considered. In addition, erosion on the object silhouette is treated. These challenging effects involve significant geometry changes, which are intractable for previous image‐based editing techniques. The effectiveness of our technique is illustrated on a variety of scenes and types of stone. While it can be fully automatic, it also allows easy user interaction.

The simulation of weathered appearance is essential in the realistic modeling of urban environments. A representative and particularly difficult effect to produce on a large scale is the effect of fluid flow. Changes in appearance due to flow are the result of both the global effect of large-scale shape, and local effects, such as the detailed roughness of a surface. With digital photography and Internet image collections, visual examples of flow effects are readily available. These images, however, mix the appearance of flows with the specific local context. We present a methodology to extract parameters and detail maps from existing imagery in a form that allows new target-specific flow effects to be produced, with natural variations in the effects as they are applied in different locations in a new scene. In this article, we focus on producing a library of parameters and detail maps for generating flow patterns; and this methodology can be used to extend the library with additional image exemplars. To illustrate our methodology, we show a rich collection of patterns applied to urban models.

Architecture is design in spatial context. The only current methods for representing context involve designing in a heavyweight computer-aided design system, using a full model of existing buildings and landscape, or sketching on a panoramic photo. The former is too cumbersome; the latter is too restrictive in viewpoint and in the handling of occlusions and topography. We introduce a novel approach to presenting context such that it is an integral component in a lightweight conceptual design system. We represent sites through a fusion of data available from different sources. We derive a site model from geographic elevation data, on-site point-to-point distance measurements, and images of the site. To acquire and process the data, we use publicly available data sources, multidimensional scaling techniques and refinements of recent bundle adjustment techniques. We offer a suite of interactive tools to acquire, process, and combine the data into a lightweight stroke and image-billboard representation. We create multiple and linked pop-ups derived from images, forming a lightweight representation of a three-dimensional environment. We implemented our techniques in a stroke-based conceptual design system we call Insitu . We developed our work through continuous interaction with professional designers. We present designs created with our new techniques integrated in a conceptual design system.

We present the first physically-based interactive system to facilitate the appearance design at different scales consistently, through manipulations of both small-scale geometry and materials. The core of our system is a novel reflectance filtering algorithm, which rapidly computes the large-scale appearance from small-scale details, by exploiting the low-rank structures of the Bidirectional Visible Normal Distribution Function and pre-rotated BRDFs in the matrix formulation of our rendering problem. Our algorithm is three orders of magnitude faster than a ground-truth method. We demonstrate various editing results of different small-scale geometry with analytical and measured BRDFs. In addition, we show the applications of our system to physical realization of appearance, as well as modeling of real-world materials using very sparse measurements.

Abstract Papercraft models can serve as inexpensive prototypes in shape design applications. However, in making the models some geometric detail is necessarily lost, and artificial creases may be visible, thereby limiting the utility of these models. To compensate for these practical limitations, we introduce the use of printed patterns on papercraft models to enhance the perception of the shape they are intended to represent. We propose pattern generation schemes that modulate the sizes, directions, and densities of glyphs of patterns based on geometric attributes. We present a psychophysical experiment designed to explore the effect that printed patterns have on the perception of the papercraft model shapes. We find that models with printed patterns are perceived to represent the intended shape more accurately, and, further, that the type of printed pattern has an impact on the perceived shape.

Abstract Texture synthesis techniques require nearly uniform texture samples, however identifying suitable texture samples in an image requires significant data preprocessing. To eliminate this work, we introduce a fully automatic pipeline to detect dominant texture samples based on a manifold generated using the diffusion distance. We define the characteristics of dominant texture and three different types of outliers that allow us to efficiently identify dominant texture in feature space. We demonstrate how this method enables the analysis/synthesis of a wide range of natural textures. We compare textures synthesized from a sample image, with and without dominant texture detection. We also compare our approach to that of using a texture segmentation technique alone, and to using Euclidean, rather than diffusion, distances between texture features.

Abstract Extremely dense spatial sampling is often needed to prevent aliasing when rendering objects with high frequency variations in geometry and reflectance. To accelerate the rendering process, we introduce characteristic point maps (CPMs), a hierarchy of view‐independent points, which are chosen to preserve the appearance of the original model across different scales. In preprocessing, randomized matrix column sampling is used to reduce an initial dense sampling to a minimum number of characteristic points with associated weights. In rendering, the reflected radiance is computed using a weighted average of reflectances from characteristic points. Unlike existing techniques, our approach requires no restrictions on the original geometry or reflectance functions.

In modern computer graphics applications, textures play an important role in conveying the appearance of real-world materials. But while surface appearance can often be effectively captured with a photograph, it is difficult to use example imagery to synthesize fully three-dimensional (3D) solid textures that are perceptually similar to their inputs. Specifically, this research focuses on human perception of 3D solid textures composed of aggregate particles in a binding matrix. Holding constant an established algorithm for approximating particledistributions, we examine the problem of estimating particleshape. We consider four methods for approximating plausible particle shapes—including two methods of our own contribution. We compare the performance of these methods under a variety of input conditions using automated, perceptually motivated metrics, as well as a psychophysical experiment. In the course of assessing the relative performance of the four algorithms, we also evaluate the reliability of the automated metrics in predicting the results of the experiment.

In this article, we introduce sketching reality , the process of converting a freehand sketch into a realistic-looking model. We apply this concept to architectural designs. As the sketch is being drawn, our system periodically interprets its 2.5D-geometry by identifying new junctions, edges, and faces, and then analyzing the extracted topology. The user can add detailed geometry and textures through sketches as well. This is possible through the use of databases that match partial sketches to models of detailed geometry and textures. The final product is a realistic texture-mapped 2.5D-model of the building. We show a variety of buildings that have been created using this system.

We describe a computer graphics system that supports conceptual architectural design and analysis. We use as a starting point the traditional sketchbook drawings that architects use to experiment with various views, sections, and details. Rather than interpret or infer 3D structure from drawings, our system is designed to allow the designer to organize concept drawings in 3D, and gradually fuse a series of possibly geometrically-inconsistent sketches into a set of 3D strokes. Our system uses strokes and planar "canvases" as basic primitives; the basic mode of input is traditional 2D drawing. We introduce methods for the user to control stroke visibility and transfer strokes between canvases. We also introduce methods for the user to position and orient the canvases that have infinite extent. We demonstrate the use of the system to analyze existing structures and conceive new designs.

Interesting textures form on the surfaces of objects as the result of external chemical, mechanical, and biological agents. Simulating these textures is necessary to generate models for realistic image synthesis. The textures formed are progressively variant, with the variations depending on the global and local geometric context. We present a method for capturing progressively varying textures and the relevant context parameters that control them. By relating textures and context parameters, we are able to transfer the textures to novel synthetic objects. We present examples of capturing chemical effects, such as rusting; mechanical effects, such as paint cracking; and biological effects, such as the growth of mold on a surface. We demonstrate a user interface that provides a method for specifying where an object is exposed to external agents. We show the results of complex, geometry-dependent textures evolving on synthetic objects.

This paper presents a framework for the real-time rendering of plant leaves with global illumination effects. Realistic rendering of leaves requires a sophisticated appearance model and accurate lighting computation. For leaf appearance we introduce a parametric model that describes leaves in terms of spatially-variant BRDFs and BTDFs. These BRDFs and BTDFs, incorporating analysis of subsurface scattering inside leaf tissues and rough surface scattering on leaf surfaces, can be measured from real leaves. More importantly, this description is compact and can be loaded into graphics hardware for fast run-time shading calculations, which are essential for achieving high frame rates. For lighting computation, we present an algorithm that extends the Precomputed Radiance Transfer (PRT) approach to all-frequency lighting for leaves. In particular, we handle the combined illumination effects due to low-frequency environment light and high-frequency sunlight. This is done by decomposing the local incident radiance of sunlight into direct and indirect components. The direct component, which contains most of the high frequencies, is not pre-computed with spherical harmonics as in PRT; instead it is evaluated on-the-fly using pre-computed light-visibility convolution data. We demonstrate our framework by the rendering of a variety of leaves and assemblies thereof.

We describe the use of traditional stereological methods to synthesize 3D solid textures from 2D images of existing materials. We first illustrate our approach for aggregate materials of spherical particles, and then extend the technique to apply to particles of arbitrary shapes. We demonstrate the effectiveness of the approach with side-by-side comparisons of a real material and a synthetic model with its appearance parameters derived from its physical counterpart. Unlike ad hoc methods for texture synthesis, stereology provides a disciplined, systematic basis for predicting material structure with well-defined assumptions.

We introduce billboard clouds -- a new approach for extreme simplification in the context of real-time rendering. 3D models are simplified onto a set of planes with texture and transparency maps. We present an optimization approach to build a billboard cloud given a geometric error threshold. After computing an appropriate density function in plane space, a greedy approach is used to select suitable representative planes. A good surface approximation is ensured by favoring planes that are "nearly tangent" to the model. This method does not require connectivity information, but instead avoids cracks by projecting primitives onto multiple planes when needed. For extreme simplification, our approach combines the strengths of mesh decimation and image-based impostors. We demonstrate our technique on a large class of models, including smooth manifolds and composite objects.

We present a procedural approach to authoring layered, solid models. Using a simple scripting language, we define the internal structure of a volume from one or more input meshes. Sculpting and simulation operators are applied within the context of the language to shape and modify the model. Our framework treats simulation as a modeling operator rather than simply as a tool for animation, thereby suggesting a new paradigm for modeling as well as a new level of abstraction for interacting with simulation environments.Capturing real-world effects with standard modeling techniques is extremely challenging. Our key contribution is a concise procedural approach for seamlessly building and modifying complex solid geometry. We present an implementation of our language using a flexible tetrahedral representation. We show a variety of complex objects modeled in our system using tools that interface with finite element method and particle system simulations.

We present a new technique for the display of high-dynamic-range images, which reduces the contrast while preserving detail. It is based on a two-scale decomposition of the image into a base layer, encoding large-scale variations, and a detail layer. Only the base layer has its contrast reduced, thereby preserving detail. The base layer is obtained using an edge-preserving filter called the bilateral filter. This is a non-linear filter, where the weight of each pixel is computed using a Gaussian in the spatial domain multiplied by an influence function in the intensity domain that decreases the weight of pixels with large intensity differences. We express bilateral filtering in the framework of robust statistics and show how it relates to anisotropic diffusion. We then accelerate bilateral filtering by using a piecewise-linear approximation in the intensity domain and appropriate subsampling. This results in a speed-up of two orders of magnitude. The method is fast and requires no parameter setting.

We present a novel drawing system for composing and rendering perspective scenes. Our approach uses a projective 2D representation for primitives rather than a conventional 3D description. This allows drawings to be composed with the same ease as traditional illustrations, while providing many of the advantages of a 3D model. We describe a range of user-interface tools and interaction techniques that give our system its 3D-like capabilities. We provide vanishing point guides and perspective grids to aid in drawing freehand strokes and composing perspective scenes. Our system also has tools for intuitive navigation of a virtual camera, as well as methods for manipulating drawn primitives so that they appear to undergo 3D translations and rotations. We also support automatic shading of primitives using either realistic or non-photorealistic styles. Our system supports drawing and shading of extrusion surfaces with automatic hidden surface removal and highlighted silhouettes. Casting shadows from an infinite light source is also possible with minimal user intervention. CR Categories: I.3.3 [Computer Graphics]: Graphics Utilities--- Graphics Editors; I.3.6 [Computer Graphics]: Methodologies and Techniques---Interaction Techniques Keywords: Image-based Modeling and Rendering, Misc. 2D graphics, Non-Euclidean Spaces, Non-Photorealistic Rendering 1

This paper presents a physically-based model of the night sky for realistic image synthesis. We model both the direct appearance of the night sky and the illumination coming from the Moon, the stars, the zodiacal light, and the atmosphere. To accurately predict the appearance of night scenes we use physically-based astronomical data, both for position and radiometry. The Moon is simulated as a geometric model illuminated by the Sun, using recently measured elevation and albedo maps, as well as a specialized BRDF. For visible stars, we include the position, magnitude, and temperature of the star, while for the Milky Way and other nebulae we use a processed photograph. Zodiacal light due to scattering in the dust covering the solar system, galactic light, and airglow due to light emission of the atmosphere are simulated from measured data. We couple these components with an accurate simulation of the atmosphere. To demonstrate our model, we show a variety of night scenes rendered with a Monte Carlo ray tracer.

Cellular patterns are all around us, in masonry, tiling, shingles, and many other materials. Such patterns, especially in architectural settings, are influenced by geometric features of the underlying shape. Bricks turn corners, stones frame windows and doorways, and patterns on disconnected portions of a building align to achieve a particular aesthetic goal. We present a strategy for feature-based cellular texturing, where the resulting texture is derived from both patterns of cells and the geometry to which they are applied. As part of this strategy, we perform texturing operations on features in a well-defined order that simplifies the interdependence between cells of adjacent patterns. Occupancy maps are used to indicate which regions of a feature are already occupied by cells of its neighbors, and which regions remain to be textured. We also introduce the notion of a pattern generator — the cellular texturing analogy of a shader used in local illumination — and show how several can be used together to build complex textures. We present results obtained with an implementation of this strategy and discuss details of some example pattern generators.

We present an image-based modeling and editing system that takes a single photo as input. We represent a scene as a layered collection of depth images, where each pixel encodes both color and depth. Starting from an input image, we employ a suite of user-assisted techniques, based on a painting metaphor, to assign depths and extract layers. We introduce two specific editing operations. The first, a “clone brushing tool,” permits the distortion-free copying of parts of a picture, by using a parameterization optimization technique. The second, a “texture-illuminance decoupling filter,” discounts the effect of illumination on uniformly textured areas, by decoupling large- and small-scale features via bilateral filtering. Our system enables editing from different viewpoints, extracting and grouping of image-based objects, and modifying the shape, color, and illumination of these objects.

Visibility determination is a key requirement in a wide range of graphics algorithms. This paper introduces a new approach to the computation of volume visibility, the detection of occluded portions of space as seen from a given region. The method is conservative and classifies regions as occluded only when they are guaranteed to be invisible. It operates on a discrete representation of space and uses the opaque interior of objects as occluders. This choice of occluders facilitates their extension into adjacent opaque regions of space, in essence maximizing their size and impact. Our method efficiently detects and represents the regions of space hidden by such occluders. It is the first one to use the property that occluders can also be extended into empty space provided this space is itself occluded from the viewing volume. This proves extremely effective for computing the occlusion by a set of occluders, effectively realizing occluder fusion. An auxiliary data structure represents occlusion in the scene and can then be queried to answer volume visibility questions. We demonstrate the applicability to visibility preprocessing for real-time walkthroughs and to shadow-ray acceleration for extended light sources in ray tracing, with significant acceleration in both cases.

Article Free Access Share on Modeling and rendering of weathered stone Authors: Julie Dorsey Laboratory for Computer Science, Massachusetts Institute of Technology Laboratory for Computer Science, Massachusetts Institute of TechnologyView Profile , Alan Edelman Laboratory for Computer Science, Massachusetts Institute of Technology Laboratory for Computer Science, Massachusetts Institute of TechnologyView Profile , Henrik Wann Jensen Laboratory for Computer Science, Massachusetts Institute of Technology Laboratory for Computer Science, Massachusetts Institute of TechnologyView Profile , Justin Legakis Laboratory for Computer Science, Massachusetts Institute of Technology Laboratory for Computer Science, Massachusetts Institute of TechnologyView Profile , Hans Køhling Pedersen Laboratory for Computer Science, Massachusetts Institute of Technology Laboratory for Computer Science, Massachusetts Institute of TechnologyView Profile Authors Info & Claims SIGGRAPH '99: Proceedings of the 26th annual conference on Computer graphics and interactive techniquesJuly 1999 Pages 225–234https://doi.org/10.1145/311535.311560Online:01 July 1999Publication History 176citation1,637DownloadsMetricsTotal Citations176Total Downloads1,637Last 12 Months6Last 6 weeks1 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteeReaderPDF

Freehand sketching has long had appeal as an artistic medium for conceptual design because of its immediacy in capturing and communicating design intent and visual experience. We present a sketching paradigm that supports the early stages of design by preserving the fluidity of traditional freehand drawings. In addition, it attempts to fill the gap between 2D drawing programs, which have fixed views, and 3D modeling programs that allow arbitrary views. We implement our application as a two-dimensional drawing program that utilizes a projective representation of points — i.e. points that lie on the surface of a unit sphere centered at the viewpoint. This representation facilitates the production of novel re-projections generated from an initial perspective sketch and gives the user the impression of being immersed in the drawing or space. We describe a method for aligning a sketch drawn outside the system using its vanishing points, allowing the integration of computer sketching and freehand sketching on paper in an iterative manner. The user interface provides a virtual camera, projective grids to guide in the construction of proportionate scenes, and the ability to underlay sketches with other drawings or photographic panoramas.

This paper describes the successful combination of pre‐generated and dynamically updated image‐based representations to accelerate the visualization of complex virtual environments. We introduce a new type of impostor, which has the desirable property of limiting de‐occlusion errors to a user‐specified amount. This impostor, composed of multiple layers of textured meshes, replaces the distant geometry and is much faster to draw. It captures the relevant depth complexity in the model without resorting to a complete sampling of the scene. We show that layers can be dynamically updated during visualization. This guarantees bounded scene complexity in each frame and also exploits temporal coherence to improve image quality when possible. We demonstrate the strengths of this approach in the context of city walkthroughs.

Ray tracers, which sample radiance, are usually regarded as offline rendering algorithms that are too slow for interactive use. In this article we present a system that exploits object-space, ray-space, image-space, and temporal coherence to accelerate ray tracing. Our system uses per-surface interpolants to approximate radiance both interactive and batch ray tracers. Our approach explicity decouples the two primary operations of a ray tracer—shading and visibility determination—and accelerates each of them independently. Shading is accelerated by quadrilinearily interpolating lazily acquired radiance samples. Interpolation error does not exceed a user-specified bound, allowing the user to control performance/quality tradeoffs. Error is bounded by adaptive sampling at discontinuities and radiance nonlinearities. Visibility determination at pixels is accelerated by reprojecting interpolants as the user's viewpoint changes. A fast scan-line alogoithm then achieves high performance without sacrificing image quality. For a smoothly varying viewpoint, the combination of lazy interpolants and projection substantially accelerates the ray tracer. Additionally, an efficient cache management algorithm keeps the memory footprint of the system small with negilgible overhead.

Article Modeling and rendering of metallic patinas Share on Authors: Julie Dorsey Massachusetts Institute of Technology, Room NE43-213, 545 Technology Square, Cambridge, MA Massachusetts Institute of Technology, Room NE43-213, 545 Technology Square, Cambridge, MAView Profile , Pat Hanrahan Stanford University, 370 Gates Computer Science Building 3B, Stanford, CA Stanford University, 370 Gates Computer Science Building 3B, Stanford, CAView Profile Authors Info & Claims SIGGRAPH '96: Proceedings of the 23rd annual conference on Computer graphics and interactive techniquesAugust 1996 Pages 387–396https://doi.org/10.1145/237170.237278Online:01 August 1996Publication History 111citation1,141DownloadsMetricsTotal Citations111Total Downloads1,141Last 12 Months6Last 6 weeks1 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access

An important, largely unexplored area of computer image generation is the simulation of weathering and its effects on appearance.Weathering results from the interaction of the environment with the materials in the world.The flow of water is one of the most pervasive and important natural forces involved in the weathering of materials, producing a distinctive set of patterns of washes and stains.This paper presents an intuitive phenomenological model for the flow of water over surfaces that is capable of generating such changes in appearance.We model the flow as a particle system, each particle representing a "drop" of water.The motion of the water particles is controlled by parameters such as gravity, friction, wind, roughness, and constraints that force the particles to maintain contact with the surface.The chemical interaction of the water with the surface materials is governed by a set of coupled differential equations describing both the rate of absorption of water by the surface and the rate of solubility and sedimentation of deposits on the surface.To illustrate the power of this simple model, we show examples of flows over complex geometries made from different materials; the resulting patterns are striking and very difficult to achieve using traditional texturing techniques.

We describe a new framework for efficiently computing and storing global illumination effects for complex, animated environments. The new framework allows the rapid generation of sequences representing any arbitrary path in a "view space" within an environment in which both the viewer and objects move. The global illumination is stored as time sequences of range-images at base locations that span the view space. We present algorithms for determining locations for these base images, and the time steps required to adequately capture the effects of object motion. We also present algorithms for computing the global illumination in the base images that exploit spatial and temporal coherence by considering direct and indirect illumination separately. We discuss an initial implementation using the new framework. Results and analysis of our implementation demonstrate the effectiveness of the individual phases of the approach; we conclude with an application of the complete framework to a complex environment that includes object motion.

Article Painting with light Share on Authors: Chris Schoeneman View Profile , Julie Dorsey View Profile , Brian Smits View Profile , James Arvo View Profile , Donald Greenberg View Profile Authors Info & Claims SIGGRAPH '93: Proceedings of the 20th annual conference on Computer graphics and interactive techniquesSeptember 1993 Pages 143–146https://doi.org/10.1145/166117.166135Online:01 September 1993Publication History 98citation1,320DownloadsMetricsTotal Citations98Total Downloads1,320Last 12 Months47Last 6 weeks2 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access

A major problem challenging opera designers is the inability to co-ordinate lighting, projection systems, and set designs in the preliminary planning phase. New computer graphics techniques, which provide the set and lighting designer the opportunity to evaluate, test, and control opera designs prior to the construction of full scale systems are presented. These techniques---light source input, simulation of directional lighting, modeling of scenic projection systems, and full three-dimensional simulation---show the potential for the use of computer graphics in theater design.The light source input component consists of a program for assigning light source attributes with a set of theater lighting icons. This module allows a designer to specify light source characteristics in a way familiar to the discipline and to make preliminary evaluations of the lighting conditions.An extended progressive radiosity method is introduced to simulate the directional lighting characteristics which are specified by the input program.A new projection approach is presented to simulate the optical effects of scenic projectors. In addition, a solution to the distortion problem produced by angular projections is described.The above components are integrated to produce full three-dimensional simulations of the global illumination effects in an opera scene.