About 25% of the world’s population live in informal urban settlements containing densely packed buildings (approximately 8,000 houses per square-km) which do not lend themselves favorably to state-of-the-art satellite-based building segmentation methods due to, for example, occlusion, vegetation, shadows and low resolution. To address these challenges, we introduce a novel instance segmentation and counting approach for dense buildings. Our system first extracts a conservative set of tentative building center points using a deep network for jumpstarting a Segment Anything Model 2 (SAM2) module to produce an initial over-segmentation. Second, we use a graph neural network to refine the over-segmented regions into polygons representing accurate building masks. Experiments show that our approach achieves higher accuracy in instance segmentation and counting especially in challenging densely packed building areas in Brazil, Mexico, India, Pakistan, and Kenya, for instance.

Multimodal Large Language Models (MLLMs) struggle with accurately capturing camera-object relations, especially for object orientation, camera viewpoint, and camera shots. This stems from the fact that existing MLLMs are trained on images with limited diverse camera-object relations and corresponding textual descriptions. To address this, we propose a synthetic generation pipeline to create large-scale 3D visual instruction datasets. Our framework takes 3D assets as input and uses rendering and diffusion-based image generation models to create photorealistic images preserving precise camera-object relations. Additionally, large language models (LLMs) are used to generate text prompts for guiding visual instruction tuning and controlling image generation. We create Ultimate3D, a dataset of 240K VQAs with precise camera-object annotations, and corresponding benchmark. MLLMs fine-tuned on our proposed dataset outperform commercial models by a large margin, achieving an average accuracy improvement of 33.4% on camera-object relation recognition tasks. Our code, dataset, and benchmark will contribute to broad MLLM applications.

Point cloud completion helps restore partial incomplete point clouds suffering occlusions. Current self-supervised methods fail to give high fidelity completion for large objects with missing surfaces and unbalanced distribution of available points. In this paper, we present a novel method for restoring large-scale point clouds with limited and unbalanced ground-truth. Using rough boundary annotations for a region of interest, we project the original point clouds into a multiple-center-of-projection (MCOP) image, where fragments are projected to images of 5 channels (RGB, depth, and rotation). Completion of the original point cloud is reduced to inpainting the missing pixels in the MCOP images. Due to lack of complete structures and an unbalanced distribution of existing parts, we develop a self-supervised scheme which learns to infill the MCOP image with points resembling existing "complete" patches. Special losses are applied to further enhance the regularity and consistency of completed MCOP images, which is mapped back to 3D to form final restoration. Extensive experiments demonstrate the superiority of our method in completing 600+ incomplete and unbalanced archaeological structures in Peru.

Personalized image generation has emerged from the recent advancements in generative models. However, these generated personalized images often suffer from localized artifacts such as incorrect logos, reducing fidelity and fine-grained identity details of the generated results. Furthermore, there is little prior work tackling this problem. To help improve these identity details in the personalized image generation, we introduce a new task: reference-guided artifacts refinement. We present Refine-by-Align, a first-of-its-kind model that employs a diffusion-based framework to address this challenge. Our model consists of two stages: Alignment Stage and Refinement Stage, which share weights of a unified neural network model. Given a generated image, a masked artifact region, and a reference image, the alignment stage identifies and extracts the corresponding regional features in the reference, which are then used by the refinement stage to fix the artifacts. Our model-agnostic pipeline requires no test-time tuning or optimization. It automatically enhances image fidelity and reference identity in the generated image, generalizing well to existing models on various tasks including but not limited to customization, generative compositing, view synthesis, and virtual try-on. Extensive experiments and comparisons demonstrate that our pipeline greatly pushes the boundary of fine details in the image synthesis models.

Recent multimodal image generators such as GPT-4o, Gemini 2.0 Flash, and Gemini 2.5 Pro excel at following complex instructions, editing images and maintaining concept consistency. However, they are still evaluated by disjoint toolkits: text-to-image (T2I) benchmarks that lacks multi-modal conditioning, and customized image generation benchmarks that overlook compositional semantics and common knowledge. We propose MMIG-Bench, a comprehensive Multi-Modal Image Generation Benchmark that unifies these tasks by pairing 4,850 richly annotated text prompts with 1,750 multi-view reference images across 380 subjects, spanning humans, animals, objects, and artistic styles. MMIG-Bench is equipped with a three-level evaluation framework: (1) low-level metrics for visual artifacts and identity preservation of objects; (2) novel Aspect Matching Score (AMS): a VQA-based mid-level metric that delivers fine-grained prompt-image alignment and shows strong correlation with human judgments; and (3) high-level metrics for aesthetics and human preference. Using MMIG-Bench, we benchmark 17 state-of-the-art models, including Gemini 2.5 Pro, FLUX, DreamBooth, and IP-Adapter, and validate our metrics with 32k human ratings, yielding in-depth insights into architecture and data design.

Generative object compositing emerges as a promising new avenue for compositional image editing. However, the requirement of object identity preservation poses a significant challenge, limiting practical usage of most existing methods. In response, this paper introduces IMPRINT, a novel diffusion-based generative model trained with a two-stage learning framework that decouples learning of identity preservation from that of compositing. The first stage is targeted for context-agnostic, identity-preserving pretraining of the object encoder, enabling the encoder to learn an embedding that is both view-invariant and conducive to enhanced detail preservation. The subsequent stage leverages this representation to learn seamless harmonization of the object composited to the background. In addition, IMPRINT incorporates a shape-guidance mechanism offering user-directed control over the compositing process. Extensive experiments demonstrate that IMPRINT significantly outperforms existing methods and various baselines on identity preservation and composition quality. Project page: https://song630.github.io/IMPRINT-Project-Page/

We present a novel approach to perform instance segmentation and counting for densely packed self-similar trees using a top-view RGB image sequence. We propose a solution that leverages pixel content, shape, and self-occlusion. First, we perform an initial over-segmentation of the image sequence and aggregate structural characteristics into a contour graph with temporal information incorporated. Second, using a graph convolutional network and its inherent local messaging passing abilities, we merge adjacent tree crown patches into a final set of tree crowns. Per various studies and comparisons, our method is superior to all prior methods and results in high-accuracy instance segmentation and counting despite the trees being tightly packed. Finally, we provide various forest image sequence datasets suitable for subsequent benchmarking and evaluation captured at different altitudes and leaf conditions.

Object compositing based on 2D images is a challenging problem since it typically involves multiple processing stages such as color harmonization, geometry correction and shadow generation to generate realistic results. Furthermore, annotating training data pairs for compositing requires substantial manual effort from professionals, and is hardly scalable. Thus, with the recent advances in generative models, in this work, we propose a selfsupervised framework for object compositing by leveraging the power of conditional diffusion models. Our framework can hollistically address the object compositing task in a unified model, transforming the viewpoint, geometry, color and shadow of the generated object while requiring no manual labeling. To preserve the input object's characteristics, we introduce a content adaptor that helps to maintain categori-cal semantics and object appearance. A data augmentation method is further adopted to improve the fidelity of the generator. Our method outperforms relevant baselines in both realism and faithfulness of the synthesized result images in a user study on various real-world images.

Modeling and designing urban building layouts is of significant interest in computer vision, computer graphics, and urban applications. A building layout consists of a set of buildings in city blocks defined by a network of roads. We observe that building layouts are discrete structures, consisting of multiple rows of buildings of various shapes, and are amenable to skeletonization for mapping arbitrary city block shapes to a canonical form. Hence, we propose a fully automatic approach to building layout generation using graph attention networks. Our method generates realistic urban layouts given arbitrary road networks, and enables conditional generation based on learned priors. Our results, including user study, demonstrate superior performance as compared to prior layout generation networks, support arbitrary city block and varying building shapes as demonstrated by generating layouts for 28 large cities.

Abstract We propose a framework to create projectively‐correct and seam‐free cube‐map images using generative adversarial learning. Deep generation of cube‐maps that contain the correct projection of the environment onto its faces is not straightforward as has been recognized in prior work. Our approach extends an existing framework, StyleGAN3, to produce cube‐maps instead of planar images. In addition to reshaping the output, we include a cube‐specific volumetric initialization component, a projective resampling component, and a modification of augmentation operations to the spherical domain. Our results demonstrate the network's generation capabilities trained on imagery from various 3D environments. Additionally, we show the power and quality of our GAN design in an inversion task, combined with navigation capabilities, to perform novel view synthesis.

Urban and environmental researchers seek to obtain building features (e.g., building shapes, counts, and areas) at large scales. However, blurriness, occlusions, and noise from prevailing satellite images severely hinder the performance of image segmentation, super-resolution, or deep-learning-based translation networks. In this article, we combine globally available satellite images and spatial geometric feature datasets to create a generative modeling framework that enables obtaining significantly improved accuracy in per-building feature estimation and the generation of visually plausible building footprints. Our approach is a novel design that compensates for the degradation present in satellite images by using a novel deep network setup that includes segmentation, generative modeling, and adversarial learning for instance-level building features. Our method has proven its robustness through large-scale prototypical experiments covering heterogeneous scenarios from dense urban to sparse rural. Results show better quality over advanced segmentation networks for urban and environmental planning, and show promise for future continental-scale urban applications.

Abstract This paper proposes and evaluates a sketching language to author crowd motion. It focuses on the path, speed, thickness, and density parameters of crowd motion. A sketch‐based vocabulary is proposed for each parameter and evaluated in a user study against complex crowd scenes. A sketch recognition pipeline converts the sketches into a crowd simulation. The user study results show that 1) participants at various skill levels and can draw accurate crowd motion through sketching, 2) certain sketch styles lead to a more accurate representation of crowd parameters, and 3) sketching allows to produce complex crowd motions in a few seconds. The results show that some styles although accurate actually are less preferred over less accurate ones.

Abstract Urban procedural modeling has benefited from recent advances in deep learning and computer graphics. However, few, if any, approaches have automatically produced procedural building roof models from a single overhead satellite image. Large‐scale roof modeling is important for a variety of applications in urban content creation and in urban planning (e.g., solar panel planning, heating/cooling/rainfall modeling). While the allure of modeling only from satellite images is clear, unfortunately structures obtained from the satellite images are often in low‐resolution, noisy and heavily occluded, thus getting a clean and complete view of urban structures is difficult. In this paper, we present a framework that exploits the inherent structure present in man‐made buildings and roofs by explicitly identifying the compact space of potential building shapes and roof structures. Then, we utilize this relatively compact space with a two‐component solution combining procedural modeling and deep learning. Specifically, we use a building decomposition component to separate the building into roof parts and predict regularized building footprints in a procedural format, and use a roof ridge detection component to refine the individual roof parts by estimating the procedural roof ridge parameters. Our qualitative and quantitative assessments over multiple satellite datasets show that our method outperforms various state‐of‐the‐art methods.

We introduce a technique to synthetically increase the framerate of semi-repetitive videos (i.e., videos of motion that repeats but not in an identical fashion) to aid in visualization. By reordering and combining frames from all repetitions, we produce a single non-repetitive sequence with much higher temporal resolution. Then, we use a novel frame warping technique based on a dense corrective flow to counteract differences between repetitions. The resulting video maintains smoothness of motion and additionally allows for seamless, infinite looping. We demonstrate the effectiveness of the proposed solution both quantitatively, by measuring the improvement over existing methods, and qualitatively, by performing a user evaluation and providing several examples in the article and accompanying video.

We introduce a new inverse modeling method to interactively design crowd animations. Few works focus on providing succinct high-level and large-scale crowd motion modeling. Our methodology is to read in real or virtual agent trajectory data and automatically infer a set of parameterized crowd motion models. Then, components of the motion models can be mixed, matched, and altered enabling rapidly producing new crowd motions. Our results show novel animations using real-world data, using synthetic data, and imitating real-world scenarios. Moreover, by combining our method with our interactive crowd trajectory sketching tool, we can create complex spatio-temporal crowd animations in about a minute.

Automatic creation of lightweight 3D building models from satellite image data enables large and widespread 3D interactive urban rendering. Towards this goal, we present an inverse procedural modeling method to automatically create building envelopes from satellite imagery. Our key observation is that buildings exhibit regular properties. Hence, we can overcome the low-resolution, noisy, and partial building data obtained from satellite by using a two stage inverse procedural modeling technique. Our method takes in point cloud data obtained from multi-view satellite stereo processing and produces a crisp and regularized building envelope suitable for fast rendering and optional projective texture mapping. Further, our results show highly complete building models with quality superior to that of other compared-to approaches.

Abstract We present a system to generate a procedural environment that produces a desired crowd behaviour. Instead of altering the behavioural parameters of the crowd itself, we automatically alter the environment to yield such desired crowd behaviour. This novel inverse approach is useful both to crowd simulation in virtual environments and to urban crowd planning applications. Our approach tightly integrates and extends a space discretization crowd simulator with inverse procedural modelling. We extend crowd simulation by goal exploration (i.e. agents are initially unaware of the goal locations), variable‐appealing sign usage and several acceleration schemes. We use Markov chain Monte Carlo to quickly explore the solution space and yield interactive design. We have applied our method to a variety of virtual and real‐world locations, yielding one order of magnitude faster crowd simulation performance over related methods and several fold improvement of crowd indicators.

Abstract We introduce an interactive tool for novice users to design mechanical objects made of 2.5D linkages. Users simply draw the shape of the object and a few key poses of its multiple moving parts. Our approach automatically generates a one‐degree‐of freedom linkage that connects the fixed and moving parts, such that the moving parts traverse all input poses in order without any collision with the fixed and other moving parts. In addition, our approach avoids common linkage defects and favors compact linkages and smooth motion trajectories. Finally, our system automatically generates the 3D geometry of the object and its links, allowing the rapid creation of a physical mockup of the designed object.

Abstract Creating a virtual city is demanded for computer games, movies, and urban planning, but it takes a lot of time to create numerous 3D building models. Procedural modeling has become popular in recent years to overcome this issue, but creating a grammar to get a desired output is difficult and time consuming even for expert users. In this paper, we present an interactive tool that allows users to automatically generate such a grammar from a single image of a building. The user selects a photograph and highlights the silhouette of the target building as input to our method. Our pipeline automatically generates the building components, from large‐scale building mass to fine‐scale windows and doors geometry. Each stage of our pipeline combines convolutional neural networks (CNNs) and optimization to select and parameterize procedural grammars that reproduce the building elements of the picture. In the first stage, our method jointly estimates camera parameters and building mass shape. Once known, the building mass enables the rectification of the façades, which are given as input to the second stage that recovers the façade layout. This layout allows us to extract individual windows and doors that are subsequently fed to the last stage of the pipeline that selects procedural grammars for windows and doors. Finally, the grammars are combined to generate a complete procedural building as output. We devise a common methodology to make each stage of this pipeline tractable. This methodology consists in simplifying the input image to match the visual appearance of synthetic training data, and in using optimization to refine the parameters estimated by CNNs. We used our method to generate a variety of procedural models of buildings from existing photographs.

course Public Access Share on Modeling 3D worlds: outdoor Author: Daniel Aliaga Purdue University, Purdue Purdue University, PurdueView Profile Authors Info & Claims SA '17: SIGGRAPH Asia 2017 CoursesNovember 2017 Article No.: 13Pages 1–233https://doi.org/10.1145/3134472.3134493Published:27 November 2017Publication History 0citation265DownloadsMetricsTotal Citations0Total Downloads265Last 12 Months19Last 6 weeks3 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteeReaderPDF

We present the first realistic, physically based, fully coupled, real-time weather design tool for use in urban procedural modeling. We merge designing of a 3D urban model with a controlled long-lasting spatiotemporal interactive simulation of weather. Starting from the fundamental dynamical equations similar to those used in state-of-the-art weather models, we present a novel simplified urban weather model for interactive graphics. Control of physically based weather phenomena is accomplished via an inverse modeling methodology. In our results, we present several scenarios of forward design, inverse design with high-level and detailed-level weather control and optimization, and comparisons of our method against well-known weather simulation results and systems.

Inverse procedural modeling discovers a procedural representation of an existing geometric model and the discovered procedural model then supports synthesizing new similar models. We introduce an automatic approach that generates a compact, efficient, and re-usable procedural representation of a polygonal 3D architectural model. This representation is then used for structure-aware editing and synthesis of new geometric models that resemble the original. Our framework captures the pattern hierarchy of the input model into a split tree data representation. A context-free split grammar, supporting a hierarchical nesting of procedural rules, is extracted from the tree, which establishes the base of our interactive procedural editing engine. We show the application of our approach to a variety of architectural structures obtained by procedurally editing web-sourced models. The grammar generation takes a few minutes even for the most complex input and synthesis is fully interactive for buildings composed of up to 200k polygons.

Abstract Synthesizing and exploring large‐scale realistic urban road networks is beneficial to 3D content creation, traffic animation and urban planning. In this paper, we present an interactive tool that allows untrained users to design roads with complex realistic details and styles. Roads are generated by growing a geometric graph. During a sketching phase, the user specifies the target area and the examples. During a growing phase, two types of growth are effectively applied to generate roads in the target area; example‐based growth uses patches extracted from the source example to generate roads that preserve some interesting structures in the example road networks; procedural‐based growth uses the statistical information of the source example while effectively adapting the roads to the underlying terrain and the already generated roads. User‐specified warping, blending and interpolation operations are used at will to produce new road network designs that are inspired by the examples. Finally, our method computes city blocks, individual parcels and plausible building and tree geometries. We have used our approach to create road networks covering up to 200 and containing over 3500 km of roads.

3D modeling remains a notoriously difficult task for novices despite significant research effort to provide intuitive and automated systems. We tackle this problem by combining the strengths of two popular domains: sketch-based modeling and procedural modeling. On the one hand, sketch-based modeling exploits our ability to draw but requires detailed, unambiguous drawings to achieve complex models. On the other hand, procedural modeling automates the creation of precise and detailed geometry but requires the tedious definition and parameterization of procedural models. Our system uses a collection of simple procedural grammars, called snippets, as building blocks to turn sketches into realistic 3D models. We use a machine learning approach to solve the inverse problem of finding the procedural model that best explains a user sketch. We use non-photorealistic rendering to generate artificial data for training convolutional neural networks capable of quickly recognizing the procedural rule intended by a sketch and estimating its parameters. We integrate our algorithm in a coarse-to-fine urban modeling system that allows users to create rich buildings by successively sketching the building mass, roof, facades, windows, and ornaments. A user study shows that by using our approach non-expert users can generate complex buildings in just a few minutes.

Thanks to the recent advances in computational photography and remote sensing, point clouds of buildings are becoming increasingly available, yet their processing poses various challenges. In our work, we tackle the problem of point cloud completion and editing and we approach it via inverse procedural modeling. Contrary to the previous work, our approach operates directly on the point cloud without an intermediate triangulation. Our approach consists of 1) semi-automatic segmentation of the input point cloud with segment comparison and template matching to detect repeating structures, 2) a consensus-based voting schema and a pattern extraction algorithm to discover completed terminal geometry and their patterns of usage, all encoded into a context-free grammar, and 3) an interactive editing tool where the user can create new point clouds by using procedural copy and paste operations, and smart resizing. We demonstrate our approach on editing of building models with up to 1.8M points. In our implementation, preprocessing takes up to several minutes and a single editing operation needs from one second to one minute depending on the model size and the operation type.

Recent shape retrieval and interactive modeling algorithms enable the re-use of existing models in many applications. However, most of those techniques require a pre-labeled model with some semantic information. We introduce a fully automatic approach to simultaneously segment and detect similarities within an existing 3D architectural model. Our framework approaches the segmentation problem as a weighted minimum set cover over an input triangle soup, and maximizes the repetition of similar segments to find a best set of unique component types and instances. The solution for this set-cover formulation starts with a search space reduction to eliminate unlikely combinations of triangles, and continues with a combinatorial optimization within each disjoint subspace that outputs the components and their types. We show the discovered components of a variety of architectural models obtained from public databases. We demonstrate experiments testing the robustness of our algorithm, in terms of threshold sensitivity, vertex displacement, and triangulation variations of the original model. In addition, we compare our components with those of competing approaches and evaluate our results against user-based segmentations. We have processed a database of 50 buildings, with various structures and over 200K polygons per building, with a segmentation time averaging up to 4 minutes.

We describe a technique to generate imagery with improved sharpness for individuals having refractive vision problems. Our method can reduce their dependence on corrective eyewear. It also benefits individuals with normal vision by improving visual acuity at a distance and of small details. Our approach does not require custom hardware. Instead, the calculated images can be shown on a standard computer display, on printed paper, or superimposed on a physical scene using a projector. Our technique uses a constrained total variation method to produce a deconvolution result which, upon observation, appears sharp at the edges. We introduce a novel relative total variation term that enables controlling ringing reduction, contrast gain, and sharpness. The end result is the ability to generate sharper appearing images, even for individuals with refractive vision problems including myopia, hyperopia, presbyopia, and astigmatism. Our approach has been validated in simulation, in camera-screen experiments, and in a study with human observers.

We present a framework for the conversion of existing 3D unstructured urban models into a compact procedural representation that enables model synthesis, querying, and simplification of large urban areas. During the de-instancing phase, a dissimilarity-based clustering is performed to obtain a set of building components and component types. During the proceduralization phase, the components are arranged into a context-free grammar, which can be directly edited or interactively manipulated. We applied our approach to convert several large city models, with up to 19,000 building components spanning over 180 km squares, into procedural models of a few thousand terminals, non-terminals, and 50-100 rules.

Abstract Designing and optimizing traffic behavior and animation is a challenging problem of interest to virtual environment content generation and to urban planning and design. While some traffic simulation methods have appeared in computer graphics, most related systems focus on the design of buildings, roads, or cities but without explicitly considering urban traffic. To our knowledge, our work provides the first interactive approach which enables a designer to specify a desired vehicular traffic behavior (e.g., road occupancy, travel time, emissions, etc.) and the system will automatically compute what realistic 3D urban model (e.g., an interconnected network of roads, parcels, and buildings) yields the specified behavior. Our system both altered and improved traffic behavior in novel procedurally‐generated cities and in road networks of existing cities. Our urban models contain up to 360 km of roads, 300,000 vehicles, and typically cover four hours of simulated peak traffic time. The typical editing session time to “paint” a new traffic pattern and to compute the new/changed urban model is two to five minutes.

Abstract This paper provides a comprehensive overview of urban reconstruction. While there exists a considerable body of literature, this topic is still under active research. The work reviewed in this survey stems from the following three research communities: computer graphics, computer vision and photogrammetry and remote sensing. Our goal is to provide a survey that will help researchers to better position their own work in the context of existing solutions, and to help newcomers and practitioners in computer graphics to quickly gain an overview of this vast field. Further, we would like to bring the mentioned research communities to even more interdisciplinary work, since the reconstruction problem itself is by far not solved.

One augmented reality approach is to use digital projectors to alter the appearance of a physical scene, avoiding the need for head-mounted displays or special goggles. Instead, spatial augmented reality (SAR) systems depend on having sufficient light radiance to compensate the surface's colors to those of a target visualization. However, standard SAR systems in dark room settings may suffer from insufficient light radiance causing bright colors to exhibit unexpected color shifts, resulting in a misleading visualization. We introduce a SAR framework which focuses on minimally altering the appearance of arbitrarily shaped and colored objects to exploit the presence of environment/room light as an additional light source to achieve compliancy for bright colors. While previous approaches have compensated for environment light, none have explicitly exploited the environment light to achieve bright, previously incompliant colors. We implement a full working system and compared our results to solutions achievable with standard SAR systems.

Abstract We present a method for interactive procedural generation of parcels within the urban modeling pipeline. Our approach performs a partitioning of the interior of city blocks using user‐specified subdivision attributes and style parameters. Moreover, our method is both robust and persistent in the sense of being able to map individual parcels from before an edit operation to after an edit operation – this enables transferring most, if not all, customizations despite small to large‐scale interactive editing operations. The guidelines guarantee that the resulting subdivisions are functionally and geometrically plausible for subsequent building modeling and construction. Our results include visual and statistical comparisons that demonstrate how the parcel configurations created by our method can closely resemble those found in real‐world cities of a large variety of styles. By directly addressing the block subdivision problem, we intend to increase the editability and realism of the urban modeling pipeline and to become a standard in parcel generation for future urban modeling methods.

We present a system that superimposes multiple projections onto an object of arbitrary shape and color to produce high-resolution appearance changes. Our system produces appearances at an improved resolution compared to prior works and can change appearances at near interactive rates. Three main components are central to our system. First, the problem of computing compensation images is formulated as a constrained optimization which yields high-resolution appearances. Second, decomposition of the target appearance into base and scale images enables fast swapping of appearances on the object by requiring the constrained optimization to be computed only once per object. Finally, to make high-quality appearance edits practical, an elliptical Gaussian is used to model projector pixels and their interaction between projectors. To the best of our knowledge, we build the first system that achieves high-resolution and high-quality appearance edits using multiple superimposed projectors on complex nonplanar colored objects. We demonstrate several appearance edits including specular lighting, subsurface scattering, inter-reflections, and color, texture, and geometry changes on objects with different shapes and colors.

We introduce tailored displays that enhance visual acuity by decomposing virtual objects and placing the resulting anisotropic pieces into the subject's focal range. The goal is to free the viewer from needing wearable optical corrections when looking at displays. Our tailoring process uses aberration and scattering maps to account for refractive errors and cataracts. It splits an object's light field into multiple instances that are each in-focus for a given eye sub-aperture. Their integration onto the retina leads to a quality improvement of perceived images when observing the display with naked eyes. The use of multiple depths to render each point of focus on the retina creates multi-focus, multi-depth displays. User evaluations and validation with modified camera optics are performed. We propose tailored displays for daily tasks where using eyeglasses are unfeasible or inconvenient (e.g., on head-mounted displays, e-readers, as well as for games); when a multi-focus function is required but undoable (e.g., driving for farsighted individuals, checking a portable device while doing physical activities); or for correcting the visual distortions produced by high-order aberrations that eyeglasses are not able to.

We propose a framework that enables adding intuitive high level control to an existing urban procedural model. In particular, we provide a mechanism to interactively edit urban models, a task which is important to stakeholders in gaming, urban planning, mapping, and navigation services. Procedural modeling allows a quick creation of large complex 3D models, but controlling the output is a well-known open problem. Thus, while forward procedural modeling has thrived, in this paper we add to the arsenal an inverse modeling tool. Users, unaware of the rules of the underlying urban procedural model, can alternatively specify arbitrary target indicators to control the modeling process. The system itself will discover how to alter the parameters of the urban procedural model so as to produce the desired 3D output. We label this process inverse design.

We propose a novel approach for the reconstruction of urban structures from 3D point clouds with an assumption of Manhattan World (MW) building geometry; i.e., the predominance of three mutually orthogonal directions in the scene. Our approach works in two steps. First, the input points are classified according to the MW assumption into four local shape types: walls, edges, corners, and edge corners. The classified points are organized into a connected set of clusters from which a volume description is extracted. The MW assumption allows us to robustly identify the fundamental shape types, describe the volumes within the bounding box, and reconstruct visible and occluded parts of the sampled structure. We show results of our reconstruction that has been applied to several synthetic and real-world 3D point data sets of various densities and from multiple viewpoints. Our method automatically reconstructs 3D building models from up to 10 million points in 10 to 60 seconds.

We address the open problem of spatial distribution of vegetation in urban environments by introducing a user-guided simulation and procedural system for integrating plants into the interactive design process of 3D urban models. Our approach uses as input 3D geometry of an urban layout from which it infers initial conditions and parameters of procedural rules. A level of manageability is calculated for each area of the urban space. The manageability level defines the amount of influence between the wild ecosystem simulation, where the plants compete for resources and seed freely, and the managed ecosystem, where nearly no seeding is allowed and the plants grow only under well-defined conditions. The wild ecosystems are handled by a simulation of plant competition for resources, whereas the procedural generation is based on an expandable set of behavioral rules of owners and typical plant management. Our system provides an interactive semi-automatic method to calculate a spatial plant distribution and to create an urban model with plants covering an area of several square kilometers in less than a minute. It provides a high degree of controllability and works tightly with an urban simulation system. We show various examples, such as plant development over time in managed and unmanaged areas, effect of procedural rules on the plant distribution, and the effect of changing the level of manageability and the plant distribution.

Abstract Projection‐based appearances are used in a variety of computer graphics applications to impart different appearances onto physical surfaces using digitally controlled projector light. To achieve a compliant appearance, all points on the physical surface must be altered to the colours of the desired target appearance; otherwise, an incompliant appearance results in a misleading visualization. Previous systems typically assume to operate with compliant appearances or restrict themselves to the simpler case of white surfaces. To achieve compliancy, one may change the physical surface's albedo, increase the amount of projector light radiance available or modify the target appearance's colours. This paper presents an approach to modify a target appearance to achieve compliant appearance editing without altering the physical surface or the projector setup. Our system minimally alters the target appearance's colours while maintaining cues important for perceptual similarity (e.g. colour constancy). First, we discuss how to measure colour compliancy. Next, we describe our approach to partition the physical surface into patches based on the surface's colours and the target appearance's colours. Finally, we describe our appearance optimization process, which computes a compliant appearance that is as perceptually similar as possible to the target appearance's colours. We perform several real‐world projection‐based appearances and compare our results to naïve approaches, which either ignore compliancy or simply reduce the appearance's overall brightness.

Structure from motion (SFM) is the problem of recovering the geometry of a scene from a stream of images taken from unknown viewpoints. One popular approach to estimate the geometry of a scene is to track scene features on several images and reconstruct their position in 3-D. During this process, the unknown camera pose must also be recovered. Unfortunately, recovering the pose can be an ill-conditioned problem which, in turn, can make the SFM problem difficult to solve accurately. We propose an alternative formulation of the SFM problem with fixed internal camera parameters known a priori. In this formulation, obtained by algebraic variable elimination, the external camera pose parameters do not appear. As a result, the problem is better conditioned in addition to involving much fewer variables. Variable elimination is done in three steps. First, we take the standard SFM equations in projective coordinates and eliminate the camera orientations from the equations. We then further eliminate the camera center positions. Finally, we also eliminate all 3-D point positions coordinates, except for their depths with respect to the camera center, thus obtaining a set of simple polynomial equations of degree two and three. We show that, when there are merely a few points and pictures, these "depth-only equations" can be solved in a global fashion using homotopy methods. We also show that, in general, these same equations can be used to formulate a pose-free cost function to refine SFM solutions in a way that is more accurate than by minimizing the total reprojection error, as done when using the bundle adjustment method. The generalization of our approach to the case of varying internal camera parameters is briefly discussed.

THIS IEEE Transactions on Visualization and Computer Graphics (TVCG) special section brings to you extended versions of four excellent articles originally presented at the Symposium on Interactive 3D Graphics and Games (I3D) in 2010. Since 1986, I3D has been a venue for cutting-edge computer science research in interactive graphics and human interaction. The field has grown significantly since the founding of I3D, but the symposium has maintained its premier status. Today, I3D is known for the close integration of academia and industry, and such is clear from the mixture of attendees and from the mixture of authors of the articles in this section. The symposium itself is also very popular, for it provides a great venue for meeting fellow researchers, forming new collaborations, and brainstorming new solutions. I3D 2010 received 71 submissions from 19 countries in four continents. Each paper received at least three independent reviews by a group of 86 experts in the field of computer graphics and interactive techniques. After a discussion phase, 23 papers were selected for publication and presentation at the symposium (an acceptance rate of 32 percent). There have been no limits imposed on the number of accepted papers, and the decisions were based purely on the submissions’ merits. Following a tradition started with I3D 2009, we invited the authors of four of the accepted papers to expand their work with new insights and results for publication in this special section of TVCG on I3D. The choice of the invited papers was based on the evaluation and comments of our expert reviewers. These papers went through a complete journal review process, which included multiple iterations of editing and review. “Stochastic Transparency” by Eric Enderton, Erik Sintorn, Peter Shirley, and David Luebke addresses the problem of achieving order-independent transparency for interactive rendering. The solution provided by the authors presents several desirable features: the required memory does not depend on the scene’s depth complexity; it renders a fixed number of passes, with its running time growing linearly with the number of fragments; it is general in the sense that it can replace multiple transparency algorithms for different scenarios; it is also easy to implement, and is a good fit to modern, massively parallel GPUs. The technique provides a practical solution for interactive applications, being capable of rendering all kinds of transparent geometry, and providing a unifying approach for orderindependent transparency, anti-aliasing, and shadowing. “Efficient Sparse Voxel Octrees” by Samuli Laine and Tero Karras explores the use of voxel representations for rendering complex detailed geometry on current and future GPUs. In order to efficiently ray cast the resulting models, the authors propose a compact data structure for storing voxels. In this representation, voxels are augmented with contour information to increase geometry resolution. The paper also describes a new normal compression format for storing high-precision object-space normals. Using the proposed infrastructure, the authors challenge the current graphics trend of using a base mesh augmented with fine details represented by textures and displacement maps: as the amount of color and geometric detail information grows beyond a certain limit, wouldn’t it make more sense to use a voxel representation to store both geometry and its associated attributes? “Frankenrigs: Building Character Rigs from Multiple Sources” by Christian Miller, Okan Arikan, and Don Fussel addresses an important practical problem in 3D character animation. Rigging and skinning a character is a fundamental animation task, but a time-consuming and tedious one even for experts. The authors present an automatic solution for generating good quality rigging and skinning that can greatly simplify the work of animators. Given a character’s 3D mesh with annotated joints, the technique finds good matches for individual parts of the model by scanning a database of partial rigs. By transferring information from the partial rigs to the target mesh, the approach produces a skeleton and skinning weights whose quality are similar to the ones produced by expert animators. “Improving Shape Depiction under Arbitrary Rendering” by Romain Vergne, Romain Pacanowski, Pascal Barla, Xavier Granier, and Christophe Schlick introduces a technique for enhancing shape depiction by scaling the reflected light as a function of curvature and material properties. Their solution can be used with any kind of material and works with both direct and global illumination. It can also be used with different lighting environments, supports inter-reflections, and works in real time. By providing intuitive control over the resulting shading, this 1034 IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS, VOL. 17, NO. 8, AUGUST 2011

We present a passive computer vision method that exploits existing mapping and navigation databases in order to automatically create 3D building models. Our method defines a grammar for representing changes in building geometry that approximately follow the Manhattan-world assumption which states there is a predominance of three mutually orthogonal directions in the scene. By using multiple calibrated aerial images, we extend previous Manhattan-world methods to robustly produce a single, coherent, complete geometric model of a building with partial textures. Our method uses an optimization to discover a 3D building geometry that produces the same set of façade orientation changes observed in the captured images. We have applied our method to several real-world buildings and have analyzed our approach using synthetic buildings.

Modeling moving and deforming objects requires capturing as much information as possible during a very short time. When using off-the-shelf hardware, this often hinders the resolution and accuracy of the acquired model. Our key observation is that in as little as four frames both sparse surface-positional measurements and dense surface-orientation measurements can be acquired using a combination of structured light and photometric stereo, resulting in high-resolution models of moving and deforming objects. Our system projects alternating geometric and photometric patterns onto the object using a set of three projectors and captures the object using a synchronized camera. Small motion among temporally close frames is compensated by estimating the optical flow of images captured under the uniform illumination of the photometric light. Then spatial-temporal photogeometric reconstructions are performed to obtain dense and accurate point samples with a sampling resolution equal to that of the camera. Temporal coherence is also enforced. We demonstrate our system by successfully modeling several moving and deforming real-world objects.

Abstract We present an important step towards the solution of the problem of inverse procedural modeling by generating parametric context‐free L‐systems that represent an input 2D model. The L‐system rules efficiently code the regular structures and the parameters represent the properties of the structure transformations. The algorithm takes as input a 2D vector image that is composed of atomic elements, such as curves and poly‐lines. Similar elements are recognized and assigned terminal symbols of an L‐system alphabet. The terminal symbols' position and orientation are pair‐wise compared and the transformations are stored as points in multiple 4D transformation spaces. By careful analysis of the clusters in the transformation spaces, we detect sequences of elements and code them as L‐system rules. The coded elements are then removed from the clusters, the clusters are updated, and then the analysis attempts to code groups of elements in (hierarchies) the same way. The analysis ends with a single group of elements that is coded as an L‐system axiom. We recognize and code branching sequences of linearly translated, scaled, and rotated elements and their hierarchies. The L‐system not only represents the input image, but it can also be used for various editing operations. By changing the L‐system parameters, the image can be randomized, symmetrized, and groups of elements and regular structures can be edited. By changing the terminal and non‐terminal symbols, elements or groups of elements can be replaced.

Abstract Urban spaces consist of a complex collection of buildings, parcels, blocks and neighbourhoods interconnected by streets. Accurately modelling both the appearance and the behaviour of dense urban spaces is a significant challenge. The recent surge in urban data and its availability via the Internet has fomented a significant amount of research in computer graphics and in a number of applications in urban planning, emergency management and visualization. In this paper, we seek to provide an overview of methods spanning computer graphics and related fields involved in this goal. Our paper reports the most prominent methods in urban modelling and rendering, urban visualization and urban simulation models. A reader will be well versed in the key problems and current solution methods.

We present a self-calibrating photogeometric method using only off--the-shelf hardware that enables quickly and robustly obtaining multimillion point-sampled and colored models of real-world objects. Some previous efforts use a priori calibrated systems to separately acquire geometric and photometric information. Our key enabling observation is that a digital projector can be simultaneously used as either an active light source or as a virtual camera (as opposed to a digital camera, which cannot be used for both). We present our self--calibrating and multiviewpoint 3D acquisition method, based on structured light, which simultaneously obtains mutually registered surface position and surface normal information and produces a single high-quality model. Acquisition processing freely alternates between using a geometric setup and using a photometric setup with the same hardware configuration. Further, our approach generates reconstructions at the resolution of the camera and not only the projector. We show the results of capturing several high-quality models of real--world objects.

Many visualization applications benefit from displaying content on real-world objects rather than on a traditional display (e.g., a monitor). This type of visualization display is achieved by projecting precisely controlled illumination from multiple projectors onto the real-world colored objects. For such a task, the placement of the projectors is critical in assuring that the desired visualization is possible. Using ad hoc projector placement may cause some appearances to suffer from color shifting due to insufficient projector light radiance being exposed onto the physical surface. This leads to an incorrect appearance and ultimately to a false and potentially misleading visualization. In this paper, we present a framework to discover the optimal position and orientation of the projectors for such projection-based visualization displays. An optimal projector placement should be able to achieve the desired visualization with minimal projector light radiance. When determining optimal projector placement, object visibility, surface reflectance properties, and projector-surface distance and orientation need to be considered. We first formalize a theory for appearance editing image formation and construct a constrained linear system of equations that express when a desired novel appearance or visualization is possible given a geometric and surface reflectance model of the physical surface. Then, we show how to apply this constrained system in an adaptive search to efficiently discover the optimal projector placement which achieves the desired appearance. Constraints can be imposed on the maximum radiance allowed by the projectors and the projectors' placement to support specific goals of various visualization applications. We perform several real-world and simulated appearance edits and visualizations to demonstrate the improvement obtained by our discovered projector placement over ad hoc projector placement.

Obtaining models of dynamic 3D objects is an important part of content generation for computer graphics. Numerous methods have been extended from static scenarios to model dynamic scenes. If the states or poses of the dynamic object repeat often during a sequence (but not necessarily periodically), we call such a repetitive motion. There are many objects, such as toys, machines, and humans, undergoing repetitive motions. Our key observation is that when a motion-state repeats, we can sample the scene under the same motion state again but using a different set of parameters; thus, providing more information of each motion state. This enables robustly acquiring dense 3D information difficult for objects with repetitive motions using only simple hardware. After the motion sequence, we group temporally disjoint observations of the same motion state together and produce a smooth space-time reconstruction of the scene. Effectively, the dynamic scene modeling problem is converted to a series of static scene reconstructions, which are easier to tackle. The varying sampling parameters can be, for example, structured-light patterns, illumination directions, and viewpoints resulting in different modeling techniques. Based on this observation, we present an image-based motion-state framework and demonstrate our paradigm using either a synchronized or an unsynchronized structured-light acquisition method.

Abstract 3D computer graphics models and digitally‐controlled manufacturing have come together to enable the design, visualization, simulation, and automated creation of complex 3D objects. In our work, we propose and implement a framework for designing computer graphics objects and digitally manufacturing them such that no adversary can make imitations or counterfeit copies of the physical object, even if the adversary has a large number of original copies of the object, knowledge of the original object design, and has manufacturing precision that is comparable to or superior to that of the legitimate creator of the object. Our approach is to design and embed a signature on the surface of the object which acts as a certificate of genuinity of the object. The signature is detectable by a signature‐reading device, based on methods in computer graphics and computer vision, which contains some of the secret information that was used when marking the physical object. Further, the compromise of a signature‐reading device by an adversary who is able to extract all its secrets, does not enable the adversary to create counterfeit objects that fool other readers, thereby still enabling reliable copy detection. We implemented a prototype of our scheme end‐to‐end, including the production of the physical object and the genuinity‐testing device.

Many applications in computer graphics require detailed 3D digital models of real-world environments. The automatic and semi-automatic modeling of such spaces presents several fundamental challenges. In this work, we present an easy and robust camera-based acquisition approach for the modeling of 3D scenes which is a significant departure from current methods. Our approach uses a novel pose-free formulation for 3D reconstruction. Unlike self-calibration, omitting pose parameters from the acquisition process implies no external calibration data must be computed or provided. This serves to significantly simplify acquisition, to fundamentally improve the robustness and accuracy of the geometric reconstruction given noise in the measurements or error in the initial estimates, and to allow using uncalibrated active correspondence methods to obtain robust data. Aside from freely taking pictures and moving an uncalibrated digital projector, scene acquisition and scene point reconstruction is automatic and requires pictures from only a few viewpoints. We demonstrate how the combination of these benefits has enabled us to acquire several large and detailed models ranging from 0.28 to 2.5 million texture-mapped triangles.

The main contribution of our work is in closing the loop between behavioral and geometrical modeling of cities. Editing of urban design variables is performed intuitively and visually using a graphical user interface. Any design variable can be constrained or changed. The design process uses an iterative dynamical system for reaching equilibrium: a state where the demands of behavioral modeling match those of geometrical modeling. 3D models are generated in a few seconds and conform to plausible urban behavior and urban geometry. Our framework includes an interactive agent-based behavioral modeling system as well as adaptive geometry generation algorithms. We demonstrate interactive and incremental design and editing for synthetic urban spaces spanning over 200 square kilometers.

Urban simulation models and their visualization are used to help regional planning agencies evaluate alternative transportation investments, land use regulations, and environmental protection policies. Typical urban simulations provide spatially distributed data about number of inhabitants, land prices, traffic, and other variables. In this article, we build on a synergy of urban simulation, urban visualization, and computer graphics to automatically infer an urban layout for any time step of the simulation sequence. In addition to standard visualization tools, our method gathers data of the original street network, parcels, and aerial imagery and uses the available simulation results to infer changes to the original urban layout and produce a new and plausible layout for the simulation results. In contrast with previous work, our approach automatically updates the layout based on changes in the simulation data and thus can scale to a large simulation over many years. The method in this article offers a substantial step forward in building integrated visualization and behavioral simulation systems for use in community visioning, planning, and policy analysis. We demonstrate our method on several real cases using a 200 GB database for a 16,300 km2 area surrounding Seattle.

Modeling real-world scenes, beyond diffuse objects, plays an important role in computer graphics, virtual reality, and other commercial applications. One active approach is projecting binary patterns in order to obtain correspondence and reconstruct a densely sampled 3D model. In such structured-light systems, determining whether a pixel is directly illuminated by the projector is essential to decoding the patterns. When a scene has abundant indirect light, this process is especially difficult. In this paper, we present a robust pixel classification algorithm for this purpose. Our method correctly establishes the lower and upper bounds of the possible intensity values of an illuminated pixel and of a non-illuminated pixel. Based on the two intervals, our method classifies a pixel by determining whether its intensity is within one interval but not in the other. Our method performs better than standard method due to the fact that it avoids gross errors during decoding process caused by strong inter-reflections. For the remaining uncertain pixels, we apply an iterative algorithm to reduce the inter-reflection within the scene. Thus, more points can be decoded and reconstructed after each iteration. Moreover, the iterative algorithm is carried out in an adaptive fashion for fast convergence.

Structured-light methods actively generate geometric correspondence data between projectors and cameras in order to facilitate robust 3D reconstruction. In this paper, we present photogeometric structured light whereby a standard structured light method is extended to include photometric methods. Photometric processing serves the double purpose of increasing the amount of recovered surface detail and of enabling the structured-light setup to be robustly self-calibrated. Further, our framework uses a photogeometric optimization that supports the simultaneous use of multiple cameras and projectors and yields a single and accurate multi-view 3D model which best complies with photometric and geometric data.

In a wide range of applications, we often wish to quickly inspect and visualize historically significant and highly detailed objects. For such scientific illustration applications, often the focus is on high-frequency surface details and on conveying important shape and feature information. In our work, we provide a complete system to visualize objects on the spot. Our approach uses photometric and geometric processing, combined with a set of visualization methods tuned to the interactive inspection and analysis of objects. Highly accurate models are acquired in about 30 seconds using an uncalibrated setup, obtaining both detailed surface geometry and detailed surface normal information. Subsequently, captured objects are visually tracked enabling hand-held manipulation and visualization. In this paper, we demonstrate our system using several real-world objects.

In this paper, we introduce a system to virtually restore damaged or historically significant objects without needing to physically change the object in any way. Our work addresses both creating a restored synthetic version of the object as viewed from a camera and projecting the necessary light, using digital projectors, to give the illusion of the object being restored. The restoration algorithm uses an energy minimization method to enforce a set of criteria over the surface of the object and provides an interactive tool to the user which can compute a restoration in a few minutes. The visual compensation method develops a formulation that is particularly concerned with obtaining bright compensations under a specified maximum amount of light. The bound on the amount of light is of crucial importance when viewing and restoring old and potentially fragile objects. Finally, we demonstrate our system by restoring several deteriorated and old objects enabling the observer to view the original or restored object at will.

We present an interactive system for synthesizing urban layouts by example. Our method simultaneously performs both a structure-based synthesis and an image-based synthesis to generate a complete urban layout with a plausible street network and with aerial-view imagery. Our approach uses the structure and image data of real-world urban areas and a synthesis algorithm to provide several high-level operations to easily and interactively generate complex layouts by example. The user can create new urban layouts by a sequence of operations such as join, expand, and blend without being concerned about low-level structural details. Further, the ability to blend example urban layout fragments provides a powerful way to generate new synthetic content. We demonstrate our system by creating urban layouts using example fragments from several real-world cities, each ranging from hundreds to thousands of city blocks and parcels.

Modeling 3D objects and scenes is an important part of computer graphics. One approach to modeling is projecting binary patterns onto the scene in order to obtain correspondences and reconstruct a densely sampled 3D model. In such structured light systems, determining whether a pixel is directly illuminated by the projector is essential to decoding the patterns. In this paper, we introduce a robust, efficient, and easy to implement pixel classification algorithm for this purpose. Our method correctly establishes the lower and upper bounds of the possible intensity values of an illuminated pixel and of a non-illuminated pixel. Based on the two intervals, our method classifies a pixel by determining whether its intensity is within one interval and not in the other. Experiments show that our method improves both the quantity of decoded pixels and the quality of the final reconstruction producing a dense set of 3D points, inclusively for complex scenes with indirect lighting effects. Furthermore, our method does not require newly designed patterns; therefore, it can be easily applied to previously captured data.

Reconstructing large models from images is a significant challenge for computer vision, computer graphics, and related fields. In this paper, we present an approach for simplifying the reconstruction process by mathematically eliminating external camera parameters. This results in less parameters to estimate and in an overall significantly more robust and accurate reconstruction. We reformulate the problem in such a manner as to be able to identify invariants, eliminate superfluous parameters, and measure the performance of our formulation under various conditions. We compare a two-step camera orientation-free method, where the majority of the points are reconstructed using a linear equation set, and a camera position-and- orientation free method, using a degree-two equation set. Both approaches use a full perspective camera and are applied to synthetic and real-world datasets.

Computer graphics applications such as movie effects, video gaming, and product demonstration demand 3D models of dynamic objects. For this purpose, numerous methods, such as light fields, stereo reconstruction and visual hulls have been extended to model dynamic objects. These methods use multiple cameras to acquire images simultaneously and use the synchronized samples to reconstruct the model for each time instance. However, a large number of cameras are required to obtain compelling results. We introduce an efficient acquisition and modeling schema for dynamic objects with repetitive motions. Our method requires as few as two cameras. The key idea is that repetitive motions can be described by a finite number of states. Images capturing the same state can be grouped together and fed to the later modeling phase as if they are captured from multiple cameras simultaneously. Our work includes an acquisition system with interactive feedback, a graph traversal algorithm to help obtain a near minimum subset of images to sample the object and its motion, and a space-time image optimization method. We demonstrate this system using several datasets with different complexity of motion, and different number of desired viewpoints.

Accurately reconstructing the 3D geometry of a scene or object observed on 2D images is a difficult problem: there are many unknowns involved (camera pose, scene structure, depth factors) and solving for all these unknowns simultaneously is computationally intensive and suffers from numerical instability. In this paper, we algebraically decouple some of the unknowns so that they can be solved for independently. Decoupling the pose from the other variables has been previously discussed in the literature. Unfortunately, pose estimation is an ill-conditioned problem. In this paper, we algebraically eliminate all the camera pose parameters (i.e., position and orientation) from the structure-from-motion equations for an internally calibrated camera. We then also fully eliminate the structure coordinates from the equations. This yields a very simple set of homogeneous polynomial equations of low degree involving only the depths of the observed points. When considering a small number of tracked points and pictures (e.g., five points on two pictures), these equations can be solved using the sparse resultant method.

Interactive visualization of architecture provides a way to quickly visualize existing or novel buildings and structures. Such applications require both fast rendering and an effortless input regimen for creating and changing architecture using high-level editing operations that automatically fill in the necessary details. Procedural modeling and synthesis is a powerful paradigm that yields high data amplification and can be coupled with fast-rendering techniques to quickly generate plausible details of a scene without much or any user interaction. Previously, forward generating procedural methods have been proposed where a procedure is explicitly created to generate particular content. In this paper, we present our work in inverse procedural modeling of buildings and describe how to use an extracted repertoire of building grammars to facilitate the visualization and quick modification of architectural structures and buildings. We demonstrate an interactive application where the user draws simple building blocks and, using our system, can automatically complete the building "in the style of" other buildings using view-dependent texture mapping or nonphotorealistic rendering techniques. Our system supports an arbitrary number of building grammars created from user subdivided building models and captured photographs. Using only edit, copy, and paste metaphors, the entire building styles can be altered and transferred from one building to another in a few operations, enhancing the ability to modify an existing architectural structure or to visualize a novel building in the style of the others.

Structure from motion (SFM) is the problem of reconstructing the geometry of a scene from a stream of images. In this problem, the geometry of the scene must be inferred from images, along with the camera pose parameters. Bundle adjustment (BA) is a refinement method used to improve SFM solutions. It consists in simultaneously improving a set of initial estimates for all parameters (structure and camera pose) by minimizing a global cost function. It is generally considered to be highly accurate, and so is typically used as a last refinement step in most current SFM methods. Unfortunately, estimating the pose of the camera from a stream of images is an ill-conditioned problem. We thus propose a BA adjustment formulation which does not involve solving for the camera orientations. We tested this approach on several real world models. The numerical results obtained show that this approach is much less affected by noise than traditional BA.

Rendering a scene using a single depth image suffers from disocclusion errors as the view translates away from the reference view. We present the depth discontinuity occlusion camera (DDOC), a non-pinhole camera that samples surfaces which are hidden in the reference view, but are likely to become visible in nearby views. The DDOC reference image alleviates disocclusion errors; since it has a single layer, it maintains the advantages of regular depth images such as bounded number of samples, efficient incremental processing, and implicit connectivity. The DDOC is defined by the reference view and the geometry it encompasses. The reference planar pinhole camera is 3D distorted at depth discontinuities. The distortion is fine-grain controlled with a distortion map, which allows handling complex scenes. The DDOC provides fast projection so the reference image is constructed efficiently with the feed-forward graphics pipeline.

Today’s technology enables a rich set of virtual and mixedreality applications and provides a degree of connectivity and interactivity beyond that of traditional teleconferencing scenarios. In this paper, we present the Mixed-Reality Tabletop (MRT), an example of a teleconferencing framework for networked mixed reality in which real and virtual worlds coexist and users can focus on the task instead of computer interaction. For example, students could use real-world objects to participate in physical simulations such as orbital motion, collisions, and fluid flow in a common virtual environment. Our framework isolates the lowlevel system details from the developer and provides a simple programming interface for developing novel applications in as little as a few minutes, using low-cost hardware. We discuss our implementation complete with methods of hand and object tracking, user interface, and example applications focused on remote teaching and learning.

Image-based rendering (IBR) systems create photorealistic views of complex 3D environments by resampling large collections of images captured in the environment. The quality of the resampled images increases significantly with higher image capture density. Thus, a significant challenge in interactive IBR systems is to provide both fast image access along arbitrary viewpoint paths and efficient storage of large image data sets. We describe a compression scheme based on a spatial image hierarchy that meets the requirements of interactive IBR walkthroughs. By exploiting image coherence over the entire image capture plane, we achieve compression performance similar to traditional motion-compensated schema, e.g., MPEG, yet allow image access along arbitrary paths. Furthermore, by exploiting graphics hardware for image resampling, we achieve interactive display rates during IBR walkthroughs.

We present build-by-number, a technique for quickly designing architectural structures that can be rendered photorealistically at interactive rates. We combine image-based capturing and rendering with procedural modeling techniques to allow the creation of novel structures in the style of real-world structures. Starting with a simple model recovered from a sparse image set, the model is divided into feature regions, such as doorways, windows, and brick. These feature regions essentially comprise a mapping from model space to image space, and can be recombined to texture a novel model. Procedural rules for the growth and reorganization of the model are automatically derived to allow for very fast editing and design. Further, the redundancies marked by the feature labeling can be used to perform automatic occlusion replacement and color equalization in the finished scene, which is rendered using view-dependent texture mapping on standard graphics hardware. Results using four captured scenes show that a great variety of novel structures can be created very quickly once a captured scene is available, and rendered with a degree of realism comparable to the original scene.

Image-based rendering (IBR) systems enable virtual walkthroughs of photorealistic environments by warping and combining reference images to novel viewpoints under interactive user control. A significant challenge in such systems is to automatically compute image correspondences that enable accurate image warping.In this paper, we describe a new algorithm for computing a globally consistent set of image feature correspondences across a wide range of viewpoints suitable for IBR walkthroughs. We first detect point features in a dense set of omnidirectional images captured on an eye-height plane. Then, we track these features from image to image, identifying potential correspondences when two features track to the same position in the same image. Among the potential correspondences, we select the maximal consistent set using a greedy graph-labeling algorithm.A key feature of our approach is that it exploits the multiple paths that can be followed between images in order to increase the number of feature correspondences between distant images. We demonstrate the benefits of this approach in a real-time IBR walkthrough system where novel images are reconstructed as the user moves interactively.

A long-standing research problem in computer graphics is to reproduce the visual experience of walking through a large photorealistic environment interactively. On one hand, traditional geometry-based rendering systems fall short of simulating the visual realism of a complex environment. On the other hand, image-based rendering systems have to date been unable to capture and store a sampled representation of a large environment with complex lighting and visibility effects. In this paper, we present a "sea of images," a practical approach to dense sampling, storage, and reconstruction of the plenoptic function in large, complex indoor environments. We use a motorized cart to capture omnidirectional images every few inches on a eye-height plane throughout an environment. The captured images are compressed and stored in a multiresolution hierarchy suitable for real-time prefetching during an interactive walkthrough. Later, novel images are reconstructed for a simulated observer by resampling nearby captured images. Our system acquires 15,254 images over 1,050 square feet at an average image spacing of 1.5 inches. The average capture and processing time is 7 hours. We demonstrate realistic walkthroughs of real-world environments reproducing specular reflections and occlusion effects while rendering 15-25 frames per second.

Interactive walkthrough applications require detailed 3D models to give users a sense of immersion in an environment. Traditionally these models are built using computer-aided design tools to define geometry and material properties. But creating detailed models is time-consuming and it is also difficult to reproduce all geometric and photometric subtleties of real-world scenes. Computer vision attempts to alleviate this problem by extracting geometry and photogrammetry from images of the real-world scenes. However, these models are still limited in the amount of detail they recover.

Multi-resolution hierarchies of polygons and more recently of points are familiar and useful tools for achieving interactive rendering rates. We present an algorithm for tightly integrating the two into a single hierarchical data structure. The trade-off between rendering portions of a model with points or with polygons is made automatically. Our approach to this problem is to apply a bottom-up simplification process involving not only polygon simplification operations, but point replacement and point simplification operations as well. Given one or more surface meshes, our algorithm produces a hybrid hierarchy comprising both polygon and point primitives. This hierarchy may be optimized according to the relative performance characteristics of these primitive types on the intended rendering platform. We also provide a range of aggressiveness for performing point replacement operations. The most conservative approach produces a hierarchy that is better than a purely polygonal hierarchy in some places, and roughly equal in others. A less conservative approach can trade reduced complexity at the far viewing ranges for some increased complexity at the near viewing ranges. We demonstrate our approach on a number of input models, achieving primitive counts that are 1.3 to 4.7 times smaller than those of triangle-only simplification.

We present a system for rendering very complex 3D models at interactive rates.We select a subset of the model as preferred viewpoints and partition the space into virtual cells.Each cell contains near geometry, rendered using levels of detail and visibility culling, and far geometry, rendered as a textured depth mesh.Our system automatically balances the screen-space errors resulting from geometric simplification with those from textureddepth-mesh distortion.We describe our prefetching and data management schemes, both crucial for models significantly larger than available system memory.We have successfully used our system to accelerate walkthroughs of a 13 million triangle model of a large coal-fired power plant and of a 1.7 million triangle architectural model.We demonstrate the walkthrough of a 1.3 GB power plant model with a 140 MB cache footprint.

We present a preprocessing algorithm and run-time system for rendering 3D geometric models at a guaranteed frame rate.Our approach trades off space for frame rate by using images to replace distant geometry.The preprocessing algorithm automatically chooses a subset of the model to display as an image so as to render no more than a specified number of geometric primitives.We also summarize an optimized layered-depth-image warper to display images surrounded by geometry at run time.Furthermore, we show the results of applying our method to accelerate the interactive walkthrough of several complex models.

This paper presents efficient image-based rendering techniques used in the context of an architectural walkthrough system. Portals (doors and windows) are rendered by warping layered depth images (LDIs). In a preprocessing phase, for every portal, a number of pre-rendered images are combined into an LDI. The resulting LDI stores, exactly once, all surfaces visible in at least one of the images used in the construction, so most of the exposure errors are efficiently eliminated. The LDI can be warped in the McMillan occlusion compatible ordering. A substantial increase in performance is obtained by warping in parallel. Our parallelization scheme achieves good load balancing, scales with the number of processors, and preserves the occlusion compatible ordering. A fast, conservative reference-image-space clipping algorithm also reduces the warping effort.

This paper outlines a method to dynamically replace portals with textures in a cell-partitioned model. The rendering complexity is reduced to the geometry of the current cell thus increasing interactive performance. A portal is a generalization of windows and doors. It connects two adjacent cells (or rooms). Each portal of the current cell that is some distance away from the viewpoint is rendered as a texture. The portal texture (smoothly) returns to geometry when the viewpoint gets close to the portal. This way all portal sequences (not too close to the viewpoint) have a depth complexity of one. The size of each texture and distance at which the transition occurs is configurable for each portal.

We are investigating methods for simplifying complex models for interactive visualizations using texture based representations. The paper presents a simplification method which dynamically "caches" distant geometry into textures and trades off accurate rendering of the distant geometry for performance. Smooth transitions and continuous borders are defined between the geometry and textures thus the representations can be switched without sudden jumps (as is the case with many current texturing techniques). All the computations for the transitions can be done a priori without the need to change the textures each frame thereafter.

We present an interactive modeling and animation system that facilitates the integration of a variety of simulation and animation paradigms. This system permits the modeling of diverse objects that change in shape, appearance, and behaviour over time. Our system thus extends modeling tools to include animation controls. Changes can be effected by various methods of control, including scripted, gestural, and behavioral specification. The system is an extensible testbed that supports research in the interaction of disparate control methods embodied in controller objects. This paper discusses some of the issues involved in modeling such interactions and the mechanisms implemented to provide solutions to some of these issues.The system's object-oriented architecture uses delegation hierarchies to let objects change all of their attributes dynamically. Objects include displayable objects, controllers, cameras, lights, renderers, and user interfaces. Techniques used to obtain interactive performance include the use of data-dependency networks, lazy evaluation, and extensive caching to exploit inter- and intra-frame coherency.