Abstract Realistic garment rendering requires simulating complex multi‐bounce light paths within intricate fold geometries. In these regions, conventional path tracing is computationally expensive as light becomes trapped, necessitating high bounce counts for convergence. We observe that these local inter‐reflections are highly localized and exhibit radiance patterns strongly correlated with local fold shapes. Based on these insights, we propose a neural local inter‐reflection model that factorizes light transport into overall intensity and directional distribution. By learning the relationship between incident light, material properties, and a novel fold shape descriptor, our model approximates multi‐bounce effects using a compact Spherical Harmonics representation. Our approach demonstrates strong generalization to unseen geometries and various fabric textures without retraining. Compared to full path tracing, our method significantly reduces rendering time while preserving high visual fidelity.

Abstract Hand‐drawn model sheets are widely used in character design to define 3D shape and appearance through sparse multi‐view drawings. Reconstructing 3D characters from such sparse inputs has traditionally been challenging due to insufficient visual information. Recently, 3D generative models have enabled automatic reconstruction of plausible 3D characters by learning from large‐scale training data, but achieving reconstructions that accurately match the input model sheets remains limited. In this paper, we present a framework that leverages the power of a 3D generative model for initial reconstruction and enhances the output to faithfully reproduce input model sheets. For such faithful reconstruction, we must address two fundamental challenges: (1) the hand‐drawn nature inherently introduces multi‐view inconsistencies where the generated 3D geometry cannot perfectly align with all views, and (2) view‐dependent line elements along geometry boundaries interfere with accurate texture reconstruction. To address these challenges, we optimize the geometry to minimize multi‐view inconsistencies and introduce a deformable per‐pixel camera ray representation that resolves residual discrepancies in cross‐view correspondences. We also decompose drawings into three distinct layers of view‐dependent lines, view‐independent colors, and fine‐detail decals to separately handle view‐dependent and view‐independent components for consistent cross‐view reconstruction. Comprehensive experiments demonstrate that our method outperforms possible alternatives regardless of the choice of 3D generative model, while successfully preserving both artistic intent and visual fidelity of input model sheets.

This paper presents a novel framework for compactly representing a 3D indoor scene using a set of polycuboids through a deep learning-based fitting method. Indoor scenes mainly consist of man-made objects, such as furniture, which often exhibit rectilinear geometry. This property allows indoor scenes to be represented using combinations of polycuboids, providing a compact representation that benefits downstream applications like furniture rearrangement. Our framework takes a noisy point cloud as input and first detects six types of cuboid faces using a transformer network. Then, a graph neural network is used to validate the spatial relationships of the detected faces to form potential polycuboids. Finally, each polycuboid instance is reconstructed by forming a set of boxes based on the aggregated face labels. To train our networks, we introduce a synthetic dataset encompassing a diverse range of cuboid and polycuboid shapes that reflect the characteristics of indoor scenes. Our framework generalizes well to real-world indoor scene datasets, including Replica, ScanNet, and scenes captured with an iPhone. The versatility of our method is demonstrated through practical applications, such as virtual room tours and scene editing.

In this paper, we present GyroDeblurNet, a novel single-image deblurring method that utilizes a gyro sensor to resolve the ill-posedness of image deblurring. The gyro sensor provides valuable information about camera motion that can improve deblurring quality. However, exploiting real-world gyro data is challenging due to errors from various sources. To handle these errors, GyroDeblurNet is equipped with two novel neural network blocks: a gyro refinement block and a gyro deblurring block. The gyro refinement block refines the erroneous gyro data using the blur information from the input image. The gyro deblurring block removes blur from the input image using the refined gyro data and further compensates for gyro error by leveraging the blur information from the input image. For training a neural network with erroneous gyro data, we propose a training strategy based on the curriculum learning. We also introduce a novel gyro data embedding scheme to represent real-world intricate camera shakes. Finally, we present both synthetic and real-world datasets for training and evaluating gyro-based single image deblurring. Our experiments demonstrate that our approach achieves state-of-the-art deblurring quality by effectively utilizing erroneous gyro data.

Abstract Non‐rigid iterative closest point (ICP) is a popular framework for shape alignment, typically formulated as alternating iteration of correspondence search and shape transformation. A common approach in the shape transformation stage is to solve a linear least squares problem to find a smoothness‐regularized transform that fits the target shape. However, completely solving the linear least squares problem to obtain a transform is wasteful because the correspondences used for constructing the problem are imperfect, especially at early iterations. In this work, we design a novel framework to compute a transform in single step without the exact linear solve. Our key idea is to use only a single step of an iterative linear system solver, conjugate gradient, at each shape transformation stage. For this single‐step scheme to be effective, appropriate preconditioning of the linear system is required. We design a novel adaptive Sobolev‐Jacobi preconditioning method for our single‐step transform to produce a large and regularized shape update suitable for correspondence search in the next iteration. We demonstrate that our preconditioned single‐step transform stably accelerates challenging 3D surface registration tasks.

Abstract Recent methods, such as 2D Gaussian Splatting and Gaussian Opacity Fields, have aimed to address the geometric inaccuracies of 3D Gaussian Splatting while retaining its superior rendering quality. However, these approaches still struggle to reconstruct smooth and reliable geometry, particularly in scenes with significant color variation across viewpoints, due to their per‐point appearance modeling and single‐view optimization constraints. In this paper, we propose an effective multiview geometric regularization strategy that integrates multiview stereo (MVS) depth, RGB, and normal constraints into Gaussian Splatting initialization and optimization. Our key insight is the complementary relationship between MVS‐derived depth points and Gaussian Splatting‐optimized positions: MVS robustly estimates geometry in regions of high color variation through local patch‐based matching and epipolar constraints, whereas Gaussian Splatting provides more reliable and less noisy depth estimates near object boundaries and regions with lower color variation. To leverage this insight, we introduce a median depth‐based multiview relative depth loss with uncertainty estimation, effectively integrating MVS depth information into Gaussian Splatting optimization. We also propose an MVS‐guided Gaussian Splatting initialization to avoid Gaussians falling into suboptimal positions. Extensive experiments validate that our approach successfully combines these strengths, enhancing both geometric accuracy and rendering quality across diverse indoor and outdoor scenes.

Self-intersection repair in static 3D surface meshes presents unique challenges due to the absence of temporal motion and penetration depth information—two critical elements typically leveraged in physics-based approaches. We introduce a novel framework that transforms local contact handling into a global repair strategy through a combination of local signed tangent-point energies and their gradient diffusion. At the heart of our method is a key insight: rather than computing expensive global repulsive potentials, we can effectively approximate long-range interactions by diffusing energy gradients from local contacts throughout the mesh surface. In turn, resolving complex self-intersections reduces to simply propagating local repulsive energies through standard diffusion mechanics and iteratively solving tractable local optimizations. We further accelerate convergence through our momentum-based optimizer, which adaptively regulates momentum based on gradient statistics to prevent overshooting while maintaining rapid intersection repair. The resulting algorithm handles a variety of challenging scenarios, from shallow contacts to deep penetrations, while providing computational efficiency suitable for interactive applications.

Non-rigid registration of 3D shape collections using a template mesh is essential for constructing 3D datasets. Traditional non-rigid Iterative Closest Point (ICP) methods rely on manually selected template meshes, which can result in inconsistent registrations when applied to diverse shape collections. This inconsistency arises particularly when the template lacks common shape features with the input instances or when landmark annotations are sparse. To overcome this limitation, we propose a novel ICP framework that jointly optimizes a shared template shape and its instance-wise deformations. Our joint optimization framework assigns distinct roles to the shared template and instance-wise deformations: the template captures common shape features, while instance-wise deformations handle residual registration errors. We use stronger smoothness regularization on the instance-wise deformations in early iterations to prioritize the accumulation of common details on the template. Additionally, a distortion alignment energy minimizes interinstance map distortions, promoting consistent instance-wise deformations. On challenging 3D datasets with large shape variations, our method achieves state-of-the-art fitting accuracy and consistent results in shape averaging and deformation transfer. By removing the need for a carefully selected preset template, our method extends the capability of extrinsic non-rigid registration frameworks, offering a more robust and flexible solution for challenging registration scenarios.

Photometric stereo leverages variations in illumination conditions to reconstruct surface normals. Display photo-metric stereo, which employs a conventional monitor as an illumination source, has the potential to overcome limitations often encountered in bulky and difficult-to-use conventional setups. In this paper, we present differentiable display photometric stereo (DDPS), addressing an often overlooked challenge in display photometric stereo: the design of display patterns. Departing from using heuristic display patterns, DDPS learns the display patterns that yield accurate normal reconstruction for a target system in an end-to-end manner. To this end, we propose a differentiable framework that couples basis-illumination image formation with analytic photometric-stereo reconstruction. The differentiable framework facilitates the effective learning of display patterns via auto-differentiation. Also, for training supervision, we propose to use 3D printing for creating a real-world training dataset, enabling accurate reconstruction on the target real-world setup. Finally, we exploit that conventional LCD monitors emit polarized light, which allows for the optical separation of diffuse and specular reflections when combined with a polarization camera, leading to accurate normal reconstruction. Extensive evaluation of DDPS shows improved normal-reconstruction accuracy compared to heuristic patterns and demonstrates compelling properties such as robustness to pattern initialization, calibration errors, and simplifications in image for-mation and reconstruction.

Many surface reconstruction methods incorporate normal integration, which is a process to obtain a depth map from surface gradients. In this process, the input may represent a surface with discontinuities, e.g., due to self-occlusion. To reconstruct an accurate depth map from the input normal map, hidden surface gradients occurring from the jumps must be handled. To model these jumps correctly, we design a novel discretization scheme for the domain of normal integration. Our key idea is to introduce auxiliary edges, which bridge between piecewise-smooth patches in the domain so that the magnitude of hidden jumps can be explicitly expressed. Using the auxiliary edges, we design a novel algorithm to optimize the discontinuity and the depth map from the input normal map. Our method optimizes dis-continuities by using a combination of iterative re-weighted least squares and iterative filtering of the jump magnitudes on auxiliary edges to provide strong sparsity regularization. Compared to previous discontinuity-preserving normal integration methods, which model the magnitudes of jumps only implicitly, our method reconstructs subtle disconti-nuities accurately thanks to our explicit representation of jumps allowing for strong sparsity regularization.

RAW images are rarely shared mainly due to its exces-sive data size compared to their sRGB counterparts ob-tained by camera ISPs. Learning the forward and inverse processes of camera ISPs has been recently demonstrated, enabling physically-meaningful RAW-level image processing on input sRGB images. However, existing learning-based ISP methods fail to handle the large variations in the ISP processes with respect to camera parameters such as ISO and exposure time, and have limitations when used for various applications. In this paper, we propose ParamISP, a learning-based method for forward and inverse con-version between sRGB and RAW images, that adopts a novel neural-network module to utilize camera parameters, which is dubbed as ParamNet. Given the camera param-eters provided in the EXIF data, ParamNet converts them into a feature vector to control the ISP networks. Extensive experiments demonstrate that ParamISP achieve superior RAW and sRGB reconstruction results compared to previous methods and it can be effectively used for a variety of applications such as deblurring dataset synthesis, raw deblur-ring, HDR reconstruction, and camera-to-camera transfer.

We propose the first reference-based video super-resolution (RefVSR) approach that utilizes reference videos for high-fidelity results. We focus on RefVSR in a triple-camera setting, where we aim at super-resolving a low-resolution ultra-wide video utilizing wide-angle and tele-photo videos. We introduce the first RefVSR network that re-currently aligns and propagates temporal reference features fused with features extracted from low-resolution frames. To facilitate the fusion and propagation of temporal reference features, we propose a propagative temporal fusion module. For learning and evaluation of our network, we present the first RefVSR dataset consisting of triplets of ultra-wide, wide-angle, and telephoto videos concurrently taken from triple cameras of a smartphone. We also propose a two-stage training strategy fully utilizing video triplets in the proposed dataset for real-world 4 × video super-resolution. We extensively evaluate our method, and the result shows the state-of-the-art performance in 4 × super-resolution.

Abstract While motion compensation greatly improves video deblurring quality, separately performing motion compensation and video deblurring demands huge computational overhead. This paper proposes a real‐time video deblurring framework consisting of a lightweight multi‐task unit that supports both video deblurring and motion compensation in an efficient way. The multi‐task unit is specifically designed to handle large portions of the two tasks using a single shared network and consists of a multi‐task detail network and simple networks for deblurring and motion compensation. The multi‐task unit minimizes the cost of incorporating motion compensation into video deblurring and enables real‐time deblurring. Moreover, by stacking multiple multi‐task units, our framework provides flexible control between the cost and deblurring quality. We experimentally validate the state‐of‐the‐art deblurring quality of our approach, which runs at a much faster speed compared to previous methods and show practical real‐time performance (30.99dB@30fps measured on the DVD dataset).

Recently, there has been an upsurge of activity in image-based non-photorealistic rendering (NPR), and in particular portrait image stylisation, due to the advent of neural style transfer (NST). However, the state of performance evaluation in this field is poor, especially compared to the norms in the computer vision and machine learning communities. Unfortunately, the task of evaluating image stylisation is thus far not well defined, since it involves subjective, perceptual, and aesthetic aspects. To make progress towards a solution, this paper proposes a new structured, three-level, benchmark dataset for the evaluation of stylised portrait images. Rigorous criteria were used for its construction, and its consistency was validated by user studies. Moreover, a new methodology has been developed for evaluating portrait stylisation algorithms, which makes use of the different benchmark levels as well as annotations provided by user studies regarding the characteristics of the faces. We perform evaluation for a wide variety of image stylisation methods (both portrait-specific and general purpose, and also both traditional NPR approaches and NST) using the new benchmark dataset.

Three-dimensional (3D) reconstruction using an RGB-D camera has been widely adopted for realistic content creation. However, high-quality texture mapping onto the reconstructed geometry is often treated as an offline step that should run after geometric reconstruction. In this article, we propose TextureMe , a novel approach that jointly recovers 3D surface geometry and high-quality texture in real time. The key idea is to create triangular texture patches that correspond to zero-crossing triangles of truncated signed distance function (TSDF) progressively in a global texture atlas. Our approach integrates color details into the texture patches in parallel with the depth map integration to a TSDF. It also actively updates a pool of texture patches to adapt TSDF changes and minimizes misalignment artifacts that occur due to camera drift and image distortion. Our global texture atlas representation is fully compatible with conventional texture mapping. As a result, our approach produces high-quality textures without utilizing additional texture map optimization, mesh parameterization, or heavy post-processing. High-quality scenes produced by our real-time approach are even comparable to the results from state-of-the-art methods that run offline.

We propose LaplacianFusion , a novel approach that reconstructs detailed and controllable 3D clothed-human body shapes from an input depth or 3D point cloud sequence. The key idea of our approach is to use Laplacian coordinates, well-known differential coordinates that have been used for mesh editing, for representing the local structures contained in the input scans, instead of implicit 3D functions or vertex displacements used previously. Our approach reconstructs a controllable base mesh using SMPL, and learns a surface function that predicts Laplacian coordinates representing surface details on the base mesh. For a given pose, we first build and subdivide a base mesh, which is a deformed SMPL template, and then estimate Laplacian coordinates for the mesh vertices using the surface function. The final reconstruction for the pose is obtained by integrating the estimated Laplacian coordinates as a whole. Experimental results show that our approach based on Laplacian coordinates successfully reconstructs more visually pleasing shape details than previous methods. The approach also enables various surface detail manipulations, such as detail transfer and enhancement.

We propose a novel end-to-end learning-based approach for single image defocus deblurring. The proposed approach is equipped with a novel Iterative Filter Adaptive Network (IFAN) that is specifically designed to handle spatially-varying and large defocus blur. For adaptively handling spatially-varying blur, IFAN predicts pixel-wise deblurring filters, which are applied to defocused features of an input image to generate deblurred features. For effectively managing large blur, IFAN models deblurring filters as stacks of small-sized separable filters. Predicted separable deblurring filters are applied to defocused features using a novel Iterative Adaptive Convolution (IAC) layer. We also propose a training scheme based on defocus disparity estimation and reblurring, which significantly boosts the de-blurring quality. We demonstrate that our method achieves state-of-the-art performance both quantitatively and qualitatively on real-world images.

We propose deep virtual markers, a framework for estimating dense and accurate positional information for various types of 3D data. We design a concept and construct a framework that maps 3D points of 3D articulated models, like humans, into virtual marker labels. To realize the framework, we adopt a sparse convolutional neural network and classify 3D points of an articulated model into virtual marker labels. We propose to use soft labels for the classifier to learn rich and dense interclass relationships based on geodesic distance. To measure the localization accuracy of the virtual markers, we test FAUST challenge, and our result outperforms the state-of-the-art. We also observe outstanding performance on the generalizability test, unseen data evaluation, and different 3D data types (meshes and depth maps). We show additional applications using the estimated virtual markers, such as non-rigid registration, texture transfer, and realtime dense marker prediction from depth maps.

This paper proposes a novel deep learning approach for single image defocus deblurring based on inverse kernels. In a defocused image, the blur shapes are similar among pixels although the blur sizes can spatially vary. To utilize the property with inverse kernels, we exploit the observation that when only the size of a defocus blur changes while keeping the shape, the shape of the corresponding inverse kernel remains the same and only the scale changes. Based on the observation, we propose a kernel-sharing parallel atrous convolutional (KPAC) block specifically designed by incorporating the property of inverse kernels for single image defocus deblurring. To effectively simulate the invariant shapes of inverse kernels with different scales, KPAC shares the same convolutional weights among multiple atrous convolution layers. To efficiently simulate the varying scales of inverse kernels, KPAC consists of only a few atrous convolution layers with different dilations and learns per-pixel scale attentions to aggregate the outputs of the layers. KPAC also utilizes the shape attention to combine the outputs of multiple convolution filters in each atrous convolution layer, to deal with defocus blur with a slightly varying shape. We demonstrate that our approach achieves state-of-the-art performance with a much smaller number of parameters than previous methods.

Abstract This paper presents an effective method for generating a spatiotemporal (time‐varying) texture map for a dynamic object using a single RGB‐D camera. The input of our framework is a 3D template model and an RGB‐D image sequence. Since there are invisible areas of the object at a frame in a single‐camera setup, textures of such areas need to be borrowed from other frames. We formulate the problem as an MRF optimization and define cost functions to reconstruct a plausible spatiotemporal texture for a dynamic object. Experimental results demonstrate that our spatiotemporal textures can reproduce the active appearances of captured objects better than approaches using a single texture map.

We present a caricature generation framework based on shape and style manipulation using StyleGAN. Our framework, dubbed StyleCariGAN , automatically creates a realistic and detailed caricature from an input photo with optional controls on shape exaggeration degree and color stylization type. The key component of our method is shape exaggeration blocks that are used for modulating coarse layer feature maps of StyleGAN to produce desirable caricature shape exaggerations. We first build a layer-mixed StyleGAN for photo-to-caricature style conversion by swapping fine layers of the StyleGAN for photos to the corresponding layers of the StyleGAN trained to generate caricatures. Given an input photo, the layer-mixed model produces detailed color stylization for a caricature but without shape exaggerations. We then append shape exaggeration blocks to the coarse layers of the layer-mixed model and train the blocks to create shape exaggerations while preserving the characteristic appearances of the input. Experimental results show that our StyleCariGAN generates realistic and detailed caricatures compared to the current state-of-the-art methods. We demonstrate StyleCariGAN also supports other StyleGAN-based image manipulations, such as facial expression control.

For the success of video deblurring, it is essential to utilize information from neighboring frames. Most state-of-the-art video deblurring methods adopt motion compensation between video frames to aggregate information from multiple frames that can help deblur a target frame. However, the motion compensation methods adopted by previous deblurring methods are not blur-invariant, and consequently, their accuracy is limited for blurry frames with different blur amounts. To alleviate this problem, we propose two novel approaches to deblur videos by effectively aggregating information from multiple video frames. First, we present blur-invariant motion estimation learning to improve motion estimation accuracy between blurry frames. Second, for motion compensation, instead of aligning frames by warping with estimated motions, we use a pixel volume that contains candidate sharp pixels to resolve motion estimation errors. We combine these two processes to propose an effective recurrent video deblurring network that fully exploits deblurred previous frames. Experiments show that our method achieves the state-of-the-art performance both quantitatively and qualitatively compared to recent methods that use deep learning.

In this paper, we propose the first end-to-end convolutional neural network (CNN) architecture, Defocus Map Estimation Network (DMENet), for spatially varying defocus map estimation. To train the network, we produce a novel depth-of-field (DOF) dataset, SYNDOF, where each image is synthetically blurred with a ground-truth depth map. Due to the synthetic nature of SYNDOF, the feature characteristics of images in SYNDOF can differ from those of real defocused photos. To address this gap, we use domain adaptation that transfers the features of real defocused photos into those of synthetically blurred ones. Our DMENet consists of four subnetworks: blur estimation, domain adaptation, content preservation, and sharpness calibration networks. The subnetworks are connected to each other and jointly trained with their corresponding supervisions in an end-to-end manner. Our method is evaluated on publicly available blur detection and blur estimation datasets and the results show the state-of-the-art performance.In this paper, we propose the first end-to-end convolutional neural network (CNN) architecture, Defocus Map Estimation Network (DMENet), for spatially varying defocus map estimation. To train the network, we produce a novel depth-of-field (DOF) dataset, SYNDOF, where each image is synthetically blurred with a ground-truth depth map. Due to the synthetic nature of SYNDOF, the feature characteristics of images in SYNDOF can differ from those of real defocused photos. To address this gap, we use domain adaptation that transfers the features of real defocused photos into those of synthetically blurred ones. Our DMENet consists of four subnetworks: blur estimation, domain adaptation, content preservation, and sharpness calibration networks. The subnetworks are connected to each other and jointly trained with their corresponding supervisions in an end-to-end manner. Our method is evaluated on publicly available blur detection and blur estimation datasets and the results show the state-of-the-art performance.

This paper improves the accuracy of heatmap-based 3D face alignment neural networks. Many current approaches in face alignment are limited by two major problems, quantization and the lack of regularization of heatmaps. The first limitation is caused by the non-differentiable argmax function, which extracts landmark coordinates from heatmaps as integer indices. Heatmaps are generated at low-resolution to reduce the memory and computational costs, which results in heatmaps far lower than the input image’s resolution. We propose a heatmap generator network producing floating-point precision heatmaps that are scalable to higher-resolutions. To resolve the second limitation, we propose a novel deformation constraint on heatmaps. The constraint is based on graph-Laplacian and enables a heatmap generator to regularize overall shape of the output face landmarks using the global face structure. By eliminating quantization and including regularization, our method can vastly improve landmark localization accuracy, and achieves the state-of-the-art performance without adding complex network structures.

Abstract We propose a novel framework to generate a global texture atlas for a deforming geometry. Our approach distinguishes from prior arts in two aspects. First, instead of generating a texture map for each timestamp to color a dynamic scene, our framework reconstructs a global texture atlas that can be consistently mapped to a deforming object. Second, our approach is based on a single RGB‐D camera, without the need of a multiple‐camera setup surrounding a scene. In our framework, the input is a 3D template model with an RGB‐D image sequence, and geometric warping fields are found using a state‐of‐the‐art non‐rigid registration method [GXW*15] to align the template mesh to noisy and incomplete input depth images. With these warping fields, our multi‐scale approach for texture coordinate optimization generates a sharp and clear texture atlas that is consistent with multiple color observations over time. Our approach is accelerated by graphical hardware and provides a handy configuration to capture a dynamic geometry along with a clean texture atlas. We demonstrate our approach with practical scenarios, particularly human performance capture. We also show that our approach is resilient on misalignment issues caused by imperfect estimation of warping fields and inaccurate camera parameters.

Abstract This paper proposes a deep learning‐based image tone enhancement approach that can maximally enhance the tone of an image while preserving the naturalness. Our approach does not require carefully generated ground‐truth images by human experts for training. Instead, we train a deep neural network to mimic the behavior of a previous classical filtering method that produces drastic but possibly unnatural‐looking tone enhancement results. To preserve the naturalness, we adopt the generative adversarial network (GAN) framework as a regularizer for the naturalness. To suppress artifacts caused by the generative nature of the GAN framework, we also propose an imbalanced cycle‐consistency loss. Experimental results show that our approach can effectively enhance the tone and contrast of an image while preserving the naturalness compared to previous state‐of‐the‐art approaches.

Abstract Semantic segmentation partitions a given image or 3D model of a scene into semantically meaning parts and assigns predetermined labels to the parts. With well‐established datasets, deep networks have been successfully used for semantic segmentation of RGB and RGB‐D images. On the other hand, due to the lack of annotated large‐scale 3D datasets, semantic segmentation for 3D scenes has not yet been much addressed with deep learning. In this paper, we present a novel framework for generating semantically segmented triangular meshes of reconstructed 3D indoor scenes using volumetric semantic fusion in the reconstruction process. Our method integrates the results of CNN‐based 2D semantic segmentation that is applied to the RGB‐D stream used for dense surface reconstruction. To reduce the artifacts from noise and uncertainty of single‐view semantic segmentation, we introduce adaptive integration for the volumetric semantic fusion and CRF‐based semantic label regularization. With these methods, our framework can easily generate a high‐quality triangular mesh of the reconstructed 3D scene with dense (i.e., per‐vertex) semantic labels. Extensive experiments demonstrate that our semantic segmentation results of 3D scenes achieves the state‐of‐the‐art performance compared to the previous voxel‐based and point cloud‐based methods.

Abstract We propose a novel approach for detecting two kinds of partial blur, defocus and motion blur, by training a deep convolutional neural network. Existing blur detection methods concentrate on designing low‐level features, but those features have difficulty in detecting blur in homogeneous regions without enough textures or edges. To handle such regions, we propose a deep encoder‐decoder network with long residual skip‐connections and multi‐scale reconstruction loss functions to exploit high‐level contextual features as well as low‐level structural features. Another difficulty in partial blur detection is that there are no available datasets with images having both defocus and motion blur together, as most existing approaches concentrate only on either defocus or motion blur. To resolve this issue, we construct a synthetic dataset that consists of complex scenes with both types of blur. Experimental results show that our approach effectively detects and classifies blur, outperforming other state‐of‐the‐art methods. Our method can be used for various applications, such as photo editing, blur magnification, and deblurring.

This paper proposes a novel framework for non-blind de-convolution using deep convolutional network. To deal with various blur kernels, we reduce the training complexity using Wiener filter as a preprocessing step in our framework. This step generates amplified noise and ringing artifacts, but the artifacts are little correlated with the shapes of blur kernels, making the input of our network independent of the blur kernel shape. Our network is trained to effectively remove those artifacts via a residual network with long/short skip-connections. We also add a regularization to help our network robustly process untrained and inaccurate blur kernels by suppressing abnormal weights of convolutional layers that may incur overfitting. Our postprocessing step can further improve the deconvolution quality. Experimental results demonstrate that our framework can process images blurred by a variety of blur kernels with faster speed and comparable image quality to the state-of-the-art methods.

One popular approach for blind deconvolution is to formulate a maximum a posteriori (MAP) problem with sparsity priors on the gradients of the latent image, and then alternatingly estimate the blur kernel and the latent image. While several successful MAP based methods have been proposed, there has been much controversy and confusion about their convergence, because sparsity priors have been shown to prefer blurry images to sharp natural images. In this paper, we revisit this problem and provide an analysis on the convergence of MAP based approaches. We first introduce a slight modification to a conventional joint energy function for blind deconvolution. The reformulated energy function yields the same alternating estimation process, but more clearly reveals how blind deconvolution works. We then show the energy function can actually favor the right solution instead of the no-blur solution under certain conditions, which explains the success of previous MAP based approaches. The reformulated energy function and our conditions for the convergence also provide a way to compare the qualities of different blur kernels, and we demonstrate its applicability to automatic blur kernel size selection, blur kernel estimation using light streaks, and defocus estimation.

In multi-class indoor semantic segmentation using RGB-D data, it has been shown that incorporating depth feature into RGB feature is helpful to improve segmentation accuracy. However, previous studies have not fully exploited the potentials of multi-modal feature fusion, e.g., simply concatenating RGB and depth features or averaging RGB and depth score maps. To learn the optimal fusion of multimodal features, this paper presents a novel network that extends the core idea of residual learning to RGB-D semantic segmentation. Our network effectively captures multilevel RGB-D CNN features by including multi-modal feature fusion blocks and multi-level feature refinement blocks. Feature fusion blocks learn residual RGB and depth features and their combinations to fully exploit the complementary characteristics of RGB and depth data. Feature refinement blocks learn the combination of fused features from multiple levels to enable high-resolution prediction. Our network can efficiently train discriminative multi-level features from each modality end-to-end by taking full advantage of skip-connections. Our comprehensive experiments demonstrate that the proposed architecture achieves the state-of-the-art accuracy on two challenging RGB-D indoor datasets, NYUDv2 and SUN RGB-D.

With the recent advent of 360 cameras, spherical panorama images are becoming more popular and widely available. In a spherical panorama, alignment of the scene orientation to the image axes is important for providing comfortable and pleasant viewing experiences using VR headsets and traditional displays. This paper presents an automatic framework for upright adjustment of 360 spherical panorama images without any prior information, such as depths and Gyro sensor data. We take the Atlanta world assumption and use the horizontal and vertical lines in the scene to formulate a cost function for upright adjustment. Our method produces visually pleasing results for a variety of real-world spherical panoramas in less than a second.

Abstract This paper presents a novel filtering‐based method for decomposing an image into structures and textures. Unlike previous filtering algorithms, our method adaptively smooths image gradients to filter out textures from images. A new gradient operator, the interval gradient, is proposed for adaptive gradient smoothing. Using interval gradients, textures can be distinguished from structure edges and smoothly varying shadings. We also propose an effective gradient‐guided algorithm to produce high‐quality image filtering results from filtered gradients. Our method avoids gradient reversal in the filtering results and preserves sharp features better than existing filtering approaches, while retaining simplicity and highly parallel implementation. The proposed method can be utilized for various applications that require accurate structure‐texture decomposition of images.

We propose an automatic framework for quality assessment of a photograph as well as analysis of its aesthetic attributes. In contrast to the previous methods that rely on manually designed features to account for photo aesthetics, our method automatically extracts such features using a pretrained deep convolutional neural network (DCNN). To make the DCNN-extracted features more suited to our target tasks of photo quality assessment and aesthetic attribute analysis, we propose a novel feature encoding scheme, which supports vector machines-driven sparse restricted Boltzmann machines, which enhances sparseness of features and discrimination between target classes. Experimental results show that our method outperforms the current state-of-the-art methods in automatic photo quality assessment, and gives aesthetic attribute ratings that can be used for photo editing. We demonstrate that our feature encoding scheme can also be applied to general object classification task to achieve performance gains.

This paper presents an image-based rendering (IBR) system based on RGB-D images. The input of our system consists of RGB-D images captured at sparse locations in the scene and can be expanded by adding new RGB-D images. The sparsity of RGB-D images increases the usability of our system as the user need not capture a RGB-D image stream in a single shot, which may require careful planning for a hand-held camera. Our system begins with a single RGB-D image and images are incrementally added one by one. For each newly added image, a batch process is performed to align it with previously added images. The process does not include a global alignment step, such as bundle adjustment, and can be completed quickly by computing only local alignments of RGB-D images. Aligned images are represented as a graph, where each node is an input image and an edge contains relative pose information between nodes. A novel view image is rendered by picking the nearest input as the reference image and then blending the neighboring images based on depth information in real time. Experimental results with indoor scenes using Microsoft Kinect demonstrate that our system can synthesize high quality novel view images from a sparse set of RGB-D images.

Abstract This paper presents a novel method to enhance the performance of structure‐preserving image and texture filtering. With conventional edge‐aware filters, it is often challenging to handle images of high complexity where features of multiple scales coexist. In particular, it is not always easy to find the right balance between removing unimportant details and protecting important features when they come in multiple sizes, shapes, and contrasts. Unlike previous approaches, we address this issue from the perspective of adaptive kernel scales. Relying on patch‐based statistics, our method identifies texture from structure and also finds an optimal per‐pixel smoothing scale. We show that the proposed mechanism helps achieve enhanced image/texture filtering performance in terms of protecting the prominent geometric structures in the image, such as edges and corners, and keeping them sharp even after significant smoothing of the original signal.

Abstract Typical high dynamic range (HDR) imaging approaches based on multiple images have difficulties in handling moving objects and camera shakes, suffering from the ghosting effect and the loss of sharpness in the output HDR image. While there exist a variety of solutions for resolving such limitations, most of the existing algorithms are susceptible to complex motions, saturation, and occlusions. In this paper, we propose an HDR imaging approach using the coded electronic shutter which can capture a scene with row‐wise varying exposures in a single image. Our approach enables a direct extension of the dynamic range of the captured image without using multiple images, by photometrically calibrating rows with different exposures. Due to the concurrent capture of multiple exposures, misalignments of moving objects are naturally avoided with significant reduction in the ghosting effect. To handle the issues with under‐/over‐exposure, noise, and blurs, we present a coherent HDR imaging process where the problems are resolved one by one at each step. Experimental results with real photographs, captured using a coded electronic shutter, demonstrate that our method produces a high quality HDR images without the ghosting and blur artifacts.

Abstract We present a novel method for enhancing details in a digital photograph, inspired by the principle of art photography. In contrast to the previous methods that primarily rely on tone scaling, our technique provides a flexible tone transform model that consists of two operators: shifting and scaling. This model permits shifting of the tonal range in each image region to enable significant detail boosting regardless of the original tone. We optimize these shift and scale factors in our constrained optimization framework to achieve extreme detail enhancement across the image in a piecewise smooth fashion, as in art photography. The experimental results show that the proposed method brings out a significantly large amount of details even from an ordinary low‐dynamic range image.

Man-made structures often appear to be distorted in photos captured by casual photographers, as the scene layout often conflicts with how it is expected by human perception. In this paper, we propose an automatic approach for straightening up slanted man-made structures in an input image to improve its perceptual quality. We call this type of correction upright adjustment. We propose a set of criteria for upright adjustment based on human perception studies, and develop an optimization framework which yields an optimal homography for adjustment. We also develop a new optimization-based camera calibration method that performs favorably to previous methods and allows the proposed system to work reliably for a wide range of images. The effectiveness of our system is demonstrated by both quantitative comparisons and qualitative user study.

This paper presents a novel structure-preserving image decomposition operator called bilateral texture filter . As a simple modification of the original bilateral filter [Tomasi and Manduchi 1998], it performs local patch-based analysis of texture features and incorporates its results into the range filter kernel. The central idea to ensure proper texture/structure separation is based on patch shift that captures the texture information from the most representative texture patch clear of prominent structure edges. Our method outperforms the original bilateral filter in removing texture while preserving main image structures, at the cost of some added computation. It inherits well-known advantages of the bilateral filter, such as simplicity, local nature, ease of implementation, scalability, and adaptability to other application scenarios.

Despite increasing popularity of stereo capture and display systems, creating stereo artwork remains a challenge. This paper presents a stereo painting system, which enables effective from-scratch creation of high-quality stereo artwork. A key concept of our system is a stereo layer, which is composed of two RGBAd (RGBA + depth) buffers. Stereo layers alleviate the need for fully formed representational 3D geometry required by most existing 3D painting systems, and allow for simple, essential depth specification. RGBAd buffers also provide scalability for complex scenes by minimizing the dependency of stereo painting updates on the scene complexity. For interaction with stereo layers, we present stereo paint and stereo depth brushes, which manipulate the photometric (RGBA) and depth buffers of a stereo layer, respectively. In our system, painting and depth manipulation operations can be performed in arbitrary order with real-time visual feedback, providing a flexible WYSIWYG workflow for stereo painting. Our data structures allow for easy interoperability with existing image and geometry data, enabling a number of applications beyond from-scratch art creation, such as stereo conversion of monoscopic artwork and mixed-media art. Comments from artists and experimental results demonstrate that our system effectively aides in the creation of compelling stereo paintings.

Stereoscopic 3D exploits the effects of stereopsis where the depth perception is triggered by binocular disparity, a difference in image location of an object by the left and right eyes. Despite the dissimilarity of stereo projections in terms of disparity and shape, the human visual system can find the matching stereo pair to fuse using their similarities in terms of color, size, shading, texture, shadows, that are normally present in photorealistic stereo imaging. Now if some of these elements (or depth cues) are missing, such as in non-photorealistic stereo imaging, how will it affect the depth perception and stereo fusion? In this paper, we investigate this issue by conducting a perceptual study on stereoscopic 3D line drawing, in which many of these cues have been abstracted away. We first evaluate the validity of using stereo images composed of lines only, and then compare its performance to stereo images of normal shading. We also examine the effect of changing line style as well as the prospect of using lines as an additional depth cue. Our study shows that stereo line drawing, when compared to stereo shading, does weaken the depth perception but only to a minor degree. On the other hand, modification of line style, and superimposition of stereo lines both have potentials to strengthen the perception of depth.

Despite increasing popularity of stereo capture and display systems, creating stereo artwork remains a challenge. This paper presents a stereo painting system, which enables effective from-scratch creation of high-quality stereo artwork. A key concept of our system is a stereo layer, which is composed of two RGBAd (RGBA + depth) buffers. Stereo layers alleviate the need for fully formed representational 3D geometry required by most existing 3D painting systems, and allow for simple, essential depth specification. RGBAd buffers also provide scalability for complex scenes by minimizing the dependency of stereo painting updates on the scene complexity. For interaction with stereo layers, we present stereo paint and stereo depth brushes, which manipulate the photometric (RGBA) and depth buffers of a stereo layer, respectively. In our system, painting and depth manipulation operations can be performed in arbitrary order with real-time visual feedback, providing a flexible WYSIWYG workflow for stereo painting. Comments from artists and experimental results demonstrate that our system effectively aides in the creation of compelling stereo paintings.

Abstract We propose various simulation strategies to generate single‐frame fire effects for images, as opposed to multi‐frame fire effects for animations. To accelerate 3D simulation and to provide a user with early hints on the final effect, we propose a 2D‐guided 3D simulation approach, which runs a faster 2D simulation first, and then guides 3D simulation using the 2D simulation result. To achieve this, we explore various boundary conditions and develop a constrained projection method. Since only the final frame will be used while intermediate frames are abandoned, earlier intermediate frames can take larger time steps and have large noise applied, quickly generating turbulent flow structures. As the final frame approaches, we increase the flow quality by reducing the time step and not adding any noise. This adaptive time stepping allows us to use more computational resource near or at the final frame. We also develop divergence and buoyancy modification methods to guide flames along arbitrary, even physically implausible, directions. Our simulation methods can effectively and efficiently generate a variety of fire effects useful for image decoration.

This paper discusses stereoscopic 3D imaging based on line drawing of 3D shapes. We describe the major issues and challenges in generating stereoscopic 3D effects using lines only, with a couple of relatively simple approaches called each-eye-based and center-eye-based. Each of these methods has its shortcomings, such as binocular rivalry and inaccurate lines. We explain why and how these problems occur, then describe the concept of stereo-coherent lines and an algorithm to extract them from 3D shapes. We also propose a simple method to stylize stereo lines that ensures the stereo coherence of stroke textures across binocular views. The proposed method provides viewers with unique visual experience of watching 2D drawings popping out of the screen like 3D.

Man-made structures often appear to be distorted in photos captured by casual photographers, as the scene layout often conflicts with how it is expected by human perception. In this paper we propose an automatic approach for straightening up slanted man-made structures in an input image to improve its perceptual quality. We call this type of correction upright adjustment. We propose a set of criteria for upright adjustment based on human perception studies, and develop an optimization framework which yields an optimal homography for adjustment. We also develop a new optimization-based camera calibration method that performs favorably to previous methods and allows the proposed system to work reliably for a wide variety of images. The effectiveness of our system is demonstrated by both quantitative comparisons and qualitative user studies.

Most automotive companies are facing key challenges of improving quality and reducing time to market with limited resources. One of the methods for achieving key challenges is virtual simulation test in early development phase. The methodology of creating a model-driven distributed system with communication database as well as automatically generating models from specifications is proposed. In order to detect faults of functions and inconsistency of communication signals between specifications and vehicle network, the manual and automated test methods which can verify a model-driven distributed system is represented. This was applied successfully in the production project. The usefulness of the developed methodology and the tool is well ensured through the discovery of defects.

Abstract This paper proposes an algorithm which uses image registration to estimate a non‐uniform motion blur point spread function (PSF) caused by camera shake. Our study is based on a motion blur model which models blur effects of camera shakes using a set of planar perspective projections (i.e., homographies). This representation can fully describe motions of camera shakes in 3D which cause non‐uniform motion blurs. We transform the non‐uniform PSF estimation problem into a set of image registration problems which estimate homographies of the motion blur model one‐by‐one through the Lucas‐Kanade algorithm. We demonstrate the performance of our algorithm using both synthetic and real world examples. We also discuss the effectiveness and limitations of our algorithm for non‐uniform deblurring.

Videos captured by hand-held Cameras often contain significant camera shake, causing many frames to be blurry. Restoring shaky videos not only requires smoothing the camera motion and stabilizing the content, but also demands removing blur from video frames. However, video blur is hard to remove using existing single or multiple image deblurring techniques, as the blur kernel is both spatially and temporally varying. This paper presents a video deblurring method that can effectively restore sharp frames from blurry ones caused by camera shake. Our method is built upon the observation that due to the nature of camera shake, not all video frames are equally blurry. The same object may appear sharp on some frames while blurry on others. Our method detects sharp regions in the video, and uses them to restore blurry regions of the same content in nearby frames. Our method also ensures that the deblurred frames are both spatially and temporally coherent using patch-based synthesis. Experimental results show that our method can effectively remove complex video blur under the presence of moving objects and other outliers, which cannot be achieved using previous deconvolution-based approaches.

Non-blind deconvolution is a key component in image deblurring systems. Previous deconvolution methods assume a linear blur model where the blurred image is generated by a linear convolution of the latent image and the blur kernel. This assumption often does not hold in practice due to various types of outliers in the imaging process. Without proper outlier handling, previous methods may generate results with severe ringing artifacts even when the kernel is estimated accurately. In this paper we analyze a few common types of outliers that cause previous methods to fail, such as pixel saturation and non-Gaussian noise. We propose a novel blur model that explicitly takes these outliers into account, and build a robust non-blind deconvolution method upon it, which can effectively reduce the visual artifacts caused by outliers. The effectiveness of our method is demonstrated by experimental results on both synthetic and real-world examples.

This paper presents a compression scheme for mesh geometry, which is suitable for mobile graphics. The main focus is to enable real-time decoding of compressed vertex positions while providing reasonable compression ratios. Our scheme is based on local quantization of vertex positions with mesh partitioning. To prevent visual seams along the partitioning boundaries, we constrain the locally quantized cells of all mesh partitions to have the same size and aligned local axes. We propose a mesh partitioning algorithm to minimize the size of locally quantized cells, which relates to the distortion of a restored mesh. Vertex coordinates are stored in main memory and transmitted to graphics hardware for rendering in the quantized form, saving memory space and system bus bandwidth. Decoding operation is combined with model geometry transformation, and the only overhead to restore vertex positions is one matrix multiplication for each mesh partition.

We present a novel method for decomposing an image into base and texture layers. Our method is simple and effective, and can handle textures of high contrast, which traditional image filtering techniques may not handle efficiently. The method first removes high-frequency texture information using low-pass filtering, and then restores structural information of the image using a deconvolution operation. Experimental results demonstrate the effectiveness of our method.

We propose a novel technique for extracting a series of displaced subdivision surfaces sharing the same topology and the same displacement map from a given animated mesh. Our motion-based mesh simplification method creates control meshes with a small number of vertices but keeps the motion information. Extracted control meshes are simpler than the original meshes, and so easier to edit and take less storage. Our method uses only one displacement map for all frames, which greatly reduces the amount of data.

Abstract Hierarchical culling is a key acceleration technique used to efficiently handle massive models for ray tracing, collision detection, etc. To support such hierarchical culling, bounding volume hierarchies (BVHs) combined with meshes are widely used. However, BVHs may require a very large amount of memory space, which can negate the benefits of using BVHs. To address this problem, we present a novel hierarchical‐culling oriented compact mesh representation, HCCMesh, which tightly integrates a mesh and a BVH together. As an in‐core representation of the HCCMesh, we propose an i‐HCCMesh representation that provides an efficient random hierarchical traversal and high culling efficiency with a small runtime decompression overhead. To further reduce the storage requirement, the in‐core representation is compressed to our out‐of‐core representation, o‐HCCMesh, by using a simple dictionary‐based compression method. At runtime, o‐HCCMeshes are fetched from an external drive and decompressed to the i‐HCCMeshes stored in main memory. The i‐HCCMesh and o‐HCCMesh show 3.6:1 and 10.4:1 compression ratios on average, compared to a naively compressed (e.g., quantized) mesh and BVH representation. We test the HCCMesh representations with ray tracing, collision detection, photon mapping, and non‐photorealistic rendering. Because of the reduced data access time, a smaller working set size, and a low runtime decompression overhead, we can handle models ten times larger in commodity hardware without the expensive disk I/O thrashing. When we avoid the disk I/O thrashing using our representation, we can improve the runtime performances by up to two orders of magnitude over using a naively compressed representation.

This paper presents a simple and effective image retargeting method that preserves visually important parts while reducing unwanted distortions of an image. Our approach is based on a novel importance diffusion scheme, which propagates importance of removed pixels to their neighbors for preserving visual contexts and avoiding over-shrinkage of unimportant parts. Importance diffusion enables even a simple row/column removal method, which removes the least important rows/columns repeatedly, to produce visually pleasant results. It also provides control over the trade-off between uniform and non-uniform sampling for the row/column removal and seam carving methods. Experimental result demonstrates that importance diffusion successfully improves the retargeting results of row/column removal and seam carving.

This paper presents a fast deblurring method that produces a deblurring result from a single image of moderate size in a few seconds. We accelerate both latent image estimation and kernel estimation in an iterative deblurring process by introducing a novel prediction step and working with image derivatives rather than pixel values. In the prediction step, we use simple image processing techniques to predict strong edges from an estimated latent image, which will be solely used for kernel estimation. With this approach, a computationally efficient Gaussian prior becomes sufficient for deconvolution to estimate the latent image, as small deconvolution artifacts can be suppressed in the prediction. For kernel estimation, we formulate the optimization function using image derivatives, and accelerate the numerical process by reducing the number of Fourier transforms needed for a conjugate gradient method. We also show that the formulation results in a smaller condition number of the numerical system than the use of pixel values, which gives faster convergence. Experimental results demonstrate that our method runs an order of magnitude faster than previous work, while the deblurring quality is comparable. GPU implementation facilitates further speed-up, making our method fast enough for practical use.

This paper presents a fast color-to-gray conversion algorithm which robustly reproduces the visual appearance of a color image in grayscale. The conversion preserves feature discriminability and reasonable color ordering, while respecting the original lightness of colors, by simple optimization of a nonlinear global mapping. Experimental results show that our method produces convincing results for a variety of color images. We further extend the method to temporally coherent color-to-gray video conversion.

Previous mesh compression techniques provide decent properties such as high compression ratio, progressive decoding, and out-of-core processing. However, only a few of them supports the random accessibility in decoding, which enables the details of any specific part to be available without decoding other parts. This paper proposes an effective framework for the random accessibility of mesh compression. The key component of the framework is a wire-net mesh constructed from a chartification of the given mesh. Charts are compressed separately for random access to mesh parts and a wire-net mesh provides an indexing and stitching structure for the compressed charts. Experimental results show that random accessibility can be achieved with competent compression ratio, which is only a little worse than single-rate and comparable to progressive encoding. To demonstrate the merits of the framework, we apply it to process huge meshes in an out-of-core manner, such as out-of-core rendering and out-of-core editing.

We present a non-photorealistic rendering technique that automatically delivers a stylized abstraction of a photograph. Our approach is based on shape/color filtering guided by a vector field that describes the flow of salient features in the image. This flow-based filtering significantly improves the abstraction performance in terms of feature enhancement and stylization. Our method is simple, fast, and easy to implement. Experimental results demonstrate the effectiveness of our method in producing stylistic and feature-enhancing illustrations from photographs.

Abstract This paper presents a simple algorithm for producing stylistic abstraction of a photograph. Based on mean curvature flow in conjunction with shock filter, our method simplifies both shapes and colors simultaneously while preserving important features. In particular, we develop a constrained mean curvature flow, which outperforms the original mean curvature flow in conveying the directionality of features and shape boundaries. The proposed algorithm is iterative and incremental, and therefore the level of abstraction is intuitively controlled. Optionally, simple user masking can be incorporated into the algorithm to selectively control the abstraction speed and to protect particular regions. Experimental results show that our method effectively produces highly abstract yet feature‐preserving illustrations from photographs.

Abstract This paper presents an automatic method for producing stipple renderings from photographs, following the style of professional hedcut illustrations. For effective depiction of image features, we introduce a novel dot placement algorithm which adapts stipple dots to the local shapes. The core idea is to guide the dot placement along ‘feature flow’ extracted from the feature lines, resulting in a dot distribution that conforms to feature shapes. The sizes of dots are adaptively determined from the input image for proper tone representation. Experimental results show that such feature‐guided stippling leads to the production of stylistic and feature‐emphasizing dot illustrations.

Line-art illustrations are effective tools for conveying shapes and shading of complex objects. We present a set of new algorithms to render line-art illustrations of dynamic and specular (reflective and refractive) surfaces. We first introduce a real-time principal direction estimation algorithm to determine the line stroke directions on dynamic opaque objects using neighboring normal ray triplets. To render reflections or refractions in a line-art style, we develop a stroke direction propagation algorithm by using multi-perspective projections to propagate the stroke directions from the nearby opaque objects onto specular surfaces. Finally, we present an image-space stroke mapping method to draw line strokes using the computed or propagated stroke directions. We implement these algorithms using a GPU and demonstrate real-time illustrations of scenes with dynamic and specular 3D models in line-art styles.

We propose a method for removing non-uniform motion blur from multiple blurry images. Traditional methods focus on estimating a single motion blur kernel for the entire image. In contrast, we aim to restore images blurred by unknown, spatially varying motion blur kernels caused by different relative motions between the camera and the scene. Our algorithm simultaneously estimates multiple motions, motion blur kernels, and the associated image segments. We formulate the problem as a regularized energy function and solve it using an alternating optimization technique. Real- world experiments demonstrate the effectiveness of the proposed method.

We present a novel scheme for automatically generating line drawings from 2D images, aiming to facilitate effective visual communication. In contrast to conventional edge detectors, our technique imitates the human line drawing process and consists of two parts: line extraction and line rendering. We propose a novel line extraction method based on likelihood-function estimation, which effectively finds the genuine shape boundaries. We consider the feature scale and the blurriness of lines with which the detail and the focus-level of lines are controlled in the rendering. We also employ stroke textures to provide a variety of illustration styles. Experimental results demonstrate that our technique generates various kinds of line drawings from 2D images enabled by the control over detail, focus, and style.

We describe a GPU-based algorithm for rendering a 3D model as a line drawing, based on the insight that a line drawing can be understood as an abstraction of a shaded image. We thus render lines along tone boundaries or thin dark areas in the shaded image. We extend this notion to the dual: we render highlight lines along thin bright areas and tone boundaries. We combine the lines with toon shading to capture broad regions of tone. The resulting line drawings effectively convey both shape and material cues. The lines produced by the method can include silhouettes. creases, and ridges, along with a generalization of suggestive contours that responds to lighting as well as viewing changes. The method supports automatic level of abstraction, where the size of depicted shape features adjusts appropriately as the camera zooms in or out. Animated models can be rendered in real time because costly mesh curvature calculations are not needed.

This paper proposes a general framework for overfitting control in surface reconstruction from noisy point data. The problem we deal with is how to create a model that will capture as much detail as possible and simultaneously avoid reproducing the noise of the input points. The proposed framework is based on extra-sample validation. It is fully automatic and can work in conjunction with any surface reconstruction algorithm. We test the framework with a Radial Basis Function algorithm, Multi-level Partition of Unity implicits, and the Power Crust algorithm.

We present a method to view-dependently control the size of shape features depicted in computer-generated line drawings of 3D meshes. Our method exhibits good temporal coherence during level of detail transitions, and is fast because the calculations are carried out entirely on the GPU. The strategy is to pre-compute, via a digital geometry processing technique, a sequence of filtered versions of the mesh that eliminate shape features at progressively larger scales. Each filtered mesh retains the original connectivity, providing a direct correspondence between meshes.At run-time, the meshes are loaded onto the graphics card and a vertex program interpolates curvatures and positions between corresponding vertices in adjacent meshes of the sequence. A fragment program then renders silhouettes and suggestive contours to produce a line drawing for which the size of depicted shape features follows a user-specified "target size". For example, we can depict fine shape features over nearby surfaces, and appropriately coarse-scaled features in more distant regions. More general level-of-detail policies could be implemented on top of our approach by letting the target size vary with scene attributes such as depth, image location, or annotations provided by the scene designer.

Abstract This paper presents a novel approach for mesh compression, which we call multiresolution random accessible mesh compression. In contrast to previous mesh compression techniques, the approach enables us to progressively decompress an arbitrary portion of a mesh without decoding other non‐interesting parts. This simultaneous support of random accessibility and progressiveness is accomplished by adapting selective refinement of a multiresolution mesh to the mesh compression domain. We present a theoretical analysis of our connectivity coding scheme and provide several experimental results. The performance of our coder is about 11 bits for connectivity and 21 bits for geometry with 12‐bit quantization, which can be considered reasonably good under the constraint that no fixed neighborhood information can be used for coding to support decompression in a random order. Categories and Subject Descriptors (according to ACM CCS): I.3.5 [Computer Graphics]: Computational Geometry and Object Modeling

We propose an out-of-core method for creating semi-regular surface representations from large input surface meshes. Our approach is based on a streaming implementation of the MAPS remesher of Lee et al. [18]. Our remeshing procedure consists of two stages. First, a simplification process is used to obtain the base domain. During simplification, we maintain the mapping information between the input and the simplified meshes. The second stage of remeshing uses the mapping information to produce samples of the output semi-regular mesh. The out-of-core operation of our method is enabled by the synchronous streaming of a simplified mesh and the mapping information stored at the original vertices. The synchronicity of two streaming buffers is maintained using a specially designed write strategy for each buffer. Experimental results demonstrate the remeshing performance of the proposed method, as well as other applications that use the created mapping between the simplified and the original surface representations.

Feature sensitive mesh segmentation is important for many computer graphics and geometric modeling applications. In this paper, we develop a mesh segmentation method, which is capable of producing high-quality shape partitioning. It respects fine shape features and works well on various types of shapes, including natural shapes and mechanical parts. The method combines a procedure for clustering mesh normals with a modification of the mesh clarification technique. For clustering of mesh normals, we adopt Mean Shift, a powerful general purpose technique for clustering scattered data. We demonstrate advantages of our method by comparing it with two state-of-the-art mesh segmentation techniques.

We present a 2D feature-based technique for morphing 3D objects represented by light fields. Existing light field morphing methods require the user to specify corresponding 3D feature elements to guide morph computation. Since slight errors in 3D specification can lead to significant morphing artifacts, we propose a scheme based on 2D feature elements that is less sensitive to imprecise marking of features. First, 2D features are specified by the user in a number of key views in the source and target light fields. Then the two light fields are warped view by view as guided by the corresponding 2D features. Finally, the two warped light fields are blended together to yield the desired light field morph. Two key issues in light field morphing are feature specification and warping of light field rays. For feature specification, we introduce a user interface for delineating 2D features in key views of a light field, which are automatically interpolated to other views. For ray warping, we describe a 2D technique that accounts for visibility changes and present a comparison to the ideal morphing of light fields. Light field morphing based on 2D features makes it simple to incorporate previous image morphing techniques such as nonuniform blending, as well as to morph between an image and a light field.

Mesh partitioning and parts extraction have become key ingredients for many mesh manipulation applications both manual and automatic. In this paper, we present an intelligent scissoring operator for meshes which supports both automatic segmentation and manual cutting. Instead of segmenting the mesh by clustering, our approach concentrates on finding and defining the contours for cutting. This approach is based on the minima rule, which states that human perception usually divides a surface into parts along the contours of concave discontinuity of the tangent plane. The technique uses feature extraction to find such candidate feature contours. Subsequently, such a contour can be selected either automatically or manually, or the user may draw a 2D line to start the scissoring process. The given open contour is completed to form a loop around a specific part of the mesh, and this loop is used as the initial position of a 3D geometric snake. The snake moves by relaxation until it settles to define the final scissoring position. This process uses several fundamental geometric mesh attributes, such as curvature and centricity, and enables both automatic segmentation and an easy-to-use intelligent-scissoring operator.

Multiresolution geometry streaming has been well studied in recent years. The client can progressively visualize a triangle mesh from the coarsest resolution to the finest one while a server successively transmits detail information. However, the streaming order of the detail data usually depends only on the geometric importance, since basically a mesh simplification process is performed backwards in the streaming. Consequently, the resolution of the model changes globally during streaming even if the client does not want to download detail information for the invisible parts from a given view point. In this paper, we introduce a novel framework for view-dependent streaming of multiresolution meshes. The transmission order of the detail data can be adjusted dynamically according to the visual importance with respect to the client's current view point. By adapting the truly selective refinement scheme for progressive meshes, our framework provides efficient view-dependent streaming that minimizes memory cost and network communication overhead. Furthermore, we reduce the per-client session data on the server side by using a special data structure for encoding which vertices have already been transmitted to each client. Experimental results indicate that our framework is efficient enough for a broadcast scenario where one server streams geometry data to multiple clients with different view points.

A surface light field is a function that gives the colors of each object point viewed from different directions. Object representation with a surface light field provides a nice structure for 3D photography. This paper presents a feature-based morphing technique for two objects equipped with surface light fields. The technique consists of geometry morphing and in-between light field mapping. Geometry morphing is accomplished by 3D mesh morphing, where we introduce a vertex merging technique to generate a simpler metamesh. In in-between light field mapping, an in-between object is rendered by extracting necessary fragments from input surface light fields. We also propose an acceleration technique for rendering an in-between object. Experimental results with real and synthetic data show natural and plausible morphing between objects with surface light fields. The proposed morphing technique can be used as an editing tool for 3D photography.

One of the goals and defining characteristics of virtual reality systems is to create "presence" and fool the user into believing that one is, or is doing something "in" the synthetic environment. Most research and papers on presence to date have been directed toward coming up with the definitions of presence, and based on them, identifying key elements that affect presence. We carried out an elaborate experiment in which presence levels were measured (with subjective questionnaire) in test virtual worlds configured with different combinations of six visual presence elements.

The paper investigates the set of all selectively refined meshes that can be obtained from a progressive mesh. We call the set the transitive mesh space of a progressive mesh and present a theoretical analysis of the space. We define selective edge collapse and vertex split transformations, which we use to traverse all selectively refined meshes in the transitive mesh space. We propose a complete selective refinement scheme for a progressive mesh based on the transformations and compare the scheme with previous selective refinement schemes in both theoretical and experimental ways. In our comparison, we show that the complete scheme always generates selectively refined meshes with smaller numbers of vertices and faces than previous schemes for a given refinement criterion. The concept of dual pieces of the vertices in the vertex hierarchy plays a central role in the analysis of the transitive mesh space and the design of selective edge collapse and vertex split transformations.

3D mesh morphing based on a metamesh has fundamental limitations of complicated in-between meshes and no topology (connectivity) changes in a metamorphosis. This paper presents a novel approach for 3D mesh morphing, which is not based on a metamesh. The approach simultaneously interpolates the topology and geometry of input meshes. In our approach, an in-between mesh contains only the vertices from the source and target meshes. Since no additional vertices are introduced, the in-between meshes are much simpler than those generated by previous techniques.

Feature detection is important in various mesh processing techniques, such as mesh editing, mesh morphing, mesh compression, and mesh signal processing. In spite of much research in computer vision, automatic feature detection even for images still remains a difficult problem. To avoid this difficulty, semi-automatic or interactive techniques for image feature detection have been investigated. In this paper, we propose a geometric snake as an interactive tool for feature detection on a 3D triangular mesh. A geometric snake is an extension of an image snake, which is an active contour model that slithers from its initial position specified by the user to a nearby feature while minimizing an energy functional. To constrain the movement of a geometric snake onto the surface of a mesh, we use the parameterization of the surrounding region of a geometric snake. Although the definition of a feature may vary among applications, we use the normal changes of faces to detect features on a mesh. Experimental results demonstrate that geometric snakes can successfully capture nearby features from user-specified initial positions.

Uniform cubic B-spline functions have been used for mapping functions in various areas such as image warping and morphing, 3D deformation, and volume morphing. The injectivity (one-to-one property) of a mapping function is important to obtain good results in these areas. We consider the local injectivity conditions of 2D and 3D uniform cubic B-spline functions. We propose a geometric interpretation of the local injectivity of a uniform cubic B-spline function, with which 2D and 3D cases can be handled in a similar way. Based on the geometric interpretation, we present sufficient conditions for the local injectivity that are represented in terms of control point displacements. These sufficient conditions are simple and easy to check and will be useful to guarantee the injectivity of mapping functions in application areas.

This paper presents a novel approach to multiresolution editing of a triangular mesh. The basic idea is to embed an editing area of a mesh onto a 2D rectangle and interpolate the user‐specified editing information over the 2D rectangle. The result of the interpolation is mapped back to the editing area and then used to update the mesh. We adopt harmonic maps for the embedding and multilevel B‐splines for the interpolation. The proposed mesh editing technique can handle an arbitrary mesh without any preprocessing such as remeshing. It runs fast enough to support interactive editing and produces intuitive editing results.

This paper proposes a new approach to image‐based rendering that generates an image viewed from an arbitrary camera position and orientation by rendering optical flows extracted from reference images. To derive valid optical flows, we develop an analysis technique that improves the quality of stereo matching. Without using any special equipments such as range cameras, this technique constructs reliable optical flows from a sequence of matching results between reference images. We also derive validity conditions of optical flows and show that the obtained flows satisfy those conditions. Since environment geometry is inferred from the optical flows, we are able to generate more accurate images with this additional geometric information. Our approach makes it possible to combine an image rendered from optical flows with an image generated by a conventional rendering technique through a simple Z‐buffer algorithm.

The paper describes a fast algorithm for scattered data interpolation and approximation. Multilevel B-splines are introduced to compute a C/sup 2/ continuous surface through a set of irregularly spaced points. The algorithm makes use of a coarse to fine hierarchy of control lattices to generate a sequence of bicubic B-spline functions whose sum approaches the desired interpolation function. Large performance gains are realized by using B-spline refinement to reduce the sum of these functions into one equivalent B-spline function. Experimental results demonstrate that high fidelity reconstruction is possible from a selected set of sparse and irregular samples.

This paper presents a new image morphing method using a two-dimensional deformation technique which provides an intuitive model for a warp. The deformation technique derives aC1-continuous and one-to-one warp from a set of point pairs overlaid on two images. The resulting in-between image precisely reflects the correspondence of features specified by an animator. We also control the transition behaviour in a metamorphosis sequence by taking another deformable surface model, which is simpler and thus more efficient than the deformation technique for a warp. The proposed method separates transition control from feature interpolation and is easier to use than the previous techniques. The multigrid relaxation method is employed to solve a linear system in deriving a warp or transition rates. This method makes our image morphing technique fast enough for an interactive environment.

This paper describes an image metamorphosis technique to handle scattered feature constraints specified with points, polylines, and splines. Solutions to the following three problems are presented: feature specification, warp generation, and transition control. We demonstrate the use of snakes to reduce the burden of feature specification. Next, we propose the use of multilevel free-form deformations (MFFD) to compute C/sup 2/-continuous and one-to-one mapping functions among the specified features. The resulting technique, based on B-spline approximation, is simpler and faster than previous warp generation methods. Furthermore, it produces smooth image transformations without undesirable ripples and foldovers. Finally, we simplify the MFFD algorithm to derive transition functions to control geometry and color blending. Implementation details are furnished and comparisons among various metamorphosis techniques are presented.

This paper presents new solutions to the following three problems in image morphing: feature specification, warp generation, and transition control.To reduce the burden of feature specification, we first adopt a computer vision technique called snakes.We next propose the use of multilevel free-form deformations (MFFD) to achieve C 2 -continuous and one-to-one warps among feature point pairs.The resulting technique, based on B-spline approximation, is simpler and faster than previous warp generation methods.Finally, we simplify the MFFD method to construct C 2 -continuous surfaces for deriving transition functions to control geometry and color blending.

This paper presents a new image morphing technique using deformable surfaces. Drawbacks of previous techniques are overcome by a physically-based approach which provides an intuitive model for a warp. A warp is derived by two deformable surfaces which specify horizontal and vertical displacements of points on an image. This paper also considers the control of transition behavior in a metamorphosis sequence. The presented technique separates the transition control from interpolating features making it much easier to use than the previous techniques. The multigrid relaxation method is used to compute a deformable surface for a warp or transition rates. This method makes the presented image morphing technique fast enough for an interactive environment.>