Sketch colorization plays an important role in animation and digital illustration production tasks. However, existing methods still meet problems in that text-guided methods fail to provide accurate color and style reference, hint-guided methods still involve manual operation, and image-referenced methods are prone to cause artifacts. To address these limitations, we propose a diffusion-based framework inspired by real-world animation production work-flows. Our approach leverages the sketch as the spatial guidance and an RGB image as the color reference, and separately extracts foreground and background from the reference image with spatial masks. Particularly, we introduce a split cross-attention mechanism with LoRA (Low-Rank Adaptation) modules. They are trained separately with foreground and background regions to control the corresponding embeddings for keys and values in cross-attention. This design allows the diffusion model to integrate information from foreground and background independently, preventing interference and eliminating the spatial artifacts. During inference, we design switchable inference modes for diverse use scenarios by changing modules activated in the framework. Extensive qualitative and quantitative experiments, along with user studies, demonstrate our advantages over existing methods in generating high-qualigy artifact-free results with geometric mismatched references. Ablation studies further confirm the effectiveness of each component. Codes are available at https://github.com/tellurion-kanata/colorizeDiffusion.

Diffusion models have achieved great success in dual-conditioned image generation. However, they still face significant challenges in image-guided sketch colorization, where reference and sketch images usually exhibit different semantics and spatial structures. This mismatch, termed “distribution shift” in this peper, results in various artifacts and degrades the colorization quality. To address this issue, we conducted thorough investigations into the image-prompted latent diffusion model and developed a two-stage training framework to mitigate the effects of distribution shift based on our analysis. Comprehensive quantitative comparisons, qualitative evaluations, and user studies were performed to demonstrate the superiority of our proposed methods. Additionally, ablation studies were conducted to assess the impact of the distribution shift and the selection of reference embeddings. Codes are made publicly available at https://github.com/tellurion-kanata/colorizeDiffusion.

Inbetweening, interpolating between two keyframes, is an essential step in traditional handdrawn animation production. Many interpolation methods used to assist in animation production require correspondences between lines in two consecutive keyframes. This paper proposes a method for finding line correspondences that can apply to two keyframes, even those with occlusions. Our method determines one-to-many line correspondences resulting from occlusions and other factors by iteratively finding line correspondences using minimum weight edge cover problems on a bipartite graph using a set of lines on two keyframes. In advance to finding these line correspondences, we decompose lines on keyframes into evenly spaced points and determine which lines in the other keyframe these points correspond to. Using the result, we calculate the weights for the minimum weight edge cover problem. Experiments show that our proposed method achieves higher accuracy in line correspondences, including one-to-many correspondences than five previous methods.

In this paper, we propose a method for generating environment maps for sky illumination by using an atmospheric radiative transfer simulator, AT3D, with meteorological satellite data used for cloud modeling as volumetric grid data and previously measured concentrations of atmospheric molecular and aerosols. We present generated sky environment maps, their respective luminance intensity histograms, and images rendered using the maps as sky illumination and compare them with those of captured spherical images.

Creating realistic materials is essential in the construction of immersive virtual environments. While existing techniques for material capture and conditional generation rely on flash-lit photos, they often produce artifacts when the illumination mismatches the training data. In this study, we introduce DiffMat, a novel diffusion model that integrates the CLIP image encoder and a multi-layer, cross-attention denoising backbone to generate latent materials from images under various illuminations. Using a pre-trained StyleGAN-based material generator, our method converts these latent materials into high-resolution SVBRDF textures, a process that enables a seamless fit into the standard physically based rendering pipeline, reducing the requirements for vast computational resources and expansive datasets. DiffMat surpasses existing generative methods in terms of material quality and variety, and shows adaptability to a broader spectrum of lighting conditions in reference images.

Abstract Automatic sketch colourization is a highly interestinged topic in the image‐generation field. However, due to the absence of texture in sketch images and the lack of training data, existing reference‐based methods are ineffective in generating visually pleasant results and cannot edit the colours using text tags. Thus, this paper presents a conditional generative adversarial network (cGAN)‐based architecture with a pre‐trained convolutional neural network (CNN), reference‐based channel‐wise attention (RBCA) and self‐adaptive multi‐layer perceptron (MLP) to tackle this problem. We propose two‐step training and spatial latent manipulation to achieve high‐quality and colour‐adjustable results using reference images and text tags. The superiority of our approach in reference‐based colourization is demonstrated through qualitative/quantitative comparisons and user studies with existing network‐based methods. We also validate the controllability of the proposed model and discuss the details of our latent manipulation on the basis of experimental results of multi‐label manipulation.

With the continued increase of communication speeds and improvements to edge computing environments, server-side rendering is once again attracting attention due to the diversification of user terminals.There are a few new structures that can be scaled in accordance with the number of users. However, in the case of applications where many people share the same 3D space in real time, the number of unique models and the inherent complexity of the models uploaded by users make it difficult for cloud-based rendering systems to handle such applications.In this paper, we introduce a new image-based distributed rendering architecture that allows multiple clients sharing a space to receive rendering results. Further, we reduce the throughput with data compression and add realism with shadow effect. Finally, we report the measurement results of one system based on our architecture using the grid computers TSUBAME 3.0.

When rendering a 3D model represented by polygons, subdividing a greater number of polygons means that more details can be expressed. Unfortunately rendering a large number of polygons requires heavy calculation. It is therefore necessary to divide polygons appropriately using a tessellation procedure to reduce the rendering computation cost while maintaining visual quality. In this paper, we propose a real-time subdivision control method that utilizes a binary classification table pre-stored by a support sector machine (SVM) to choose one subdivision level from two options. The SVM takes the four features obtained in the real-time rendering procedure as input and performs a binary classification to determine whether a polygon should be subdivided or not. Our experimental results show that the proposed method reduces the rendering time by approximately 22.3% compared to subdividing all polygons.

Techniques to render 3D models like hand-drawings are often required. In this paper, we propose an approach that generates line-drawing with various styles by machine learning. We train two Convolutional neural networks (CNNs), of which one is a line extractor from the depth and normal images of a 3D object, and the other is a line thickness applicator. The following process to CNNs interprets the thickness of the lines as intensity to control properties of a line style. Using the obtained intensities, non-uniform line styled drawings are generated. The results show the efficiency of combining the machine learning method and the interpreter.

We propose a method that allows a user to create folded surfaces with a good sense of reality from simply sketches directly drawing curves on the surface of a cloth 3D model. The user can obtain a one-shot image of a simple cloth fold simulation result with a good sense of reality through our method in a short time, even for the one who has little or no knowledge of the cloth apparel cutting structure and the physical properties of the fabric. The method consists of a simulation method that allows the user to make a preliminary adjustment to their cloth model, a reconstruction method to design the folded surface by sketching on the surface, and a refinement method to modify the shape of the folded surface.

Generating face images in various directions from an image will be useful to create avatars in VR. In this paper, we introduce a new deep generative model to generate turnaround face images from an image via a latent code space with a parameter. The model was learned with a large scale image dataset annotated with attributes but not including exact target images.

Yamato-e and Ukiyo-e style are Japanese traditional painting styles that have two characteristics, linearity and verticality. We propose a novel generalized projection method to produce such images using 3D computer graphics. The method enables orthographic, perspective, reverse perspective, oblique projections, and a projection with a different field of view vertically and horizontally. The key idea is to control the tilt angles of four-sided planes of a view volume indirectly and independently with the projection reference plane. It can be applied easily to user interactive CG creating tools, and it can produce pseudo multi-perspective projected images. We can see them from our implementation and the resultant images.

This article addresses the relighting of outdoor and large indoor scenes illuminated by nondistant lights, which has seldom been discussed in previous works. We propose a method for users to interactively edit the illumination of a scene by moving existing lights and inserting synthetic lights into the scene that requires only a small amount of user annotation and a single low-dynamic range (LDR) image. We achieve this by adopting a top-down approach that estimates the scene reflectance by fitting a diffuse illumination model to a photograph. This approach gains stability and robustness by estimating the camera, scene geometry, and light sources in sequence and by using a confidence map, which is a per-pixel weight map. The results of our evaluation demonstrates that the proposed method can estimate a scene accurately enough for realistic relighting of images. Moreover, the experimental results of our user studies show that the synthesized images are so realistic as to be almost indistinguishable from real photographs.

Abstract In this paper, we propose a method to maintain the temporal coherence of stylized feature lines extracted from 3D models and preserve an artistically intended stylization provided by the user. We formally define the problem of combining spatio‐temporal continuity and artistic intention as a weighted energy minimization problem of competing constraints. The proposed method updates the style properties to provide real‐time smooth transitions from current to goal stylization, by assuring first‐ and second‐order temporal continuity, as well as spatial continuity along each stroke. The proposed weighting scheme guarantees that the stylization of strokes maintains motion coherence with respect to the apparent motion of the underlying surface in consecutive frames. This weighting scheme emphasizes temporal continuity for small apparent motions where the human vision system is able to keep track of the scene, and prioritizes the artistic intention for large apparent motions where temporal coherence is not expected. The proposed method produces temporally coherent and visually pleasing animations without the flickering artifacts of previous methods, while also maintaining the artistic intention of a goal stylization provided by the user.

We propose a method to stylize individual lines and preserve their properties as the viewpoint is modified. As the camera position changes and view-dependent lines move across the surface, we track each line in order to extract and store the areas of the surface in which they appear. Each line has an unique reference that we use to check an indexed table from which we can recover its stylization after arbitrary camera movements. We call this region the tracked area of the line. Additionally, we provide a method for line segmentation of occluding contours. In our algorithm, we divide the contour lines at inflection points in order to segment them in a way that is suited for localized stylization as well as to better approximate artist strokes. We also optimize the line tracking process by making use of the Gaussian curvature to separate the elliptic and hyperbolic areas of the surface. In this paper we show how each line only appears in a limited area of the surface corresponding to its tracked area. We also discuss how to eliminate the over-segmentation caused by numerical instability of the approximations in polygonal surfaces as well as how smoothing can be used in order to deal with highly detailed models. Finally, we highlight the achievements of our system for localized stylization of the contour lines of 3D models.

We propose a method to generate grasping poses which can be used as the initial state of a motion to manipulate a given object.

In this paper, we introduce the visualization methods of constraints on color harmony, and the color harmonization method on a hue, based on Coloroid color space, which is defined by Nemcsics.

Shadow mapping is an efficient method to generate shadows in real time computer graphics and has broad variations from hard to soft shadow synthesis. Soft shadowing based on shadow mapping is a blurring technique on a shadow map or on screen space. Blurring on screen space can easily combine with a deferred shading pipeline. However, blurring on screen space has a drawback: the generated shadow is not correct when a view direction has a large angle to the normal of the shadowed plane. In this paper, we introduce a new screen space based method for soft shadowing that is faster and generates more accurate soft shadows than the previous screen space soft shadow mapping method. The resultant images show shadows by our method just stand in the same place, while shadows by the previous method change in terms of penumbra while the view moves. Surprisingly, although our method is more complex than the previous method, the measurement of the calculation time shows our method is also faster than the previous method. This is because our method controls the blurring area more accurately and thus successfully reduces multiplications for blurring.

Abstract Human‐like grasp planning is difficult because a human hand has a high number of degrees of freedom, and there are many grasping styles depending on the shape of an object and purpose. We propose a fast grasp synthesis system which enables a user to choose the desired grasping styles from a set of grasp types in a human grasp taxonomy. Given a 3D model of an object, our system detects graspable positions and generates grasping hand postures in every applicable grasp types in the grasp taxonomy for each grasping position. Hand postures are generated separately for each digit, and hand alignment is then refined iteratively. A user can also specify the grasping position by moving the cursor onto object surface, as well as grasp type. The generated hand postures are shown as a table of thumbnail images, and the user can select the grasping hand posture by clicking on one of those postures. Our system enables interactive generation of various grasping hand postures in real time.

Digital two-dimensional images are usually sampled on square lattices, while the receptors of the human eye are following a hexagonal structure. It is the main motivation for adopting hexagonal lattices. The fundamental operation in many image processing algorithms is to extract the gradient information. As such, various gradient operators have been proposed for square lattices and have been thoroughly optimized. Accurate gradient operators for hexagonal lattices have, however, not been researched well enough, while the distance between neighbor pixels is constant. We therefore derive consistent gradient operators on hexagonal lattices and compare them with the existing optimized filters on square lattices. The results show that the derived filters on hexagonal lattices achieve a better signal-to-noise ratio than those on square lattices. Results on artificial images also show that the derived filters on hexagonal lattices outperform the square ones with respect to accuracy of gradient intensity and orientation detection.

We propose in this paper an interactive sketch-based system for simulating oriental brush strokes on complex shapes. We introduce a contour-driven approach where the user inputs contours to represent complex shapes, the system estimates automatically the optimal trajectory of the brush, and then renders them into oriental ink painting. Unlike previous work where the brush trajectory is explicitly specified as input, we automatically estimate this trajectory given the outline of the shape to paint. Existing methods can be classified into: (1) methods that explicitly model a virtual 3D brush and mimic its effect on a paper [Wang and Wang 2007], and (2) methods that simulate the rendering effect on a 2D canvas without an explicit 3D brush model [Okabe et al. 2007]. Our approach falls into the second category. Figure 1 shows four results generated by our algorithm.

Shape perception depends on a lot of factors, for example texture or shade. When rendering a surface, the question whether the viewer correctly understands the shape of the rendered object is open. The use of different rendering methods can improve or harm shape recognition. This paper presents various styles of line drawings, three traditionally used and a new one, that we describe here, then evaluates their success at conveying the shape of the underlying object. Each line drawing is generated according to simple rules, using geometric properties of the object, for example the normal vector orientation or the principal directions of curvature. An psychophysical evaluation is then performed to compare the various styles. Participants had to adjust a probe until it matches the normal vector they perceived. Results show that the new style that we propose performs better than the principal directions of curvature, for particular cases.

Color transformation is the most effective method to improve the mood of an image, because color has a large influence in forming the mood. However, conventional color transformation tools have a tradeoff between the quality of the resultant image and the amount of manual operation. To achieve a more detailed and natural result with less labor, we previously suggested a method that performs an example-based color stylization of images using perceptual color categories. In this paper, we extend this method to make the algorithm more robust and to stylize the colors of video frame sequences. We present a variety of results, arguing that these images and videos convey a different, but coherent mood.

The action of grasping an object is very important for a humanoid agent with hand, evolving autonomously in a virtual world. If an agent is given the order "grasp the object", he has to determine the portion of the object to grasp. In order to determine a grasping position, it is necessary to recognize the form of the 3-dimensional geometry, and detect portions which are suitable for grasp. Assuming that the object is grasped by one hand, we propose two techniques for detecting appropriate portions to be grasped on the surface of an object and for generating the grasping postures. In this paper, our main contributions are as follows: the first is the detection of appropriate portions to be grasped on the surface of an object; the second is the formation of the hand shape to grasp various shaped objects. Using the two proposing methods, grasped positions and the hand shapes for each of them are computed. Moreover, these positions are evaluated by their stabilities using the robotics technique

In this paper, we propose a novel technique to estimate joint ROM (range of motion) from Mocap (motion capture) data. For each joint, the probability density function of joint angle is estimated by mixture of Gaussian, and the angle space is divided into Voronoi cells which have a kernel of Gaussian at these centers. Then, the sets of the Voronoi cells in two different joints, which occur in the Mocap data, are used to approximate the ROM taking into account the change related to the angle of the other joint. As all of this estimation processes are performed on the simple ball joint model, we do not need to introduce any complex musculo-skeletal model. Our method can also estimate the ROMs even for the joints which have difficulty to measure the angle directly by physical measurement as well as the joints of limbs

We propose a new method of estimating object contours from thresholded binary edge images. Our work is motivated by the edge-based video object plane (VOP) generation proposed by Meier, targeted at MPEG-4 object-based video coding. When objects are detected in the form of binary edge images, we have to estimate the complete object contour from edges that do not form a closed contour. The proposed method is designed to carry out this task using only binary edge images. To achieve this, we first introduce a probability field representing the probability that the object contour lies on each pixel, and then take a modified geodesic active contour approach to make the initial contour converge with the contour of the object according to the probability field.

We describe a new computational approach to stylize the colors of an image by using a reference image. During processing, we take the characteristics of human color perception into account to generate more appealing results. Our system starts by classifying each pixel value into one of the basic color categories, derived from our psychophysical experiments. The basic color categories are perceptual categories that are universal to everyone, regardless of nationality or cultural background. These categories are used to provide restrictions on color transformations to avoid generating unnatural results. Our system then renders a new image by transferring colors from a reference image to the input image, based on these categorizations. To avoid artifacts due to the explicit clustering, our system defines fuzzy categorization when pseudocontours appear in the resulting image. We present a variety of results and show that our method performs a large, yet natural, color transformation without any sense of incongruity and that the resulting images automatically capture the characteristics of the colors used in the reference image.

Presently, most virtual reality systems use upper body parts to interact with objects in the virtual environment. This situation is caused by technological limitations of current interface devices. Starting from this viewpoint we developed a new interface for detecting ankle motions relative to the knee. We believe that hands-free navigation, unlike the majority of navigation techniques based on hand motions, has the greatest potential for maximizing the interactivity of virtual environments since navigation modes are more direct motion of the feet. We therefore, created a simple device to detect ankle movements with rotary encoders sensors. These sensors rotate according to the amount and direction of the movement of the foot. The sensors are attached to a sandal and can be used for many purposes including virtual games. Validation of our approach is given by discussion and illustration of some experimental results.

Usually, paintings are more appealing than photographic images. This is because paintings have styles. This style can be distinguished by looking at elements such as motif, color, shape deformation and brush texture. We focus on the effect of "color" element and devise a method for transforming the color of an input photograph according to a reference painting. To do this, we consider basic color category concepts in the color transformation process. By doing so, we achieve large but natural color transformations of an image.

We describe an algorithm that generates lines of varying thickness without any special information about the lines. The thickness is derived from only curvature and lengths from both ends. Therefore the algorithm is applicable in a variety of line rendering situations, for example 3D rendering engine for high quality cel-animation-like effects, reuse of geometrical data designed by CAD for advertising purposes, and photo retouching processes with edge detection methods and so on. In addition, using the generated varying thickness, we also describe algorithms for shading and embossing line rendering.