Image inpainting is the task of filling in missing or masked regions of an image with semantically meaningful content. Recent methods have shown significant improvement in dealing with large missing regions. However, these methods usually require large training datasets to achieve satisfactory results, and there has been limited research into training such models on a small number of samples. To address this, we present a novel data-efficient generative residual image inpainting method that produces high-quality inpainting results. The core idea is to use an iterative residual reasoning method that incorporates convolutional neural networks (CNNs) for feature extraction and transformers for global reasoning within generative adversarial networks, along with image-level and patch-level discriminators. We also propose a novel forged-patch adversarial training strategy to create faithful textures and detailed appearances. Extensive evaluation shows that our method outperforms previous methods on the data-efficient image inpainting task, both quantitatively and qualitatively.

Existing facial editing methods have achieved remarkable results, yet they often fall short in supporting multimodal conditional local facial editing. One of the significant evidences is that their output image quality degrades dramatically after several iterations of incremental editing, as they do not support local editing. In this paper, we present a novel multimodal generative and fusion framework for globally-consistent local facial editing (FACEMUG) that can handle a wide range of input modalities and enable fine-grained and semantic manipulation while remaining unedited parts unchanged. Different modalities, including sketches, semantic maps, color maps, exemplar images, text, and attribute labels, are adept at conveying diverse conditioning details, and their combined synergy can provide more explicit guidance for the editing process. We thus integrate all modalities into a unified generative latent space to enable multimodal local facial edits. Specifically, a novel multimodal feature fusion mechanism is proposed by utilizing multimodal aggregation and style fusion blocks to fuse facial priors and multimodalities in both latent and feature spaces. We further introduce a novel self-supervised latent warping algorithm to rectify misaligned facial features, efficiently transferring the pose of the edited image to the given latent codes. We evaluate our FACEMUG through extensive experiments and comparisons to state-of-the-art (SOTA) methods. The results demonstrate the superiority of FACEMUG in terms of editing quality, flexibility, and semantic control, making it a promising solution for a wide range of local facial editing tasks.

Blinks have been widely studied in various fields including medical and human computer interactions, and in driver fatigue. Automatic detection of blinks has valuable practical importance. While existing deep neural networks excel in extracting spatial features from images and demonstrate impressive performance in visual object recognition, their application for blink detection in videos on a frame-by-frame basis is suboptimal, as they only consider spatial features from single images. In this paper, we developed a spatial-temporal-aware deep learning framework that capitalizes on the rapid advancements of the existing state-of-the-art visual object recognition networks, aiming to enhance their performance specifically in blink detection. Our framework takes consecutive frames as input to extract spatial and temporal features simultaneously for better detection of eye movements. We also propose a sliding window re-sampling strategy to mitigate overfitting on training data. Extensive experimental evaluations and comparisons demonstrate the feasibility of the proposed algorithm, which delivers excellent performance for detecting blinks.

Contour detection plays an important role in both academic research and real-world applications. As the basic building block of many applications, its accuracy and efficiency highly influence the subsequent stages. In this work, we propose a novel lightweight system for contour detection that achieves state-of-the-art performance while keeps ultra-slim model size. The proposed method is built on an efficient encoder in a bottom-up/top-down fashion. Specially, we propose a novel decoder that compresses side features from an encoder and effectively decodes compact contextual information for high-accurate boundary localization. Besides, we propose a novel loss function that is able to assist a model to produce crisp object boundaries.

In this paper, we propose a new single-image dehazing method for synthetic and real-world hazy images. Based on the color attenuation prior, this proposed dehazing method improves it in two aspects. First, we estimate the atmospheric light with the haze-lines prior, which is based on the observation that pixel values of a hazy image can be modeled as lines in the RGB color space that intersects at the air-light. Second, the dynamic scattering coefficient, which is an exponential function of image depth, is proposed to replace the constant scattering coefficient. Experimental results demonstrate that the dehazed image of proposed algorithm is clearer and more natural than that of the color attenuation prior. The proposed algorithm can effectively improve the effect of dehazing.

In this paper, we present a computationally simple yet effective image recoloring method based on color harmonization. Our method permits the user to obtain recolored results interactively by rotating a harmonious template after completing color harmonization. Two main improvements are made in this paper. Firstly, we give a new strategy for finding the most harmonious scheme, in terms of finding the template which best matches the hue distribution of the input image. Secondly, in order to achieve spatially coherent harmonization, geodesic distances are used to move hues lying outside the harmonious sectors to inside them. Experiments show that our approach can produce higher-quality visually pleasing recolored images than existing methods. Moreover, our method is simple and easy to implement, and has good runtime performance.

We present style-aware image cloning, a novel image editing approach for artworks, which allows users to seamlessly insert any photorealistic or artificial objects into an artwork to create a new image that shares the same artistic style with the original artwork. To this end, a real-time image transfer algorithm is developed to stylize the cloned object according to a distance metric based on the artistic styles and semantic information. Several interactive functions, such as layering, shadowing, semantic labeling, and direction field editing, are provided to enhance the harmonization of the composite image. Extensive experimental results demonstrate the effectiveness of our method.

This paper propose a new real-time image smoothing algorithm based on true edges. We observed that the pixels marked as edge points by the well-known Canny operator are as close as possible to the center of true edges. This gives us an opportunity to judge how well edges are preserved for an edge-preserving smoothing filter. We therefore propose a nonlinear edge-preserving interpolation technique guided by the Canny true edges in an image. Our new edge-preserving smoothing algorithm smoothes high-contrast details and, at the same time, effectively preserves the true edges in the input image. In order to achieve a real-time performance, we further implement the algorithm using NVIDIA CUDA by taking advantage of the high parallelism of modern graphics hardware. The proposed algorithm smoothes low-contrast regions while effectively preserving true edges, and can also be used to produce vivid abstracted effects from realistic photographs.

This paper presents an algorithm to approximate a solid model by a hierarchical set of bounding ellipsoids having optimal shape and volume approximation errors. The ellipsoid-tree is constructed in a top-down splitting framework. Starting from the root of hierarchy the volume occupied by a given model is divided into k sub-volumes where each is approximated by a volume bounding ellipsoid and will be later subdivided into k ellipsoids for the next level in hierarchy. The difficulty for implementing this algorithm comes from how to evaluate the volume of an ellipsoid outside the given model effectively and efficiently (i.e., the outside-volume-error). A new method - analytical computation based - is presented in this paper to compute the outside-volume-error. One application of ellipsoid-tree approximation has also been given at the end of the paper.

This paper presents a novel bilateral filtering using fuzzy-median for image manipulations such as denoising and tone mapping. Our proposed bilateral filtering consists of the standard bilateral filter and the estimation of the pixel values by the fuzzy median filter. We have applied the proposed fuzzy filtering for image denoising with both the impulse and Gaussian random noise, which achieves better results than the bilateral filtering based denoising approaches, the Perona-Maliks anisotropic diffusion filter, the fuzzy vector median filter and the non-local means filter. Further, we develop the tone mapping algorithm of high dynamic range images incorporating the proposed fuzzy filtering, which does not introduce unpleasant visual halo artifacts.

This paper presents a novel approach for real-time photo style transfer. The automatic image manipulation technique is performed in the oRGB color space, which is a new color model based on the psychologically opponent color theory. We transfer color from an appropriate source image to the target image using a simple statistical analysis. In addition, we match the global luminance histogram to achieve better photographic look. Note that the whole pipeline is highly parallel, enabling a GPU-based real-time implementation. Several experimental results are shown to demonstrate the effectiveness and efficiency of the proposed method.

In this paper we present a novel tone mapping algorithm for high dynamic range (HDR) images using the retinal adaptation model (RAM). The physiological evidence suggests that the RAM is obtained by measuring intensity-response functions to flashes of light presented under varying adaptation conditions, which leads to a theoretic-sound model that can be flexibly adapted for tone reproduction. The multiplicative-subtractive process of the model can provide high quality tone mapping results for rendering the HDR images. The experimental results demonstrate that our RAM-based tone mapping approach is effective to produce pleasing results on HDR images in a wide range of real-world scenarios.

Abstract This paper presents the fireworks controller, a novel real‐time shape‐constrained fireworks animation system. We depict the shape of a firework by a 3D mesh. In order to approximate the mesh using evenly distributed points, we propose a fast point sampling method by extending the dual depth peeling algorithm. The samples are then taken as input to shape‐constrained fireworks whose physically plausible animations are based on inverse dynamics. We present a highly parallel iterative clustering algorithm to support multi‐level fireworks explosion. In order to simulate natural fuzzy fireworks, we impose extra random particles with a parallel random number generator. Several novel intuitive user interfaces are introduced to improve the usability of the system. Experimental results demonstrate the prettiness and efficiency of the proposed approach. Copyright © 2009 John Wiley & Sons, Ltd.

High dynamic range rendering attempts to take an HDR image and produce a more realistic representation on a limited range computer monitor. Although several tone mapping operators have been proposed in recent years, no evaluation has yet been undertaken to explore which operator is more suitable for hardware implementation. In this paper, we begin with our novel GPU implementations of two state-of-the-art operators in real time. Then several experimental results using eight GPU-based tone mapping operators are presented to evaluate which one is better with regard to running efficiency. Our GPU implementation of the Pattanaik operator can achieve real-time performance even on high-resolution HDR images. In addition, we believe that many real-time applications, including HDR video player and environment mapping with HDR textures in games, will benefit from our novel approach.

As ellipsoids have been employed in the collision handling of many applications in physical simulation and robotics systems, we present a novel algorithm for generating a bounding volume hierarchy (BVH) from a given model with ellipsoids as primitives. Our algorithm approximates the given model by a hierarchical set of optimized bounding ellipsoids. The ellipsoid-tree is constructed by a top-down splitting. Starting from the root of hierarchy, the volume occupied by a given model is divided into k sub-volumes where each is approximated by a volume bounding ellipsoid. Recursively, each sub-volume is then subdivided into ellipsoids for the next level in the hierarchy. The k ellipsoids at each hierarchy level for a sub-volume bounding is generated by a bottom-up algorithm - simply, the sub-volume is initially approximated by m spheres (m » k), which will be iteratively merged into k volume bounding ellipsoids and globally optimized to minimize the approximation error. Benefited from the anisotropic shape of primitives, the ellipsoid-tree constructed in our approach gives tighter volume bound and higher shape fidelity than another widely used BVH, sphere-tree.