We present a novel text-driven approach for light field (LF) synthesis. Existing methods typically generate LFs from given images, requiring users to find reference images, which makes it difficult to construct the desired scene directly and limits scene diversity. Moreover, existing methods are mainly designed for limited baselines from training datasets, making it difficult to implement various viewpoint changes and consequently limiting the flexibility of motion. In contrast, our method directly synthesizes LFs from user-provided text descriptions by leveraging the scene understanding capabilities of a multi-modal large language model (LLM) and the generative power of a diffusion model. Given a text prompt describing the desired LF, the multimodal LLM extracts relevant information for LF synthesis, which then guides a diffusion model to produce diverse scenes and motions. This approach enables LF synthesis even with a pre-trained model not initially designed for this purpose, requiring only minimal fine-tuning. The proposed framework enables visually diverse LF synthesis with only text input. Experimental results demonstrate that the synthesized LFs exhibit geometric consistency and achieve advanced synthesis quality compared to existing methods.

Fast 3D clothed human reconstruction from monocular video remains a significant challenge in computer vision, particularly in balancing computational efficiency with reconstruction quality. Current approaches are either focused on static image reconstruction but too computationally intensive, or achieve high quality through per-video optimization that requires minutes to hours of processing, making them unsuitable for real-time applications. To this end, we present TemPoFast3D, a novel method that leverages temporal coherency of human appearance to reduce redundant computation while maintaining reconstruction quality. Our approach is a "plug-and play" solution that uniquely transforms pixel-aligned reconstruction networks to handle continuous video streams by maintaining and refining a canonical appearance representation through efficient coordinate mapping. Extensive experiments demonstrate that TemPo-Fast3D matches or exceeds state-of-the-art methods across standard metrics while providing high-quality textured reconstruction across diverse pose and appearance, with a maximum speed of 12 FPS.

In facial image synthesis, the style of the source image is converted using a reference image, or images with different styles are synthesized by each attribute using a facial attribute segmentation map. However, previous works cannot deal with the fine areas because the style is changed mostly in large areas such as hair, eyes, and mouth. To overcome the limitation, we propose a novel method of synthesizing a facial image with micro-level facial elements. A deep learning-based high-resolution image synthesis model is employed after generating a label image from the face RGB image through skin micro-element segmentation and face attribute segmentation. In the process of generating a label image for synthesizing skin micro-elements, we propose a technique for controlling skin micro-elements, enabling the generation of various label images from a single face label image. Throughout the proposed method, the areas of skin micro-elements can be edited and different skin types can be simulated. The experimental results show that the generated face is significantly improved by applying the proposed method. Moreover, various faces can be synthesized by changing the types and stages of skin micro-elements.

Existing face restoration models have relied on general assessment metrics that do not consider the characteristics of facial regions. Recent works have therefore assessed their methods using human studies, which is not scalable and involves significant effort. This paper proposes a novel face-centric metric based on an adversarial framework where a generator simulates face restoration and a discriminator assesses image quality. Specifically, our per-pixel discriminator enables interpretable evaluation that cannot be provided by traditional metrics. Moreover, our metric emphasizes facial primary regions considering that even minor changes to the eyes, nose, and mouth significantly affect human cognition. Our face-oriented metric consistently surpasses existing general or facial image quality assessment metrics by impressive margins. We demonstrate the generalizability of the proposed strategy in various architectural designs and challenging scenarios. Interestingly, we find that our IFQA can lead to performance improvement as an objective function. The code and models are available at https://github.com/VCLLab/IFQA.

Recent 3D body reconstruction works are achieving state-of-the-art performances. Each of the prior works applied specific modifications to their respective modules, allowing the ability to show plausible predicted 3D body poses and shapes to human eyes. Unfortunately, those outputs may sometimes be far from the correct position. In contrast to these works, we took a different perspective on how to re-improve this limitation. Without any addition or modification at the module level, we propose a test-time adaptation strategy that fine-tunes the module directly. Our approach is inspired by the science of vaccination that leverages surrogate information, which is helpful and not harmful in improving the human immune system. This notion is translated to our adaptation strategy by fine-tuning the surrogate 3D body module using reliable virtual data. In doing so, the proposed work can revisit the prior state-of-the-art works and improve their performances directly in the test phase. The experimental results demonstrate our strategy's ability to straightforwardly improve the prior works, even with fast adaptation capability.

Face age transformation aims to synthesize past or future face images by reflecting the age factor on given faces. Ideally, this task should synthesize natural-looking faces across various age groups while maintaining identity. However, most of the existing work has focused on only one of these or is difficult to train while unnatural artifacts still appear. In this work, we propose Re-Aging GAN (RAGAN), a novel single framework considering all the critical factors in age transformation. Our framework achieves state-of-the-art personalized face age transformation by compelling the input identity to perform the self-guidance of the generation process. Specifically, RAGAN can learn the personalized age features by using high-order interactions between given identity and target age. Learned personalized age features are identity information that is recalibrated according to the target age. Hence, such features encompass identity and target age information that provides important clues on how an input identity should be at a certain age. Experimental result shows the lowest FID and KID scores and the highest age recognition accuracy compared to previous methods. The proposed method also demonstrates the visual superiority with fewer artifacts, identity preservation, and natural transformation across various age groups.

Commercially available light field cameras have difficulty in capturing 5D (4D + time) light field videos. They can only capture still light field images or are excessively expensive for normal users to capture the light field video. To tackle this problem, we propose a deep learning-based method for synthesizing a light field video from a monocular video. We propose a new synthetic light field video dataset that renders photorealistic scenes using Unreal Engine because no light field video dataset is available. The proposed deep learning framework synthesizes the light field video with a full set (9 × 9) of sub-aperture images from a normal monocular video. The proposed network consists of three sub-networks, namely, feature extraction, 5D light field video synthesis, and temporal consistency refinement. Experimental results show that our model can successfully synthesize the light field video for synthetic and real scenes and outperforms the previous frame-by-frame method quantitatively and qualitatively.

360° images represent scenes captured in all possible viewing directions and enable viewers to navigate freely around the scene thereby providing an immersive experience. Conversely, conventional images represent scenes in a single viewing direction with a small or limited field of view (FOV). As a result, only certain parts of the scenes are observed, and valuable information about the surroundings is lost. In this paper, a learning-based approach that reconstructs the scene in 360° × 180°from a sparse set of conventional images (typically 4 images) is proposed. The proposed approach first estimates the FOV of input images relative to the panorama. The estimated FOV is then used as the prior for synthesizing a high-resolution 360°panoramic output. The proposed method overcomes the difficulty of learning-based approach in synthesizing high resolution images (up to 512×1024). Experimental results demonstrate that the proposed method produces 360° panorama with reasonable quality. Results also show that the proposed method outperforms the alternative method and can be generalized for non-panoramic scenes and images captured by a smartphone camera.

In recent decades, 3D morphable model (3DMM) has been commonly used in image-based photorealistic 3D face reconstruction. However, face images are often corrupted by serious occlusion by non-face objects including eyeglasses, masks, and hands. Such objects block the correct capture of landmarks and shading information. Therefore, the reconstructed 3D face model is hardly reusable. In this paper, a novel method is proposed to restore de-occluded face images based on inverse use of 3DMM and generative adversarial network. We utilize the 3DMM prior to the proposed adversarial network and combine a global and local adversarial convolutional neural network to learn face de-occlusion model. The 3DMM serves not only as geometric prior but also proposes the face region for the local discriminator. Experiment results confirm the effectiveness and robustness of the proposed algorithm in removing challenging types of occlusions with various head poses and illumination. Furthermore, the proposed method reconstructs the correct 3D face model with de-occluded textures.

The popularity of parallax-based image processing is increasing while in contrast early works on recovering sharp light field from its blurry input (deblurring) remain stagnant. State-of-the-art blind light field deblurring methods suffer from several problems such as slow processing, reduced spatial size, and simplified motion blur model. In this paper, we solve these challenging problems by proposing a novel light field recurrent deblurring network that is trained under 6 degree-of-freedom camera motion-blur model. By combining the real light field captured using Lytro Illum and synthetic light field rendering of 3D scenes from UnrealCV, we provide a large-scale blurry light field dataset to train the network. The proposed method outperforms the state-of-the-art methods in terms of deblurring quality, the capability of handling full-resolution, and a fast runtime.

Depth estimation is essential in many light field applications. Numerous algorithms have been developed using a range of light field properties. However, conventional data costs fail when handling noisy scenes in which occlusion is present. To address this problem, we introduce a light field depth estimation method that is more robust against occlusion and less sensitive to noise. Two novel data costs are proposed, which are measured using the angular patch and refocus image, respectively. The constrained angular entropy cost (CAE) reduces the effects of the dominant occluder and noise in the angular patch, resulting in a low cost. The constrained adaptive defocus cost (CAD) provides a low cost in the occlusion region, while also maintaining robustness against noise. Integrating the two data costs is shown to significantly improve the occlusion and noise invariant capability. Cost volume filtering and graph cut optimization are applied to improve the accuracy of the depth map. Our experimental results confirm the robustness of the proposed method and demonstrate its ability to produce high-quality depth maps from a range of scenes. The proposed method outperforms other state-of-the-art light field depth estimation methods in both qualitative and quantitative evaluations.

The external example-driven single image super-resolution (SISR) method that uses a deep convolutional neural network (CNN) has exhibited superior performance as compared to previously developed SISR methods. However, the advantages of jointly using external and internal examples on a deep CNN framework have not been sufficiently investigated. In this letter, we present a novel method for single image super-resolution by exploiting a complementary relation between external and internal example-based SISR methods. The proposed deep CNN model consists of two subnetworks, a global residual network and a self-residual network, to utilize the advantages of both external and internal examples. In contrast with conventional joint SISR methods, the proposed method is the first deep CNN-based SISR method that does not require a retraining process, which tends to be inefficient. The proposed method outperformed existing methods in both quantitative and qualitative evaluations.

The image quality of an in-vehicle black box camera is often degraded by the reflections of internal objects, dirt, and dust on the windshield. In this paper, we propose a novel algorithm that simultaneously removes the reflections and small dirt artifacts from in-vehicle black box videos under fast forward camera motion. The algorithm exploits the spatiotemporal coherence of the reflection and dirt, which remain stationary relative to the fast-moving background. Unlike previous algorithms, the algorithm first separates stationary reflection and then restores the background scene. To this end, we propose an average image prior, thereby imposing spatiotemporal coherence. The separation model is a two-layer model composed of stationary and background layers, where different gradient sparsity distributions are utilized in a region-based manner. Motion compensation in postprocessing is proposed to alleviate layer jitter due to vehicle vibrations. In evaluation experiments, the proposed algorithm successfully extracts the stationary layer from several real and synthetic black box videos.

Light field depth estimation is an essential part of many light field applications. Numerous algorithms have been developed using various light field characteristics. However, conventional methods fail when handling noisy scene with occlusion. To remedy this problem, we present a light field depth estimation method which is more robust to occlusion and less sensitive to noise. Novel data costs using angular entropy metric and adaptive defocus response are introduced. Integration of both data costs improves the occlusion and noise invariant capability significantly. Cost volume filtering and graph cut optimization are utilized to improve the accuracy of the depth map. Experimental results confirm that the proposed method is robust and achieves high quality depth maps in various scenes. The proposed method outperforms the state-of-the-art light field depth estimation methods in qualitative and quantitative evaluation.

The in-vehicle black box camera (dashboard camera) has become a popular device in many countries for security monitoring and event capturing. The readability of video content is the most critical matter, however, the content is often degraded due to the windscreen reflection of objects inside. In this paper, we propose a novel method to remove the reflection on the windscreen from in-vehicle black box videos. The method exploits the spatio-temporal coherence of reflection, which states that a vehicle is moving forward while the reflection of the internal objects remains static. The average image prior is proposed by imposing a heavy-tail distribution with a higher peak to remove the reflection. The two-layered scene composed of reflection and background layers is the basis of the separation model. A non-convex cost function is developed based on this property and optimized in a fast way in a half quadratic form. Experimental results demonstrate that the proposed approach successfully separates the reflection layer in several real black box videos.

In this paper, we present a joint iterative anaglyph stereo matching and colorization framework for obtaining a set of disparity maps and colorized images. Conventional stereo matching algorithms fail when addressing anaglyph images that do not have similar intensities on their two respective view images. To resolve this problem, we propose two novel data costs using local color prior and reverse intensity distribution factor for obtaining accurate depth maps. To colorize an anaglyph image, each pixel in one view is warped to another view using the obtained disparity values of non-occluded regions. A colorization algorithm using optimization is then employed with additional constraint to colorize the remaining occluded regions. Experimental results confirm that the proposed unified framework is robust and produces accurate depth maps and colorized stereo images.

This letter proposes a novel warping-based method for the content-driven retargeting of stereoscopic images. Conventional algorithms in single image retargeting generally do not consider the scene depth saliency and disparity consistency when applied independently to left and right images . Therefore, the salient region and stereoscopic correlation of independently retargeted images can become corrupted. On the other hand, the proposed algorithm retains the stereo consistency of the retargeted images by matching the vertices of the grid and preserving the correspondence between them. Vertex disparity is propagated by a GPU interpolation to construct a sparse disparity map. To improve the capability of preserving visually important regions, the sparse disparity is used in conjunction with the image gradient in a saliency map computation. The experimental results show that the proposed method retains the correct disparity, while the salient objects remain undistorted in the retargeted stereoscopic image pairs.

We construe key factors in design and evaluation of image processing algorithms on the massive parallel graphics processing units (GPUs) using the compute unified device architecture (CUDA) programming model. A set of metrics, customized for image processing, is proposed to quantitatively evaluate algorithm characteristics. In addition, we show that a range of image processing algorithms map readily to CUDA using multiview stereo matching, linear feature extraction, JPEG2000 image encoding, and nonphotorealistic rendering (NPR) as our example applications. The algorithms are carefully selected from major domains of image processing, so they inherently contain a variety of subalgorithms with diverse characteristics when implemented on the GPU. Performance is evaluated in terms of execution time and is compared to the fastest host-only version implemented using OpenMP. It is shown that the observed speedup varies extensively depending on the characteristics of each algorithm. Intensive analysis is conducted to show the appropriateness of the proposed metrics in predicting the effectiveness of an application for parallel implementation.

For motion deblurring from a single blurred image, it is of utmost importance to estimate the blur kernel accurately. In this paper, we propose a new anisotropic regularization blur kernel estimation algorithm which preserves the point spread function (PSF) path while keeping the properties of motion PSF into seeking the solution of the blur kernel. In order to preserve the PSF path, the nonquadratic regularization and the refinement of the blur kernel are incorporated in the iterative process to improve the precision of the blur kernel. A single motion blurred image can be restored well after the accurate motion PSF is estimated. Experimental results demonstrate that the proposed approach is efficient and effective to reduce motion blur with arbitrary direction in a single image.

In this paper, we present a framework for efficiently integrating programming resources of both GPU and CPU. We introduce an object oriented framework for GPGPU-based image processing. We illustrate a set of classes exploiting the design and programming advantages of an object oriented language, such as code reusability/extensibility, flexibility, information hiding, and complexity hiding. This class structure is supplemented with shader (GLSL) and kernel (CUDA) programming to facilitate full functionality. We demonstrate the potential of our approach with application scenarios and discuss the framework's performance in terms of programming effort, execution overhead, and speedup factor achieved over CPU.

The advent of GPUs with programmable shaders on handheld devices has motivated embedded application developers to utilize GPU to offload computationally intensive tasks and relieve the burden from embedded CPU. In this work, we propose an image processing toolkit on handheld GPU with programmable shaders using OpenGL ES 2.0 API. By using the image processing toolkit, we show that a range of image processing algorithms map readily to handheld GPU. We employ real-time video scaling, cartoon-style non-photorealistic rendering, and Harris corner detector as our example applications. In addition, we propose techniques to achieve increased performance with optimized shader design and efficient sharing of GPU workload between vertex and fragment shaders. Performance is evaluated in terms of frames per second at varying video stream resolution.

In this paper, we explore the key factors in the design and implementation of visual computing (image processing and computer vision) algorithms on the massive parallel GPU (graphics processing units). The goal of the exploration is to provide common perspective and guidelines of using GPU for visual computing applications. We have selected three nontrivial applications (multiview stereo matching, linear feature extraction, and JPEG2000 image encoding) for the benchmarks, which show different characteristics in GPU parallel computing. Intensive analysis is performed to evaluate the characteristic of each algorithm and its effect on the performance. Based on this, we draw general guidelines of using GPU for the visual computing algorithms.

Abstract This paper addresses an image‐based method for modeling 3D objects with curved surfaces based on the non‐uniform rational B‐splines (NURBS) representation. The user fits the feature curves on a few calibrated images with 2D NURBS curves using the interactive user interface. Then, 3D NURBS curves are constructed by stereo reconstruction of the corresponding feature curves. Using these as building blocks, NURBS surfaces are reconstructed by the known surface building methods including bilinear surfaces, ruled surfaces, generalized cylinders, and surfaces of revolution. In addition to them, we also employ various advanced techniques, including skinned surfaces, swept surfaces, and boundary patches. Based on these surface modeling techniques, it is possible to build various types of 3D shape models with textured curved surfaces without much effort. Copyright © 2007 John Wiley & Sons, Ltd.

This paper describes a new family of three-dimensional (3-D) representations for computer graphics and animation, called depth image-based representations (DIBR), which have been adopted into MPEG-4 Part16: Animation Framework eXtension (AFX). Idea of the approach is to build a compact and photorealistic representation of a 3-D object or scene without using polygonal mesh. Instead, images accompanied by depth values for each pixel are used. This type of representation allows us to build and render novel views of objects and scene with an interactive rate. There are many different methods for the image-based rendering with depths, and the DIBR format is designed to efficiently represent the information necessary for such methods. The main formats of the DIBR family are SimpleTexture (an image together with depth array), PointTexture (an image with multiple pixels along each line of sight), and OctreeImage (octree-like data structure together with a set of images containing viewport parameters). In order to store and transmit the DIBR object, we develop a compression algorithm and bitstream format for OctreeImage representation.

Abstract Mesh simplification is the process of reducing the number of triangles in a mesh representation of object surface. For a given level of detail or error tolerance, the conventional mesh simplification algorithms maximize the edge length globally, without explicitly considering local object shape. In this paper, we present a shape‐adaptive mesh simplification algorithm that locally maximizes edge length, depending on local shape. The proposed algorithm achieves shape‐adaptive simplification by iteratively maximizing edges between vertices, based on comparison with the ‘optimal’ edge lengths derived from local directional curvatures for a given error tolerance. Edge‐based processing facilitates the local shape adaptation and preserves sharp features. Experimental results demonstrate the efficacy of the proposed algorithm, by showing good visual quality and extremely small approximation error. Copyright © 2003 John Wiley & Sons, Ltd.

In this paper, we propose a new efficient topology estimation algorithm to construct a multiresolution polygonal mesh from measured three-dimensional (3-D) range data. The topology estimation problem is defined under the constraints of cognition, compactness, and regularity, and the algorithm is designed to be applied to either a cloud of points or a dense mesh. The proposed algorithm initially segments the range data into a finite number of Voronoi patches using the K-means clustering algorithm. Each patch is then approximated by an appropriate polygonal and eventually a triangular mesh model. In order to improve the equiangularity of the mesh, we employ a dynamic mesh model, in which the mesh finds its equilibrium state adaptively, according to the equiangularity constraint. Experimental results demonstrate that satisfactory equiangular triangular mesh models can be constructed rapidly at various resolutions, while yielding tolerable modeling error.

We describe a novel depth image-based representation (DIBR) that has been adopted into the MPEG-4 animation framework extension (AFX). The idea of this approach is to build a compact representation of a 3D object or scene without storing the geometry information in traditional polygonal form. The main formats of the DIBR family are simple texture (an image together with depth array), point texture (a view of a scene from a single input camera but with multiple pixels along each line of sight), and octree image (octree data structure together with a set of images and their viewport parameters). The designed node specifications and rendering algorithms are addressed. The experimental results show the efficacy and fidelity of the proposed approach.

Mesh simplification is the process of reducing the number of triangles in a mesh representation of object surface. For a given level of detail or error tolerance, the conventional mesh simplification algorithms maximize the edge length globally, without explicitly considering the local object shape. In this paper we present a shape-adaptive mesh simplification algorithm that locally maximizes the edge length, depending on the local shape. The proposed algorithm achieves shape-adaptive simplification by iteratively maximizing edges between vertices, based on a comparison with the "optimal" edge lengths derived from local directional curvatures for a given error tolerance.

We address a new algorithm for recognition and reconstruction of 3D polyhedral objects, based on perceptual grouping and graph search technique. Perceptual grouping is performed in a model-based framework, in which decision tree classifier is employed for learning and retrieving geometric information of the 3D model object. On the other hand, in order to extract the polygonal patch structure, initial grouping result is represented by a Gestalt graph. Polygonal patch hypotheses are then generated by graph search and verified by the consistency test with the model. In the experiments, it is shown that the model-based grouping reduces the number of the generated hypotheses efficiently, and furthermore, robust recognition and reconstruction are achieved by means of the graph search technique.

We address a new perceptual grouping algorithm for aerial images, which employs a decision tree classifier and hierarchical multilevel grouping strategy in a bottom-up fashion. In our approach, grouping is performed perceptually on 3D features extracted from 2D images, in which the gestalt principles including collinearity, parallelism and L-typed convergence are encoded by the decision tree learning technique. The decision tree is constructed using training samples obtained from the given 3D reference model. Then, each pair of the extracted 3D line features of an input image is classified into one of the learned gestalt primitives. On the other hand, in multilevel grouping procedure, grouping of collated features are performed from lower to higher level, yielding the structured target model. In order to evaluate the proposed algorithm, experiments are carried out on RADIUS model board images. The results show that grouping is performed effectively to extract man-made structures in aerial images.

We propose an algorithm to produce a 3-D CAD model from a set of range data, based on non-uniform rational B-splines (NURBS) surface fitting technique. Our goal is to construct continuous geometric models, assuming that the topology of surface is unknown. In our approach, a divide-and-conquer strategy is adopted, in which the whole range data is partitioned into surface patches. Each patch is sequentially processed to form the quadrilateral face model, which is used to construct the NURBS patch network. Experiments are carried out to evaluate the performance of the proposed algorithm. It is shown that the continuous 3-D model is successfully generated automatically with tolerable computational complexity.