3D human pose estimation from 2D keypoint observation has been used in many human-centered computer vision applications. In this work, we tackle the task by formulating a novel grid representation learning paradigm that relies on grid convolution (GridConv), mimicking the wisdom of regular convolution operations in image space. GridConv is defined based on Semantic Grid Transformation (SGT) which leverages a binary assignment matrix to map standard skeleton 2D pose onto a regular weave-like grid pose joint by joint. We provide two ways to implement SGT: handcrafted and learnable SGT. Surprisingly, both designs turn out to achieve promising results and the learnable one is better, demonstrating the great potential of this new lifting representation learning formulation. To improve the ability of GridConv to encode contextual cues, we introduce an attention module over the convolutional kernel, making grid convolution operations input-dependent, spatial-aware and grid-specific. Besides our spatial grid lifting network for single-frame input, we also present a spatial-temporal grid lifting network for video-based input, which relies on an efficient multi-scale grid learning strategy to encode spatial and temporal joint variations. Extensive experiments demonstrate that the proposed grid lifting network outperforms existing approaches by remarkable margins on Human3.6M and MPI-INF-3DHP datasets. Our grid lifting networks also exhibit good generalization ability across three other keypoint-based tasks: 3D hand pose estimation, head pose estimation, and action recognition.

Few-shot semantic segmentation (FSS) aims to segment unseen-category objects given only a few annotated samples. Although significant progress has been made in the field of FSS, selecting an appropriate feature matching method remains a challenge. Traditional prototype-based methods can preserve high-level semantic features, but they tend to lose detailed information. On the other hand, pixel-level comparison methods retain fine-grained details but are vulnerable to distractors and noise, leading to poor robustness. To address these issues, this paper proposes a target-agnostic object-based method. Specifically, we propose a set of learnable "object queries" to extract object features, which preserve both high-level semantic information and fine-grained details. Additionally, during the training phase, we exploit the prior knowledge of foreground and background embedded in the samples to enhance the model's performance. In the inference phase, the model utilizes both the support set and the learned prior knowledge to perform segmentation tasks, mitigating the data distribution bias caused by limited samples. Extensive experiments on benchmark datasets demonstrate that our method outperforms state-of-the-art approaches in both accuracy and robustness. Code is available at https://github.com/wenbo456/OTBNet.

Although few-shot segmentation (FSS) methods have achieved remarkable results, there remain challenges associated with the limited number of support samples.i)The objects in support and query images may have substantially different appearances even though they belong to the same category, which is known as the prototype bias problem.ii)Most methods neglect the background information, especially the query background during the inference stage. To address these problems, we propose DPRNet, a novel network with dual branch of foreground and background prototype refinement modules. Specifically, we first present a Variational Feature Semantic Enhancement (VFSE) module, in which we refine the object prototype with a variational autoencoder and word-text labels. In this way, the biased class-wise prototype caused by the limited support samples can be aligned, achieving better performance. Second, we design a Background Prototype Refinement (BPR) module that effectively explores the potential information in the background for both the support and query images. More importantly, it is designed to generate online predictions of the query background during the training stage to fully mimic the inference stage. These advancements enhance the robustness and generalizability of our method, and the results of experiments demonstrate its effectiveness. In the 1-way 5-shot setting on PASCAL-$5^{i}$, our method achieves a mean-IoU improvement of 1.59% over the competing method.

This is a corrigendum for the article “Wavelet Fluids” published in ACM Trans. Graph. 44, 6, Article 270 (December 2025), 17 pages.

Joint-embedding predictive architectures (JEPAs) have shown substantial promise in self-supervised representation learning, yet their application in generative modeling remains underexplored. Conversely, diffusion models have demonstrated significant efficacy in modeling arbitrary probability distributions. In this paper, we introduce Denoising with a Joint-Embedding Predictive Architecture (D-JEPA), pioneering the integration of JEPA within generative modeling. By recognizing JEPA as a form of masked image modeling, we reinterpret it as a generalized next-token prediction strategy, facilitating data generation in an auto-regressive manner. Furthermore, we incorporate diffusion loss to model the per-token probability distribution, enabling data generation in a continuous space. We also adapt flow matching loss as an alternative to diffusion loss, thereby enhancing the flexibility of D-JEPA. Empirically, with increased GFLOPs, D-JEPA consistently achieves lower FID scores with fewer training epochs, indicating its good scalability. Our base, large, and huge models outperform all previous generative models across all scales on ImageNet conditional generation benchmarks. Beyond image generation, D-JEPA is well-suited for other continuous data modeling, including video and audio.

This paper introduces a novel wavelet-based framework for simulating both single-phase (e.g., smoke) and two-phase (e.g., bubbly water) flows, featuring unified boundary condition handling for free surfaces and solid obstacles. In liquid simulations, conventional pressure projection methods enforce zero-pressure Dirichlet conditions at free surfaces by solving a simplified pressure Poisson equation. However, these approaches neglect air-phase incompressibility, leading to artificial bubble collapse. Stream function methods overcome this limitation by solving a density-variable vector potential Poisson equation, ensuring incompressibility in both simulated and unsimulated regions while maintaining divergence-free liquid phases independent of solver accuracy. Yet, they triple the linear system's dimensionality and exhibit poor convergence near solid boundaries. The fundamental limitation of both methods stems from their governing equations: singularities emerge as density approaches extreme values. The pressure Poisson equation becomes ill-conditioned when density nears zero (air phase), compromising air-phase incompressibility, while the vector potential equation degrades as density approaches infinity (solid phase), impeding solid-boundary convergence. To address these singularities, we first propose a novel decomposition where zero and infinite densities are well-defined. We then reformulate this decomposition as a fixed-point iteration using density-agnostic curl-free and divergence-free projections, eliminating the need for linear system solves. The error equation is derived, and a necessary and sufficient convergence condition is established. Building on this, we develop an iterative algorithm that efficiently solves the fixed-point problem through alternating wavelet-based non-orthogonal curl-free and divergence-free projections. Additionally, we investigate orthogonal curl-free projections (e.g., Fourier methods) and their complementary divergence-free counterparts, providing a comprehensive comparison between wavelet and Fourier approaches. Our method simultaneously computes pressure and stream functions, retaining the incompressibility benefits of stream function approaches while resolving their computational inefficiencies and solid-boundary convergence issues. Experiments demonstrate our framework's ability to efficiently simulate complex two-phase phenomena, such as the glugging effect during water pouring and multi-liquid-region interactions across zero-density air.

Prior research has demonstrated the efficacy of balanced trees as spatially adaptive grids for large-scale simulations. However, state-of-the-art methods for balanced tree construction are restricted by the iterative nature of the ripple effect, thus failing to fully leverage the massive parallelism offered by modern GPU architectures. We propose to reframe the construction of balanced trees as a process to merge N -balanced Minimum Spanning Trees ( N -balanced MSTs) generated from a collection of seed points. To ensure optimal performance, we propose a stack-free parallel strategy for constructing all internal nodes of a specified N -balanced MST. This approach leverages two 32-bit integer registers as buffers rather than relying on an integer array as a stack during construction, which helps maintain balanced workloads across different GPU threads. We then propose a dynamic update algorithm utilizing refinement counters for all internal nodes to enable parallel insertion and deletion operations of N -balanced MSTs. This design achieves significant efficiency improvements compared to full reconstruction from scratch, thereby facilitating fluid simulations in handling dynamic moving boundaries. Our approach is fully compatible with GPU implementation and demonstrates up to an order-of-magnitude speedup compared to the state-of-the-art method [Wang et al. 2024]. The source code for the paper is publicly available at https://github.com/peridyno/peridyno.

The convergence of large language models (LLMs) and virtual reality (VR) technologies has led to significant breakthroughs across multiple domains, particularly in healthcare and medicine. Owing to its immersive and interactive capabilities, VR technology has demonstrated exceptional utility in surgical simulation, rehabilitation, physical therapy, mental health, and psychological treatment. By creating highly realistic and precisely controlled environments, VR not only enhances the efficiency of medical training but also enables personalized therapeutic approaches for patients. The convergence of LLMs and VR extends the potential of both technologies. LLM-empowered VR can transform medical education through interactive learning platforms and address complex healthcare challenges using comprehensive solutions. This convergence enhances the quality of training, decision-making, and patient engagement, paving the way for innovative healthcare delivery. This study aims to comprehensively review the current applications, research advancements, and challenges associated with these two technologies in healthcare and medicine. The rapid evolution of these technologies is driving the healthcare industry toward greater intelligence and precision, establishing them as critical forces in the transformation of modern medicine.

Despite significant progress in utilizing pre-trained text-to-image diffusion models to guide the creation of 3D scenes, these methods often struggle to generate scenes that are sufficiently realistic, leading to "neural scene degeneration". In this work, we propose a new 3D scene generation model called Real3D. Specifically, Real3D designs a pipeline from a NeRF-like implicit renderer to a tetrahedrons-based explicit renderer, greatly improving the neural network's ability to generate various neural scenes. Moreover, Real3D introduces an additional discriminator to prevent neural scenes from falling into undesirable local optima, thus avoiding the degeneration phenomenon. Our experimental results demonstrate that Real3D outperforms all existing state-of-the-art text-to-3D generation methods, providing valuable insights to facilitate the development of learning-based 3D scene generation approaches.

The large-scale visual pretraining has significantly improve the performance of large vision models. However, we observe the low FLOPs pitfall that the existing low-FLOPs models cannot benefit from large-scale pretraining. In this paper, we introduce a novel design principle, termed ParameterNet, aimed at augmenting the number of parameters in large-scale visual pretraining models while minimizing the increase in FLOPs. We leverage dynamic convolutions to incorporate additional parameters into the networks with only a marginal rise in FLOPs. The ParameterNet approach allows low-FLOPs networks to take advantage of large-scale visual pretraining. Furthermore, we extend the ParameterNet concept to the language domain to enhance inference results while preserving inference speed. Experiments on the large-scale ImageNet-22K have shown the superiority of our ParameterNet scheme. For example, ParameterNet-600M can achieve higher accuracy than the widely-used Swin Transformer (81.6% vs. 80.9%) and has much lower FLOPs (0.6G vs. 4.5G). The code will be released at https://parameternet.github.io/.

Recently, video object segmentation (VOS) networks typically use memory-based methods: for each query frame, the mask is predicted by space-time matching to memory frames. Despite these methods having superior performance, they suffer from two issues: 1) Challenging data can destroy the space-time coherence between adjacent video frames. 2) Pixel-level matching will lead to undesired mismatching caused by the noises or distractors. To address the aforementioned issues, we first propose to generate an auxiliary frame between adjacent frames, serving as an implicit short-temporal reference for the query one. Next, we learn a prototype for each video object and prototype-level matching can be implemented between the query and memory. The experiment demonstrated that our network outperforms the state-of-the-art method on the DAVIS 2017, achieving a ℐ&ℱ score of 86.4%, and attains a competitive result 85.0% on YouTube VOS 2018. In addition, our network exhibits a high inference speed of 32+ FPS.

Abstract We introduce an efficient technique for recovering the vector potential in wavelet space to simulate pointwise incompressible fluids. This technique ensures that fluid velocities remain divergence‐free at any point within the fluid domain and preserves local volume during the simulation. Divergence‐free wavelets are utilized to calculate the wavelet coefficients of the vector potential, resulting in a smooth vector potential with enhanced accuracy, even when the input velocities exhibit some degree of divergence. This enhanced accuracy eliminates the need for additional computational time to achieve a specific accuracy threshold, as fewer iterations are required for the pressure Poisson solver. Additionally, in 3D, since the wavelet transform is taken in‐place, only the memory for storing the vector potential is required. These two features make the method remarkably efficient for recovering vector potential for fluid simulation. Furthermore, the method can handle various boundary conditions during the wavelet transform, making it adaptable for simulating fluids with Neumann and Dirichlet boundary conditions. Our approach is highly parallelizable and features a time complexity of O(n), allowing for seamless deployment on GPUs and yielding remarkable computational efficiency. Experiments demonstrate that, taking into account the time consumed by the pressure Poisson solver, the method achieves an approximate 2x speedup on GPUs compared to state‐of‐the‐art vector potential recovery techniques while maintaining a precision level of 10−6 when single float precision is employed. The source code of ‘Wavelet Potentials’ can be found in https://github.com/yours321dog/WaveletPotentials .

In recent years, small object detection has been widely applied to aerial scenes. Although existing small object detection algorithms have achieved significant success, it remains challenging to ensure an acceptable processing speed and detection accuracy simultaneously in high-resolution images. In this work, we propose a novel framework called SliNet, a slicingaided learning network with a SPPFCSPC Block and a ContentAware ReAssembly of Features (CARAFE) Block merged. Since aerial images usually have high resolutions and small detection targets, we slice images into smaller overlapping patches and integrate SPPFCSPC and CARAFE to find attention region with larger receptive field in dense-object scenes. Experimental results on dataset VisDrone-DET2019 show that the SliNet achieves competitive performance and obtains faster detection speed due to the decrease of computational cost. On the VisDrone2019- DET-val dataset, we attain a mAP score of 46.4% with a mAP50 score of 67.1%, which is better than the state-of-the-art models, demonstrating the superiority of our approach

Recently, memory-based methods have exhibited remarkable performance in Video Object Segmentation (VOS) by employing non-local pixel-wise matching between the query and memory. Nevertheless, these methods suffer from two limitations: 1) Non-local pixel-wise matching can result in the incorrect segmentation of background distractor objects, and 2) memory features with substantial temporal redundancy consume significant computing resources and reduce the inference speed. To address the limitations, we first propose a local attention mechanism to suppress background features, and we introduce a novel training framework based on contrast learning to ensure the network learns reliable and robust pixel-wise correspondence between query and memory. We adaptively determine whether to update the memory based on the variation of foreground objects. Next, we propose a dynamic memory bank, which utilizes a lightweight and differentiable soft modulation gate to determine the number of memory features to remove along the temporal dimension. This allows efficient and flexible management of memory features. Our network achieves competitive results (e.g., 92.1% on DAVIS 2016 val, 87.6%/81.3% on DAVIS 2017 val/test, 87.0% on YouTube-VOS 2018 val) compared with the state-of-the-art methods while maintaining a faster inference speed of 25+FPS. Moreover, our network demonstrates a favorable balance between performance and speed when dealing with the long-time video dataset.

Few-shot semantic segmentation aims to segment novel-class objects in a query image with only a few annotated examples in support images. Although progress has been made recently by combining prototype-based metric learning, existing methods still face two main challenges. First, various intra-class objects between the support and query images or semantically similar inter-class objects can seriously harm the segmentation performance due to their poor feature representations. Second, the latent novel classes are treated as the background in most methods, leading to a learning bias, whereby these novel classes are difficult to correctly segment as foreground. To solve these problems, we propose a dual-branch learning method. The class-specific branch encourages representations of objects to be more distinguishable by increasing the inter-class distance while decreasing the intra-class distance. In parallel, the class-agnostic branch focuses on minimizing the foreground class feature distribution and maximizing the features between the foreground and background, thus increasing the generalizability to novel classes in the test stage. Furthermore, to obtain more representative features, pixel-level and prototype-level semantic learning are both involved in the two branches. The method is evaluated on PASCAL-5i1-shot, PASCAL-5i5-shot, COCO-20i1-shot, and COCO-20i5-shot, and extensive experiments show that our approach is effective for few-shot semantic segmentation despite its simplicity.

Creating visualizations of multiple volumetric density fields is demanding in virtual reality (VR) applications, which often include divergent volumetric density distributions mixed with geometric models and physics-based simulations. Real-time rendering of such complex environments poses significant challenges for rendering quality and performance. This article presents a novel scheme for efficient real-time rendering of varying translucent volumetric density fields with global illumination (GI) effects on high-resolution binocular VR displays. Our scheme proposes creative solutions to address three challenges involved in the target problem. First, to tackle the doubled heavy workloads of binocular ray marching, we explore the anti-aliasing principles and more advanced potentials of ray marching on interior cube-map faces, and propose a coupled ray-marching technique that converges to multi-resolution cube maps with interleaved adaptive sampling. Second, we devise a fully dynamic ambient GI approximation method that leverages spherical-harmonics (SH) transform information of the phase function to reduce the huge amount of ray sampling required for GI while ensuring fidelity. The method catalyzes spatial ray-marching reuse and adaptive temporal accumulation. Third, we deploy a two-phase ray-tracing algorithm with a tiled k-buffer to achieve fast processing of order-independent transparency (OIT) for multiple volume instances. Consequently, high-quality and high-performance real-time dynamic volume rendering can be achieved under constrained budgets controlled by developers. As our solution supports mixed mesh-volume rendering, the test results prove the practical usefulness of our approach for high-resolution binocular VR rendering on hybrid multi-volumetric and geometric environments.

Existing lifting networks for regressing 3D human poses from 2D single-view poses are typically constructed with linear layers based on graph-structured representation learning. In sharp contrast to them, this paper presents Grid Convolution (GridConv), mimicking the wisdom of regular convolution operations in image space. GridConv is based on a novel Semantic Grid Transformation (SGT) which leverages a binary assignment matrix to map the irregular graph-structured human pose onto a regular weave-like grid pose representation joint by joint, enabling layer-wise feature learning with GridConv operations. We provide two ways to implement SGT, including handcrafted and learnable designs. Surprisingly, both designs turn out to achieve promising results and the learnable one is better, demonstrating the great potential of this new lifting representation learning formulation. To improve the ability of GridConv to encode contextual cues, we introduce an attention module over the convolutional kernel, making grid convolution operations input-dependent, spatial-aware and grid-specific. We show that our fully convolutional grid lifting network outperforms state-of-the-art methods with noticeable margins under (1) conventional evaluation on Human3.6M and (2) cross-evaluation on MPI-INF-3DHP. Code is available at https://github.com/OSVAI/GridConv.

Federated learning enables the privacy-preserving training of neural network models using real-world data across distributed clients. FedAvg has become the preferred optimizer for federated learning because of its simplicity and effectiveness. FedAvg uses naïve aggregation to update the server model, interpolating client models based on the number of instances used in their training. However, naïve aggregation suffers from client drift when the data is heterogenous (non-IID), leading to unstable and slow convergence. In this work, we propose a novel aggregation approach, elastic aggregation, to overcome these issues. Elastic aggregation interpolates client models adaptively according to parameter sensitivity, which is measured by computing how much the overall prediction function output changes when each parameter is changed. This measurement is performed in an unsupervised and online manner. Elastic aggregation reduces the magnitudes of updates to the more sensitive parameters so as to prevent the server model from drifting to any one client distribution, and conversely boosts updates to the less sensitive parameters to better explore different client distributions. Empirical results on real and synthetic data as well as analytical results show that elastic aggregation leads to efficient training in both convex and nonconvex settings while being fully agnostic to client heterogeneity and robust to large numbers of clients, partial participation, and imbalanced data. Finally, elastic aggregation works well with other federated optimizers and achieves significant improvements across the board.

Deep neural networks have been proven effective in a wide range of tasks. However, their high computational and memory costs make them impractical to deploy on resource-constrained devices. To address this issue, quantization schemes have been proposed to reduce the memory footprint and improve inference speed. While numerous quantization methods have been proposed, they lack systematic analysis for their effectiveness. To bridge this gap, we collect and improve existing quantization methods and propose a gold guideline for post-training quantization. We evaluate the effectiveness of our proposed method with two popular models, ResNet50 and MobileNetV2, on the ImageNet dataset. By following our guidelines, no accuracy degradation occurs even after directly quantizing the model to 8-bits without additional training. A quantization-aware training based on the guidelines can further improve the accuracy in lower-bits quantization. Moreover, we have integrated a multi-stage fine-tuning strategy that works harmoniously with existing pruning techniques to reduce cost even further. Remarkably, our results reveal that a quantized MobileNetV2 with 30% sparsity actually surpasses the performance of the equivalent full-precision model, underscoring the effectiveness and resilience of our proposed scheme.

This paper proposes a Robust and Efficient Memory Network, referred to as REMN, for studying semi-supervised video object segmentation (VOS). Memory-based methods have recently achieved outstanding VOS performance by performing non-local pixel-wise matching between the query and memory. However, these methods have two limitations. 1) Non-local matching could cause distractor objects in the background to be incorrectly segmented. 2) Memory features with high temporal redundancy consume significant computing resources. For limitation 1, we introduce a local attention mechanism that tackles the background distraction by enhancing the features of foreground objects with the previous mask. For limitation 2, we first adaptively decide whether to update the memory features depending on the variation of foreground objects to reduce temporal redundancy. Second, we employ a dynamic memory bank, which uses a lightweight and differentiable soft modulation gate to decide how many memory features need to be removed in the temporal dimension. Experiments demonstrate that our REMN achieves state-of-the-art results on DAVIS 2017, with a $\mathcal{J}\& \mathcal{F}$ score of 86.3% and on YouTube-VOS 2018, with a $\mathcal{G}$ over mean of 85.5%. Furthermore, our network shows a high inference speed of 25+ FPS and uses relatively few computing resources.

We present a lightweight and efficient semi-supervised video object segmentation network based on the space-time memory framework. To some extent, our method solves the two difficulties encountered in traditional video object segmentation: one is that the single frame calculation time is too long, and the other is that the current frame’s segmentation should use more information from past frames. The algorithm uses a global context (GC) module to achieve high-performance, real-time segmentation. The GC module can effectively integrate multi-frame image information without increased memory and can process each frame in real time. Moreover, the prediction mask of the previous frame is helpful for the segmentation of the current frame, so we input it into a spatial constraint module (SCM), which constrains the areas of segments in the current frame. The SCM effectively alleviates mismatching of similar targets yet consumes few additional resources. We added a refinement module to the decoder to improve boundary segmentation. Our model achieves state-of-the-art results on various datasets, scoring 80.1% on YouTube-VOS 2018 and a $${\cal J}{\rm{\& }}{\cal F}$$ score of 78.0% on DAVIS 2017, while taking 0.05 s per frame on the DAVIS 2016 validation dataset.

Shadow removal is a challenging computer vision and multimedia task that aims to restore image content in shadow regions. The state-of-the-art shadow removal methods introduce artifacts near shadow boundaries or inconsistencies between shadow and nonshadow areas, which can be easily noticed by the human eye at first glance. In this paper, we design a boundary-aware shadow removal network (BA-ShadowNet) that improves shadow removal accuracy by increasing the removal performance at shadow boundaries. In contrast with previously developed methods, which usually consider shadow boundary optimization to be a postprocessing technique, our method performs shadow removal and shadow boundary optimization simultaneously. For this purpose, the proposed BA-ShadowNet is designed as a multiscale encoder-decoder structure, where the decoder consists of a shadow removal branch and a shadow optimization branch. An interaction module is then introduced to fuse and exchange the features of the two branches. This module facilitates the removal branch in perceiving the locations and colors of shadow boundaries. Additionally, it optimizes the boundary branch according to the image context extracted from the removal branch. A three-term loss function is further developed to supervise the shadow removal results and to address the issue of imbalanced supervision between shadow boundary pixels and pixels inside shadows. Extensive experiments conducted on the ISTD+ and SRD datasets demonstrate that the proposed BA-ShadowNet greatly outperforms the state-of-the-art methods with respect to shadow removal.

Recent work has shown that Binarized Neural Networks (BNNs) are able to greatly reduce computational costs and memory footprints, facilitating model deployment on resource-constrained devices. However, in comparison to their full-precision counterparts, BNNs suffer from severe accuracy degradation. Research aiming to reduce this accuracy gap has thus far largely focused on specific network architectures with few or no 1 × 1 convolutional layers, for which standard binarization methods do not work well. Because 1 × 1 convolutions are common in the design of modern architectures (e.g. GoogleNet, ResNet, DenseNet), it is crucial to develop a method to binarize them effectively for BNNs to be more widely adopted. In this work, we propose an “Elastic-Link” (EL) module to enrich information flow within a BNN by adaptively adding real-valued input features to the subsequent convolutional output features. The proposed EL module is easily implemented and can be used in conjunction with other methods for BNNs. We demonstrate that adding EL to BNNs produces a significant improvement on the challenging large-scale ImageNet dataset. For example, we raise the top-1 accuracy of binarized ResNet26 from 57.9% to 64.0%. EL also aids con-vergence in the training of binarized MobileNet, for which a top-1 accuracy of 56.4% is achieved. Finally, with the integration of ReActNet, it yields a new state-of-the-art result of 71.9% top-1 accuracy.

Network architecture plays a key role in the deep learning-based computer vision system. The widely-used convolutional neural network and transformer treat the image as a grid or sequence structure, which is not flexible to capture irregular and complex objects. In this paper, we propose to represent the image as a graph structure and introduce a new \emph{Vision GNN} (ViG) architecture to extract graph-level feature for visual tasks. We first split the image to a number of patches which are viewed as nodes, and construct a graph by connecting the nearest neighbors. Based on the graph representation of images, we build our ViG model to transform and exchange information among all the nodes. ViG consists of two basic modules: Grapher module with graph convolution for aggregating and updating graph information, and FFN module with two linear layers for node feature transformation. Both isotropic and pyramid architectures of ViG are built with different model sizes. Extensive experiments on image recognition and object detection tasks demonstrate the superiority of our ViG architecture. We hope this pioneering study of GNN on general visual tasks will provide useful inspiration and experience for future research. The PyTorch code is available at \url{https://github.com/huawei-noah/Efficient-AI-Backbones} and the MindSpore code is available at \url{https://gitee.com/mindspore/models}.

Adder Neural Network (AdderNet) provides a new way for developing energy-efficient neural networks by replacing the expensive multiplications in convolution with cheaper additions (i.e., L1-norm). To achieve higher hardware efficiency, it is necessary to further study the low-bit quantization of AdderNet. Due to the limitation that the commutative law in multiplication does not hold in L1-norm, the well-established quantization methods on convolutional networks cannot be applied on AdderNets. Thus, the existing AdderNet quantization techniques propose to use only one shared scale to quantize both the weights and activations simultaneously. Admittedly, such an approach can keep the commutative law in the L1-norm quantization process, while the accuracy drop after low-bit quantization cannot be ignored. To this end, we first thoroughly analyze the difference on distributions of weights and activations in AdderNet and then propose a new quantization algorithm by redistributing the weights and the activations. Specifically, the pre-trained full-precision weights in different kernels are clustered into different groups, then the intra-group sharing and inter-group independent scales can be adopted. To further compensate the accuracy drop caused by the distribution difference, we then develop a lossless range clamp scheme for weights and a simple yet effective outliers clamp strategy for activations. Thus, the functionality of full-precision weights and the representation ability of full-precision activations can be fully preserved. The effectiveness of the proposed quantization method for AdderNet is well verified on several benchmarks, e.g., our 4-bit post-training quantized adder ResNet-18 achieves an 66.5% top-1 accuracy on the ImageNet with comparable energy efficiency, which is about 8.5% higher than that of the previous AdderNet quantization methods. Code will be available at https://gitee.com/mindspore/models/tree/master/research/cv/AdderQuant.

This paper introduces versatile filters to construct efficient convolutional neural networks that are widely used in various visual recognition tasks. Considering the demands of efficient deep learning techniques running on cost-effective hardware, a number of methods have been developed to learn compact neural networks. Most of these works aim to slim down filters in different ways, e.g., investigating small, sparse or quantized filters. In contrast, we treat filters from an additive perspective. A series of secondary filters can be derived from a primary filter with the help of binary masks. These secondary filters all inherit in the primary filter without occupying more storage, but once been unfolded in computation they could significantly enhance the capability of the filter by integrating information extracted from different receptive fields. Besides spatial versatile filters, we additionally investigate versatile filters from the channel perspective. Binary masks can be further customized for different primary filters under orthogonal constraints. We conduct theoretical analysis on network complexity and an efficient convolution scheme is introduced. Experimental results on benchmark datasets and neural networks demonstrate that our versatile filters are able to achieve comparable accuracy as that of original filters, but require less memory and computation cost.

In simulating viscous incompressible SPH fluids, incompressibility and viscosity are typically solved in two separate stages. However, the interference between pressure and shear forces could cause the missing of behaviors that include preservation of sharp surface details and remarkable viscous behaviors such as buckling and rope coiling. To alleviate this problem, we introduce for the first time the semi-implicit method for pressure linked equations (SIMPLE) into SPH to solve incompressible fluids with a broad range viscosity. We propose to link incompressibility and viscosity solvers, and impose incompressibility and viscosity constraints iteratively to gradually remove the interference between pressure and shear forces. We will also discuss how to solve the particle deficiency problem for both incompressibility and viscosity solvers. Our method is stable at simulating incompressible fluids whose viscosity can range from zero to an extremely high value. Compared to state-of-the-art methods, our method not only produces realistic viscous behaviors, but is also better at preserving sharp surface details.

Monocular depth estimation aims to predict the dense depth map from a single RGB image, which has important applications in 3D reconstruction, automatic driving, and augmented reality. However, existing methods directly feed the original RGB image into the model to extract depth features without avoiding the interference of depth-irrelevant information on depth estimation accuracy, which leads to inferior performance. To remove the influence of depth-irrelevant information and improve depth prediction accuracy, we propose RADepthNet, a novel reflectance-guided network fusing boundary features. Specifically, our method predicts depth maps using three steps: 1) Intrinsic Image Decomposition. We propose a Reflectance extraction module consisting of an encoder-decoder structure to extract depth-related reflectance. We demonstrate that the module can reduce the influence of illumination on depth estimation through an ablation study. 2) Boundary Detection. Boundary extraction module, consisting of an encoder, a refinement block, and an upsample block, is proposed to better predict depth at object boundaries utilizing gradient constraints. 3) Depth Prediction Module. Use a different encoder from 2) to obtain depth features from the reflectance map and fuse boundary features to predict depth. Besides, we proposed FIFADataset, a depth estimation dataset applied in soccer scenarios. Extensive experiments on the public dataset and our proposed FIFADataset show that our method achieves state-of-the-art performance.

Exploring the correspondences across multi-view images is the basis of many computer vision tasks. However, most existing methods are limited on accuracy under challenging conditions. In order to learn more robust and accurate correspondences, we propose the DSD-MatchingNet for local feature matching in this paper. First, we develop a deformable feature extraction module to obtain multi-level feature maps, which harvests contextual information from dynamic receptive fields. The dynamic receptive fields provided by deformable convolution network ensures our method to obtain dense and robust correspondences. Second, we utilize the sparse-to-dense matching with the symmetry of correspondence to implement accurate pixel-level matching, which enables our method to produce more accurate correspondences. Experiments have shown that our proposed DSD-MatchingNet achieves a better performance on image matching benchmark, as well as on visual localization benchmark. Specifically, our method achieves 91.3% mean matching accuracy on HPatches dataset and 99.3% visual localization recalls on Aachen Day-Night dataset.

Internet video delivery has undergone a tremendous explosion of growth over the past few years. However, the quality of video delivery system greatly depends on the Internet bandwidth. Deep Neural Networks (DNNs) are utilized to improve the quality of video delivery recently. These methods divide a video into chunks, and stream LR video chunks and corresponding content-aware models to the client. The client runs the inference of models to super-resolve the LR chunks. Consequently, a large number of models are streamed in order to deliver a video. In this paper, we first carefully study the relation between models of different chunks, then we tactfully design a joint training framework along with the Content-aware Feature Modulation (CaFM) layer to compress these models for neural video delivery. With our method, each video chunk only requires less than 1% of original parameters to be streamed, achieving even better SR performance. We conduct extensive experiments across various SR backbones, video time length, and scaling factors to demonstrate the advantages of our method. Besides, our method can be also viewed as a new approach of video coding. Our primary experiments achieve better video quality compared with the commercial H.264 and H.265 standard under the same storage cost, showing the great potential of the proposed method. Code is available at: https://github.com/Neural-video-delivery/ CaFM-Pytorch-ICCV2021

Transformer is a new kind of neural architecture which encodes the input data as powerful features via the attention mechanism. Basically, the visual transformers first divide the input images into several local patches and then calculate both representations and their relationship. Since natural images are of high complexity with abundant detail and color information, the granularity of the patch dividing is not fine enough for excavating features of objects in different scales and locations. In this paper, we point out that the attention inside these local patches are also essential for building visual transformers with high performance and we explore a new architecture, namely, Transformer iN Transformer (TNT). Specifically, we regard the local patches (\eg, 16$\times$16) as “visual sentences” and present to further divide them into smaller patches (\eg, 4$\times$4) as “visual words”. The attention of each word will be calculated with other words in the given visual sentence with negligible computational costs. Features of both words and sentences will be aggregated to enhance the representation ability. Experiments on several benchmarks demonstrate the effectiveness of the proposed TNT architecture, \eg, we achieve an 81.5\% top-1 accuracy on the ImageNet, which is about 1.7\% higher than that of the state-of-the-art visual transformer with similar computational cost. The PyTorch code is available at \url{https://github.com/huawei-noah/CV-Backbones}, and the MindSpore code is available at \url{https://gitee.com/mindspore/models/tree/master/research/cv/TNT}.

Deep convolutional neural networks (CNNs) are often of sophisticated design with numerous learnable parameters for the accuracy reason. To alleviate the expensive costs of deploying them on mobile devices, recent works have made huge efforts for excavating redundancy in pre-defined architectures. Nevertheless, the redundancy on the input resolution of modern CNNs has not been fully investigated, i.e., the resolution of input image is fixed. In this paper, we observe that the smallest resolution for accurately predicting the given image is different using the same neural network. To this end, we propose a novel dynamic-resolution network (DRNet) in which the input resolution is determined dynamically based on each input sample. Wherein, a resolution predictor with negligible computational costs is explored and optimized jointly with the desired network. Specifically, the predictor learns the smallest resolution that can retain and even exceed the original recognition accuracy for each image. During the inference, each input image will be resized to its predicted resolution for minimizing the overall computation burden. We then conduct extensive experiments on several benchmark networks and datasets. The results show that our DRNet can be embedded in any off-the-shelf network architecture to obtain a considerable reduction in computational complexity. For instance, DR-ResNet-50 achieves similar performance with an about 34% computation reduction, while gaining 1.4% accuracy increase with 10% computation reduction compared to the original ResNet-50 on ImageNet. Code will be available at https://gitee.com/mindspore/models/tree/master/research/cv/DRNet.

Abstract Compared with video de‐raining, single image de‐raining is more technically difficult due to the lack of temporally redundant information. This paper proposes a new two‐stage method for single image de‐raining. In the first stage, the authors develop an effective two‐step model to detect the rain streaks by taking pixel intensity, and direction of rain streaks as priors. In the second stage, the rain repair process is performed at the patch level. The authors first define a way to search for similar patches of each patch, and then group the similar patches together to form a matrix. Finally, a low‐rank matrix completion technique is utilized to recover the rain‐stained pixels based on the rain map obtained from the first stage. Compared with several state‐of‐the‐art methods, authors' proposed method is competitive in terms of the abilities of removing rain streaks, and preserving image details.

Abstract Images captured in haze conditions, especially at nighttime with low light, often suffer from degraded visibility, contrasts, and vividness, which makes it difficult to carry out the following vision tasks. In this article, we propose an attention‐based feature fusion network (AFF‐Dehazing) for low‐light image dehazing. Our method decomposes the low‐light image dehazing into two task‐independent streams containing four modules: image dehazing module, low‐light feature extractor module, feature fusion module, and image restoration module. The basic block of these modules is the proposed attention‐based residual dense block. Since the dual‐branch are used, AFF‐Dehazing can avoid learning the mixed degradation all‐in‐one and enhance the details of low‐light haze images. Extensive experiments show that our method surpasses previous state‐of‐the‐art image dehazing methods and low‐light enhancement methods by a very large margin both quantitatively and qualitatively.

Abstract The advection step in grid‐based fluid simulation is prone to numerical dissipation, which results in loss of detail. How to improve the advection accuracy to preserve more fluid details is still challenging. On the other hand, a common way to enhance smoke details is to use vorticity confinement. However, most of the previous methods simply used a fine‐tuned scale factor ε to adjust the strength of the confinement force, which can only amplify existing vortex details and is easy to cause instability when ε is large. In this article, we proposed an adaptive vorticity confinement method, which does not suffer from the above problems, to compensate the vorticity loss during advection with little extra cost. The main idea is to first calculate a scale factor whose value depends on the vorticity loss during advection, and then use it to adaptively control the vorticity confinement force for vorticity compensation with high stability. The experiment results show the effectiveness and efficiency of our method.

The interaction of gas and liquid can produce many interesting phenomena, such as bubbles rising from the bottom of the liquid. The simulation of two-phase fluids is a challenging topic in computer graphics. To animate the interaction of a gas and liquid, MultiFLIP samples the two types of particles, and a Euler grid is used to track the interface of the liquid and gas. However, MultiFLIP uses the fluid implicit particle (FLIP) method to interpolate the velocities of particles into the Euler grid, which suffer from additional noise and instability. To solve the problem caused by fluid implicit particles (FLIP), we present a novel velocity transport technique for two individual particles based on the affine particle-in-cell (APIC) method. First, we design a weighed coupling method for interpolating the velocities of liquid and gas particles to the Euler grid such that we can apply the APIC method to the simulation of a two-phase fluid. Second, we introduce a narrowband method to our system because MultiFLIP is a time-consuming approach owing to the large number of particles. Experiments show that our method is well integrated with the APIC method and provides a visually credible two-phase fluid animation. The proposed method can successfully handle the simulation of a twophase fluid.

The imperfect material effect is one of the most important themes to obtain photo-realistic results in rendering. Textile material rendering has always been a key area in the field of computer graphics. So far, a great deal of effort has been invested in its unique appearance and physicsbased simulation. The appearance of the dyeing effect commonly found in textiles has received little attention. This paper introduces techniques for simulation of staining effects on textiles. Pulling, wearing, squeezing, tearing, and breaking effects are more common imperfect effects of fabrics, these external forces will cause changes in the fabric structure, thus affecting the diffusion effect of stains. Based on the microstructure of yarn, we handle the effect of the stain on the imperfect textile surface. Our simulation results can achieve a photo-realistic effect.

This paper formalizes the binarization operations over neural networks from a learning perspective. In contrast to classical hand crafted rules (\eg hard thresholding) to binarize full-precision neurons, we propose to learn a mapping from full-precision neurons to the target binary ones. Each individual weight entry will not be binarized independently. Instead, they are taken as a whole to accomplish the binarization, just as they work together in generating convolution features. To help the training of the binarization mapping, the full-precision neurons after taking sign operations is regarded as some auxiliary supervision signal, which is noisy but still has valuable guidance. An unbiased estimator is therefore introduced to mitigate the influence of the supervision noise. Experimental results on benchmark datasets indicate that the proposed binarization technique attains consistent improvements over baselines.

Abstract Physically plausible fracture animation is a challenging topic in computer graphics. Most of the existing approaches focus on the fracture of isotropic materials. We proposed a frame‐field method for the design of anisotropic brittle fracture patterns. In this case, the material anisotropy is determined by two parts: anisotropic elastic deformation and anisotropic damage mechanics. For the elastic deformation, we reformulate the constitutive model of hyperelastic materials to achieve anisotropy by adding additional energy density functions in particular directions. For the damage evolution, we propose an improved phase‐field fracture method to simulate the anisotropy by designing a deformation‐aware second‐order structural tensor. These two parts can present elastic anisotropy and fractured anisotropy independently, or they can be well coupled together to exhibit rich crack effects. To ensure the flexibility of simulation, we further introduce a frame‐field concept to assist in setting local anisotropy, similar to the fiber orientation of textiles. For the discretization of the deformable object, we adopt a novel Material Point Method(MPM) according to its fracture‐friendly nature. We also give some design criteria for anisotropic models through comparative analysis. Experiments show that our anisotropic method is able to be well integrated with the MPM scheme for simulating the dynamic fracture behavior of anisotropic materials.

Abstract Physically based deformation simulation has been studied for many years in computer graphics. In order to simulate more complex geometric models and better meet the designer's requirements, many anisotropic approaches have been proposed in recent years. However, most of the approaches focus on simulating orthotropic models. In comparison with orthotropic models, nonorthotropic ones allow the objects to have anisotropic behaviors along nonorthogonal directions. In this paper, we introduce an improved approach to simulate nonorthotropic geometric models under large deformation. The improvements are mainly twofold. First, a frame field is specified on a given undeformed object, that is, each point of the object is equipped with a frame. In each local frame, we construct three independent vectors and form a nonorthogonal coordinate. Second, we design the deformation properties along each axis in the local nonorthogonal coordinate to get a local constitutive model. The final nonorthotropic model is generated by transforming the designed model from local nonorthogonal coordinates to the global standard Cartesian coordinate. To improve the stability, we introduce a time‐varying method to simultaneously track the local coordinates reorientation by pushing forward the original frame field to the deformed frame field. Experiments show that the deformation simulation using the designed nonorthotropic models exhibits anisotropic behaviors along different directions and are more stable than previous methods.

Abstract This article introduces a novel approach for embedding 3D models in offline physical environments using quick response (QR) codes. Unlike conventional methods, we consider settings where 3D models cannot be retrieved from a remote server. Our method involves generating octree models from voxelized 3D models and storing them in QR codes using a space‐efficient data structure. This allows storing 3D models that are both intelligible and purposeful on standard QR codes while addressing the major storage constraint that is present in offline situations. Furthermore, we explore 3D convolutional neural networks (CNN) and autoencoders (AE) to compress 3D models with high resolutions where using octrees alone does not suffice. To the best of our knowledge, our AE network is the first to employ octrees to further compress its encoded data. Through user‐friendly desktop and mobile applications, we allow users to encode, decode and visualize 3D models in augmented reality (AR) using QR codes, thus experiment with our methods. The proposed approach enables unique applications and future research in ubiquitous computing, 3D data compression and transmission, 3D AEs, AR and Virtual Reality, low‐cost autonomous robots, and 3D printing.

Abstract Generally, it is regarded as challenge work to grasp the drawing style of an ancient masterpiece in Chinese painting learning. This paper presents a novel approach to the generation of a Chinese ink painting in a certain style and animating its brushwork process with expert skills. In order to demonstrate the techniques of brush and ink inside a stroke, a serials of geometric properties of a brush stroke, are first extracted, then through rational deformation calculation, the best stroke source is mapped onto the stroking path, which is sketched by the user, and finally a new Chinese painting can be synthesized by style migration and natural stroke composition. So with the generated strokes, the lifelike brushwork process of the new painting can be represented dramatically. Actually, by showing the authentic painting process, the tool we implemented helps the learners, who have no profound skills and knowledge in domain of Chinese painting, master the essence of a great painting style, and also provides an easy way to art creation and the comprehension of mysterious Chinese traditional art.

Abstract The accuracy of advection has a great influence on the visual effect of fluid simulation. Constrained interpolation profile (CIP) method has been an important advection scheme because of its third‐order accuracy and the fact that it only needs to be performed over a compact stencil, but extending it to high‐dimensional advection equations is not easy, because it involves complex calculations and large memory overheads, and is usually unstable. In this article, we propose a stable and efficient three‐dimensional (3D) CIP scheme which can maintain high accuracy but requires low computation and memory cost. We first construct an efficient two‐dimensional (2D) CIP scheme based on dimensional splitting and local Taylor expansions, and then propose an effective way to extend it for 3D applications without decreasing the computational accuracy or affecting the stability. The experimental results show the advantages of our method over the state‐of‐the‐art advection schemes.

The central building block of convolutional neural networks (CNNs) is the convolution operator, which enables networks to construct informative features by fusing both spatial and channel-wise information within local receptive fields at each layer. A broad range of prior research has investigated the spatial component of this relationship, seeking to strengthen the representational power of a CNN by enhancing the quality of spatial encodings throughout its feature hierarchy. In this work, we focus instead on the channel relationship and propose a novel architectural unit, which we term the "Squeeze-and-Excitation" (SE) block, that adaptively recalibrates channel-wise feature responses by explicitly modelling interdependencies between channels. We show that these blocks can be stacked together to form SENet architectures that generalise extremely effectively across different datasets. We further demonstrate that SE blocks bring significant improvements in performance for existing state-of-the-art CNNs at slight additional computational cost. Squeeze-and-Excitation Networks formed the foundation of our ILSVRC 2017 classification submission which won first place and reduced the top-5 error to 2.251 percent, surpassing the winning entry of 2016 by a relative improvement of ∼ 25 percent. Models and code are available at https://github.com/hujie-frank/SENet.

Most motion-blurred images captured in the real world have spatially-varying point-spread functions, and some are caused by different positions and depth values, which cannot be handled by most state-of-the-art deblurring methods based on deconvolution. To overcome this problem, we propose a depth-aware motion blur model that treats a blurred image as an integration of a sequence of clear images. To restore the clear latent image, we extend the Richardson-Lucy method to incorporate our blur model with a given depth image. The empty holes in the depth image, caused by occlusion or device limitations, are fixed by PatchMatch-based depth filling. We regard the depth image as a Markov random field and select candidate labels by using belief propagation to set and smooth depth values for empty areas. Deblurring and depth filling are performed iteratively to refine the results. Our method can also be applied to real-world images with the assistance of motion estimation. The deblurring process is shown to be convergent; moreover, the number of iterations and the level of noise amplification are acceptable. The experimental results show that our method can not only handle depth-variant motion blur but also refine depth images.

Abstract Pencil drawing is a simple yet effective way to depict what people see by clearly presenting details of the scene. Existing methods usually extract strokes of the input image and adjust the result image tone to make it look like a pencil drawing. However, they do not consider the quality of the stroke image and the geometry information of lines in the stroke image, which unavoidably results in the violation of original essential structures and in a flatten pencil drawing with unrealistic appearance. We put forward a spatial rendering network for pencil drawing stylization. Spatial stroke images are extracted from the image pyramid by a single‐shot bottom‐up neural network to improve the quality of these stroke images. Unlike the former tone adjustment–based methods, we analyze perceptual cues of strokes at different stroke image levels and use the obtained geometry information to constrain the stroke shading procedure. The final pencil drawing result is achieved by the stroke shading fusion of different levels' shading results. The effectiveness of our spatial rendering network for pencil drawing stylization is demonstrated by an ablation study, comparison to the state of the art, and a user study.

Abstract Handling of missing areas is a key step for depth‐based rendering to synthesize virtual views. Existing methods usually consider finding candidate pixels from only one reference view to fill missing areas. However, the information provided by one reference view is restricted by the position of the view. By utilizing two reference views located on both the left and right sides of the virtual view, we propose to synthesize the missing areas at hole level with connected regions optimization. To avoid the appearance of ghost boundary, we apply morphological operations to generate a boundary band map for the depth map, which restricts the warping of the virtual view. We use a binary map to mark unknown pixels in a hole, label the connected unknown regions, and count the area of each connected region, which decide the order in our enhanced inpainting synthesis. Besides, we separate the foreground and background regions of the depth map to constrain the searching of candidate pixels. Multiple experiments on virtual view synthesis have shown the effectiveness and high quality of our multiview‐coherent disocclusion synthesis.

In this work, we present a supervised object segmentation algorithm for unconstrained video. Instead of arbitrarily picking a few frames for manual labeling, as in many existing supervised methods, the proposed method selects frames in a more reasonable manner, called supervision optimization. For this, we formulate a principled objective function by inferring the propagation error from appearance and motion clues. After this, we construct a multilevel segmentation model, which consists of low-level and high-level features. On the low level, image pixels are used for a more accurate estimation of motion and segmentation. On the high level, image segments are considered for a more semantic classification of the foreground and background. By integrating these in one segmentation graph, the result can be further improved by leveraging the knowledge from both levels. In experiments, the proposed approach is evaluated by different measures, and the results on a benchmark demonstrate the effectiveness in comparison with other state-of-the-art algorithms.

In this study, we propose an effective and efficient algorithm for unconstrained video object segmentation, which is achieved in a Markov random field (MRF). In the MRF graph, each node is modeled as a superpixel and labeled as either foreground or background during the segmentation process. The unary potential is computed for each node by learning a transductive SVM classifier under supervision by a few labeled frames. The pairwise potential is used for the spatial-temporal smoothness. In addition, a high-order potential based on the multinomial event model is employed to enhance the appearance consistency throughout the frames. To minimize this intractable feature, we also introduce a more efficient technique that simply extends the original MRF structure. The proposed approach was evaluated in experiments with different measures and the results based on a benchmark demonstrated its effectiveness compared with other state-of-the-art algorithms.

Modeling virtual textiles has long been an appealing topic in computer graphics. To date, considerable effort has been devoted to their distinctive appearance and physically-based simulation. The apperance of staining patterns, commonly seen on textiles, has received comparatively little attention. This paper introduces techniques for simulating staining effects on fabric. Based on the microstructure of yarn, we propose a triple-layer model (TLM) to handle the liquid-yarn interaction for the wetting and wicking computation, and we formalize the liquid spreading in woven cloth into two typical actions, the in-yarn diffusion and the cross-yarn diffusion. The dye diffusion is driven by the liquid diffusion and the concentration distribution of pigments. The warp-weft anisotropy is handled by simulation of the yarn's structure in the two directions. Experimental results demonstrate that a wide range of fabric stain phenomenon on different textile materials, such as the water ring effect, the high saturate stain contour, and the dynamic wash away effect, can be simulated effectively without loss of visual realism. The realism of our simulation results is comparable to effects shown in photographs of real-world examples.

Abstract We introduce a new biorthogonal wavelet approach to creating a water‐tight surface defined by an implicit function, from a finite set of oriented points. Our approach aims at addressing problems with previous wavelet methods which are not resilient to missing or nonuniformly sampled data. To address the problems, our approach has two key elements. First, by applying a three‐dimensional partial integration, we derive a new integral formula to compute the wavelet coefficients without requiring the implicit function to be an indicator function. It can be shown that the previously used formula is a special case of our formula when the integrated function is an indicator function. Second, a simple yet general method is proposed to construct smooth wavelets with small support. With our method, a family of wavelets can be constructed with the same support size as previously used wavelets while having one more degree of continuity. Experiments show that our approach can robustly produce results comparable to those produced by the Fourier and Poisson methods, regardless of the input data being noisy, missing or nonuniform. Moreover, our approach does not need to compute global integrals or solve large linear systems.

Abstract In this paper, a novel method with good numerical stability is proposed from the perspective of energy preserving to alleviate the numerical dissipations in the advection step of Eulerian fluid simulation. The main idea is to measure the vorticity loss during advection, calculate the lost angular kinetic energy with a proposed scheme, and then synthesize a high‐frequency incompressible details field to compensate the lost energy in a way that is consistent with Kolmogorov's theory, which prevents the synthetic details from interfering with the existing fluid flow. The method works independently of the advection scheme and can be easily combined with other advection schemes to enhance the effect. It adds only 5% to 10% of the computational overhead while producing convincing fluid details without changing the overall behavior of the original flow.

Unified simulation of versatile elastoplastic materials and different dimensions offers many advantages in animation production, contact handling, and hardware acceleration. The unstructured particle representation is particularly suitable for this task, thanks to its simplicity. However, previous meshless techniques either need too much computational cost for addressing stability issues, or lack physical meanings and fail to generate interesting deformation behaviors, such as the Poisson effect. In this paper, we study the development of an elastoplastic model under the state-based peridynamics framework, which uses integrals rather than partial derivatives in its formulation. To model elasticity, we propose a unique constitutive model and an efficient iterative simulator solved in a projective dynamics way. To handle plastic behaviors, we incorporate our simulator with the Drucker-Prager yield criterion and a reference position update scheme, both of which are implemented under peridynamics. Finally, we show how to strengthen the simulator by position-based constraints and spatially varying stiffness models, to achieve incompressibility, particle redistribution, cohesion, and friction effects in viscoelastic and granular flows. Our experiments demonstrate that our unified, meshless simulator is flexible, efficient, robust, and friendly with parallel computing.

This paper presents an approach of structure-preserving image completion with dynamic patches. Existing image completion methods may generate unnatural abnormal structures or structure disorders due to limited patches and patterns availability. Our structure-preserving image completion utilizes objective function minimization considering the coherence not only within the image cavity but also with global constraints. A series of dynamic patch-based optimizations are applied to fulfill the cavity. Unlike traditional fixed-size patch-based methods, our image completion with competitive dynamic patch-matching mechanism provides more effective structure restoration. Parallel searching of different-sized patches is performed to retrieve optimal patches for completing the image cavity with nice structure preservation. The experiments show that the realistic completed images by our approach are visually pleasing with nice structural coherence.

Abstract Gradient‐domain compositing has been widely used to create a seamless composite with gradient close to a composite gradient field generated from one or more registered images. The key to this problem is to solve a Poisson equation, whose unknown variables can reach the size of the composite if no region of interest is drawn explicitly, thus making both the time and memory cost expensive in processing multi‐megapixel images. In this paper, we propose an approximate projection method based on biorthogonal Multiresolution Analyses (MRA) to solve the Poisson equation. Unlike previous Poisson equation solvers which try to converge to the accurate solution with iterative algorithms, we use biorthogonal compactly supported curl‐free wavelets as the fundamental bases to approximately project the composite gradient field onto a curl‐free vector space. Then, the composite can be efficiently recovered by applying a fast inverse wavelet transform. Considering an n‐pixel composite, our method only requires 2n of memory for all vector fields and is more efficient than state‐of‐the‐art methods while achieving almost identical results. Specifically, experiments show that our method gains a 5× speedup over the streaming multigrid in certain cases.

Non-uniform motion blur due to object movement or camera jitter is a common phenomenon in videos. However, the state-of-the-art video deblurring methods used to deal with this problem can introduce artifacts, and may sometimes fail to handle motion blur due to the movements of the object or the camera. In this paper, we propose a non-uniform motion model to deblur video frames. The proposed method is based on superpixel matching in the video sequence to reconstruct sharp frames from blurry ones. To identify a suitable sharp superpixel to replace a blurry one, we enrich the search space with a non-uniform motion blur kernel, and use a generalized PatchMatch algorithm to handle rotation, scale, and blur differences in the matching step. Instead of using pixel-based or regular patch-based representation, we adopt a superpixel-based representation, and use color and motion to gather similar pixels. Our non-uniform motion blur kernels are estimated from the motion field of these superpixels, and our spatially varying motion model considers spatial and temporal coherence to find sharp superpixels. Experimental results showed that the proposed method can reconstruct sharp video frames from blurred frames caused by complex object and camera movements, and performs better than the state-of-the-art methods.

The papers in this special section were presented at the 2015 ACM Symposium on Virtual Reality Software and Technology (VRST’15).

Cloud simulation is important for creating images of outdoor scenes. However, the complexity of this natural phenomenon makes the simulation of large-scale clouds difficult in real time. In this paper, we present a new method for 3D cloud simulation in which cloud animation is simplified and simulated by approximating Lennard-Jones Potential. To solve the N-body problem in Lennard-Jones Potential, we minimized the interaction between particles by dividing the simulation space into many cells and we defined a cutoff distance to perform calculation between neighboring particles. Additionally, a separate distance is introduced between particles to maintain the stability in the Lennard-Jones system. Our experimental results demonstrate that our method is computationally inexpensive and suitable for real time applications where large-scale simulation of clouds is required.

Abstract Interactive isosurface visualisation has been made possible by mapping algorithms to GPU architectures. However, current state‐of‐the‐art isosurfacing algorithms usually consume large amounts of GPU memory owing to the additional acceleration structures they require. As a result, the continued limitations on available GPU memory mean that they are unable to deal with the larger datasets that are now increasingly becoming prevalent. This paper proposes a new parallel isosurface‐extraction algorithm that exploits the blocked organisation of the parallel threads found in modern many‐core platforms to achieve fast isosurface extraction and reduce the associated memory requirements. This is achieved by optimising thread co‐operation within thread‐blocks and reducing redundant computation; ultimately, an indexed triangular mesh can be produced. Experiments have shown that the proposed algorithm is much faster (up to 10×) than state‐of‐the‐art GPU algorithms and has a much smaller memory footprint, enabling it to handle much larger datasets (up to 64×) on the same GPU.

We propose a semi-Lagrangian method for multiphase interface tracking. In contrast to previous methods, our method maintains an explicit polygonal mesh, which is reconstructed from an unsigned distance function and an indicator function, to track the interface of arbitrary number of phases. The surface mesh is reconstructed at each step using an efficient multiphase polygonization procedure with precomputed stencils while the distance and indicator function are updated with an accurate semi-Lagrangian path tracing from the meshes of the last step. Furthermore, we provide an adaptive data structure, multiphase distance tree, to accelerate the updating of both the distance function and the indicator function. In addition, the adaptive structure also enables us to contour the distance tree accurately with simple bisection techniques. The major advantage of our method is that it can easily handle topological changes without ambiguities and preserve both the sharp features and the volume well. We will evaluate its efficiency, accuracy and robustness in the results part with several examples.

In this paper, a new surface tension formulation is presented for Smoothed Particle Hydrodynamics in fluid simulation, especially small-scale detailed fluid animation. The surface tension formulation is decomposed into three processes: (1) mesh smoothing exploited a Lagrangian operator in a volume-preserved way, (2) surface tension computation between the original mesh and smoothed mesh, (3) surface tension transfer from mesh vertices onto their neighbor particles. Experimental results show that the proposed surface tension formulation is effective and efficient for realistic simulations.

In this paper, we present a system that extracts the information implied by eye movements and use this information to analyze students' learning behavior. Our system uses a common webcam to capture students' facial image sequences when they are learning in front of monitors. We then process these images and establish sequences of changing location of iris, which represent the movements of eyes. With the eye movement sequences, we train a HMM classifier that can analyze their pattern and generate learning status for any given moment in the lesson. These statuses could help the computers to get a better understanding about the students' intention and behavior during online learning. The status sequences of those who view the same lesson could also be used as a reference for teaching quality assessment.

Ubiquitous motion blur easily fails multi-frame super-resolution (MFSR). Our method proposed in this paper tackles this issue by optimally searching least blurred pixels in MFSR. An EM framework is proposed to guide residual blur estimation and high-resolution image reconstruction. To suppress noise, we employ a family of sparse penalties as natural image priors, along with an effective solver. Theoretical analysis is performed on how and when our method works. The relationship between estimation errors of motion blur and the quality of input images is discussed. Our method produces sharp and higher-resolution results given input of challenging low-resolution noisy and blurred sequences.

Multi-task learning has been proposed to improve the generalization performance by learning multiple tasks jointly. One challenge for this learning paradigm is to effectively seek the shared information across multiple tasks. In this paper, we propose a novel multi-task learning method to adaptively share information. Unlike many existing multi-task learning methods which impose strong assumptions on task related-ness, our method captures the relationships among tasks and identifies the disparities of each task simultaneously, thus can flexibly exploit the shared information. Moreover, we apply it to fine-grained categorization problem, which usually suffers from the difficulties of insufficient training data and high inter-class similarity. The experimental results on two widely used datasets show the superiority of our method compared with some state-of-the-art methods.

We introduce an adaptive hybrid level set surface tracking for fluid simulation in this paper. To improve the accuracy of the convectional level set method, we propose a method that combines the higher accuracy interpolation and octree grid to improve the accuracy while still maintain good efficiency. Adaptive gradient-augmented level set is adopted for higher accuracy interpolation and two strategies are proposed to guide the building of the octree structure. We demonstrate our method with several level set tests, and couple it to several existing fluid simulators. The results show that our method facilitates significant improvements in accuracy and efficiency, and is effective for free surface fluid simulation in capturing fluid details.

We present an efficient interactive system for dressing a naked bird with feathers. In our system, a skeleton associated with guide feathers is used to describe the distribution of the body feathers. The special skeleton can be easily built by the user, given a 3D bird model as input. To address the problem of interpenetrations among feathers, the growth priority between the feather roots is defined, with which we obtain the growth order from a greedily constructed directed acyclic graph. Each feather is then adjusted in that order by a height field based collision resolution process. The height field not only provides an efficient way to detect the collision but also enables us to finely control the degree of collision during feather adjustments. The results show that our approach is capable of resolving the collisions among thousands of feathers in a few seconds. If model animation is desired, the feathers can be adjusted on the fly at interactive framerates. Details of our implementation are provided with several examples to demonstrate the effectiveness of our system.

Interactive image segmentation aims at classifying the image pixels into foreground and background classes given some foreground and background markers. In this paper, we propose a novel framework for interactive image segmentation that builds upon graph-based manifold ranking model, a graph-based semi-supervised learning technique which can learn very smooth functions with respect to the intrinsic structure revealed by the input data. The final segmentation results are improved by overcoming two core problems of graph construction in traditional models: graph structure and graph edge weights. The user provided scribbles are treated as the must-link and must-not-link constraints. Then we model the graph as an approximatively k-regular sparse graph by integrating these constraints and our extended neighboring spatial relationships into graph structure modeling. The content and labels driven locally adaptive kernel parameter is proposed to tackle the insufficiency of previous models which usually employ a unified kernel parameter. After the graph construction, a novel three-stage strategy is proposed to get the final segmentation results. Due to the sparsity and extended neighboring relationships of our constructed graph and usage of superpixels, our model can provide nearly real-time, user scribble insensitive segmentations which are two core demands in interactive image segmentation. Last but not least, our framework is very easy to be extended to multi-label segmentation, and for some less complicated scenarios, it can even get the segmented object through single line interaction. Experimental results and comparisons with other state-of-the-art methods demonstrate that our framework can efficiently and accurately extract foreground objects from background.

Abstract In virtual environments, real‐time simulation and rendering of dynamic fluids have always been the pursuit for virtual reality research. In this paper, we present a real‐time framework for realistic fluid simulation and rendering on graphics processing unit. Because of the high demand for interactive fluids with larger particle set, the computational need is becoming higher. The proposed framework can effectively reduce the computational burden through avoiding the computation in inactive areas, where many particles with similar properties and low local pressure cluster together. While in active areas, the computation is fully carried out; thus, the fluid dynamics are largely preserved. Here, a robust particle classification technique is introduced to classify particles into either active or inactive. The test results have shown that the technique improves the time performance of fluid simulation largely. We then incorporate parallel surface reconstruction technique using marching cubes to extract the surfaces of the fluid. The introduced histogram pyramid‐based marching cubes technique is fast and memory efficiency. As a result, we are able to produce plausible and interactive fluids with the proposed framework for large‐scale virtual environments. Copyright © 2013 John Wiley & Sons, Ltd.

The sparse coding technique has shown flexibility and capability in image representation and analysis. It is a powerful tool in many visual applications. Some recent work has shown that incorporating the properties of task (such as discrimination for classification task) into dictionary learning is effective for improving the accuracy. However, the traditional supervised dictionary learning methods suffer from high computation complexity when dealing with large number of categories, making them less satisfactory in large scale applications. In this paper, we propose a novel multi-level discriminative dictionary learning method and apply it to large scale image classification. Our method takes advantage of hierarchical category correlation to encode multi-level discriminative information. Each internal node of the category hierarchy is associated with a discriminative dictionary and a classification model. The dictionaries at different layers are learnt to capture the information of different scales. Moreover, each node at lower layers also inherits the dictionary of its parent, so that the categories at lower layers can be described with multi-scale information. The learning of dictionaries and associated classification models is jointly conducted by minimizing an overall tree loss. The experimental results on challenging data sets demonstrate that our approach achieves excellent accuracy and competitive computation cost compared with other sparse coding methods for large scale image classification.

Hierarchical classification models have been proposed to achieve high accuracy by transferring effective information across the categories. One important challenge for this paradigm is to design what can be transferred across the categories. In this paper, we propose a novel method to learn a sharing model by taking advantage of multi-level feature representations. Unlike many of the existing methods which learn the sharing model based on identical feature space, multi-level feature detectors enable our model to capture rich visual information in hierarchical category structure. Moreover, hierarchical classifier parameters associated with multi-level feature representations are learned to model the visual correlation in the hierarchy. The experimental results on Caltech-256 dataset and ImageNet subset demonstrate that our method achieves excellent performance compared with some state-of-the-art methods, and shows the advantage of multi-level information transfer.

We introduce a novel framework for tracking multiphase interfaces with explicit contouring technique. In our framework, an unsigned distance function and an additional indicator function are used to represent the multiphase system. Our method maintains the explicit polygonal meshes that define the multiphase interfaces. At each step, distance function and indicator function are updated via semi-Lagrangian path tracing from the meshes of the last step. Interface surfaces are then reconstructed by polygonization procedures with precomputed stencils and further smoothed with a feature-preserving non-manifold smoothing algorithm to stay in good quality. Our method is easy to be implemented and incorporated into multiphase simulation, such as immiscible fluids, crystal grain growth and geometric flows. We demonstrate our method with several level set tests, including advection, propagation, etc., and couple it to some existing fluid simulators. The results show that our approach is stable, flexible, and effective for tracking multiphase interfaces.

ABSTRACT In the previous works, the real‐time fluid‐character animation could hardly be achieved because of the intensive processing demand on the character's movement and fluid simulation. This paper presents an effective approach to the real‐time generation of the fluid flow driven by the motion of a character in full 3D space, based on smoothed‐particle hydrodynamics method. The novel method of conducting and constraining the fluid particles by the geometric properties of the character motion trajectory is introduced. Furthermore, the optimized algorithms of particle searching and rendering are proposed, by taking advantage of the graphics processing unit parallelization. Consequently, both simulation and rendering of the 3D liquid effects with realistic character interactions can be implemented by our framework and performed in real‐time on a conventional PC. Copyright © 2013 John Wiley & Sons, Ltd.

Shadow mapping is a fast and easy to use method to produce hard shadows. However, it introduces aliasing due to its uniform sampling strategy and limited shadow map resolution. In this paper, we propose a memory efficient algorithm to render high quality shadows. Our algorithm is based on a multi-resolution shadow map structure, which includes a conventional shadow map for scene regions where a low-resolution shadow map is sufficient, and a high-resolution patch buffer to capture scene regions that are susceptible to aliasing. With this data structure, we are able to capture shadow details with far less memory footprint than conventional shadow mapping. In order to maintain an appropriate performance compared to conventional shadow mapping, we designed a customized CUDA rasterizer to render the high-resolution patches.

Despite the continuous advances in local stereo matching for years, most efforts are on developing robust cost computation and aggregation methods. Little attention has been seriously paid to the disparity refinement. In this work, we study weighted median filtering for disparity refinement. We discover that with this refinement, even the simple box filter aggregation achieves comparable accuracy with various sophisticated aggregation methods (with the same refinement). This is due to the nice weighted median filtering properties of removing outlier error while respecting edges/structures. This reveals that the previously overlooked refinement can be at least as crucial as aggregation. We also develop the first constant time algorithm for the previously time-consuming weighted median filter. This makes the simple combination ``box aggregation + weighted median'' an attractive solution in practice for both speed and accuracy. As a byproduct, the fast weighted median filtering unleashes its potential in other applications that were hampered by high complexities. We show its superiority in various applications such as depth up sampling, clip-art JPEG artifact removal, and image stylization.

ABSTRACT We present a method to make an accurate and dense reconstruction from the input of video captured by a free moving handheld camera in real time. By the method firstly, the positions of the camera and sparse 3D points are estimated by simultaneous localization mapping. Then the depth maps of selected reference frames are computed from corresponding camera bundles. Lastly a novel linear algorithm is also proposed to integrate all the depth maps into dense meshes partially. The main contributions of this paper are in the following points: the reference frames and corresponding camera bundles are able to be selected automatically, then accurate and smooth depth maps are generated in real time, and the depth maps are merged into a dense mesh by using a linear algorithm based on the error clouds optimization. Our algorithm is implemented on dual CPU and graphics processing unit in a parallel framework for improving the performance. Copyright © 2013 John Wiley & Sons, Ltd.

ABSTRACT Because existing image abstraction systems can hardly incorporate with the changing light, we present an integrated image abstraction and relighting rendering system, which is based on a salient edge‐guided luminance field optimization approach. For an input image, we first adopt a sparsity prior‐based illumination decomposition method to remove its original illumination and have an intrinsic image. Meanwhile, we iteratively extract the salient edge inside by employing a message‐passing strategy. Then, we simplify this image with a proposed salient edge‐guided image abstraction optimization algorithm in the luminance field. Finally, we put forward a salient edge‐guided image relighting optimization method to simulate the effect of dynamic lighting along different directions. Experiment results show that our system can artistically adjust the illumination of the abstracted image and makes it more vivid. Copyright © 2013 John Wiley & Sons, Ltd.

ABSTRACT This paper presents a finite element method‐based framework for an object withering simulation modeled with heterogeneous material, such as fruits drying or decay. We introduce diffusion procedures for both the moisture content and decay spread, which are solved directly on a tetrahedral mesh representation of the fruit flesh. Then, we use the moisture content to control shrinkage through the initial strain, which is integrated into the Lagrangian dynamic equation, and solved with the finite element method. For the complex structure of the object, another fine triangle mesh is used to represent the skin, and its deformation is solved by a thin shell technique. To couple the motion between different layers of the fruit, a tracking force is used to pull the skin and drive its deformation together with the volume mesh. In comparison with the previous work, our method provides temporally and spatially varying parameters to model the complex phenomena of object withering. Moreover, the water diffusivity can also be given by user input to present various material properties of the cut section and skin‐covered area. Our algorithm is easy to implement and highly efficient in generating a realistic appearance for the withering effect. For a medium‐scale model, we can achieve interactive simulation. Copyright © 2012 John Wiley & Sons, Ltd.

ABSTRACT This paper presents a novel approach for simulating the dynamic coupling between a tree and raindrops based on physical deformation and fluid simulation. By the approach, tree animation in the rain can be simulated in a two‐resolution way: branch motion and leaf motion. The branch is represented by the Euler–Bernoulli beam model, and the leaf petiole is represented by the three‐prism elastic model. Interaction coupling liquid motion on the hydrophilic surface with a flexible petiole is well implemented by a special design. To simplify the computation process, instead of the computation‐intensive three‐dimensional Navier–Stokes equations, shallow water equations are used to simulate the water dynamics together with the whole leaf deformation. Simulation has been also made to various phenomena incurred from the interactive coupling. These include, among others, part of impacting raindrops splashing into the air with the remaining flowing along the slant of the leaf and merging into larger ones or hanging on the blade boundary, with the leaf rebounding and vibrating after the drops fall off the leaf. A level‐of‐detail approach is exploited to accelerate rendering in views of different distances. The experimental results illustrate that the approach can be applied to efficiently generate realistic details of the interactive coupling between a tree and raindrops. Copyright © 2012 John Wiley & Sons, Ltd.

We propose a method for intrinsic image decomposition based on retinex theory and texture analysis. While most previous methods approach this problem by analyzing local gradient properties, our technique additionally identifies distant pixels with the same reflectance through texture analysis, and uses these nonlocal reflectance constraints to significantly reduce ambiguity in decomposition. We formulate the decomposition problem as the minimization of a quadratic function which incorporates both the retinex constraint and our nonlocal texture constraint. This optimization can be solved in closed form with the standard conjugate gradient algorithm. Extensive experimentation with comparisons to previous techniques validate our method in terms of both decomposition accuracy and runtime efficiency.

Abstract Beam tracing combines the flexibility of ray tracing and the speed of polygon rasterization. However, beam tracing so far only handles linear transformations; thus, it is only applicable to linear effects such as planar mirror reflections but not to non‐linear effects such as curved mirror reflection, refraction, caustics and shadows. In this paper, we introduce non‐linear beam tracing to render these non‐linear effects. Non‐linear beam tracing is highly challenging because commodity graphics hardware supports only linear vertex transformation and triangle rasterization. We overcome this difficulty by designing a non‐linear graphics pipeline and implementing it on top of a commodity GPU. This allows beams to be non‐linear where rays within the same beam do not have to be parallel or intersect at a single point. Using these non‐linear beams, real‐time GPU applications can render secondary rays via polygon streaming similar to how they render primary rays. A major strength of this methodology is that it naturally supports fully dynamic scenes without the need to pre‐store a scene database. Utilizing our approach, non‐linear ray tracing effects can be rendered in real‐time on a commodity GPU under a unified framework.

Abstract We present a realistic, fast, and controllable model to simulate fire propagation on solid objects with the object decomposition process involved. A hybrid structure of grids is employed to simulate the whole process efficiently. An improved burning surface update scheme based on level set and a novel method for visualizing the burning surface are proposed to produce convincing results. To achieve interactive simulation speed, a few acceleration techniques are employed, including a moving grid generated to dynamically track the fire propagation, a refined Marching Cubes method to reconstruct the burning surface, and a hardware‐implemented fluid solver. By controlling a few physical and geometric parameters, we are able to simulate various solid combustion phenomena. Copyright © 2011 John Wiley & Sons, Ltd.

In the past decade, modern GPUs have provided increasing programmability with vertex, geometry and fragment shaders. However, many classical problems have not been efficiently solved using the current graphics pipeline where some stages are still fixed functions on chip. In particular, multi-fragment effects, especially order-independent transparency, require programmability of the blending stage, that makes it difficult to be solved in a single geometry pass. In this paper we present FreePipe, a system for programmable parallel rendering that can run entirely on current graphics hardware and has performance comparable with the traditional graphics pipeline. Within this framework, two schemes for the efficient rendering of multi-fragment effects in a single geometry pass have been developed by exploiting CUDA atomic operations. Both schemes have achieved significant speedups compared to the state-of-the-art methods that are based on traditional graphics pipelines.

This paper proposes a novel approach to animation realistic interactions between tree branches and raindrops in a physically-based way. A new elastic model using a three-prism structures is presented to flexibly bend and twist tree branches naturally in the first time. Various distinct forms of interactions when or after raindrops hitting on tree branches can be well simulated using a new efficient technique specially designed for liquid motion on non-rigid objects with hydrophilic surfaces. Experimental results indicate that our approach can be used to simulate the interactions between tree branches and raindrops efficiently and realistically.

Abstract We propose a volume fraction based approach to effectively simulate the miscible and immiscible flows simultaneously. In this method, a volume fraction is introduced for each fluid component and the mutual interactions between different fluids are simulated by tracking the evolution of the volume fractions. Different techniques are employed to handle the miscible and immiscible interactions and special treatments are introduced to handle flows involving multiple fluids and different kinds of interactions at the same time. With this method, second‐order accuracy is preserved in both space and time. The experiment results show that the proposed method can well handle both immiscible and miscible interactions between fluids and much richer mixing detail can be generated. Also, the method shows good controllability. Different mixing effects can be obtained by adjusting the dynamic viscosities and diffusion coefficients. Copyright © 2010 John Wiley & Sons, Ltd.

We present an accelerated depth peeling algorithm for order-independent transparency rendering on graphics hardware. Unlike traditional depth peeling which only peels one layer of transparent pixels per rendering pass, our algorithm peels multiple layers simultaneously per rendering pass. Our acceleration is achieved via our fragment program which sorts and writes multiple fragment colors and depths via MRT. A notable feature of our algorithm is that it is robust against the unreliable parallel read-after-write behavior in current graphics hardware, guaranteeing correct transparency ordering. For ordinary scenes rendered under RGBA8 color precision, we achieve up to 8x speed-up over conventional depth peeling with current generation graphics hardware. Our algorithm is simple to implement on current GPU without any hardware modification. In addition, it does not require applications to perform any pre-sorting of transparent geometry.

Both exponential shadow maps and variance shadow maps are efficient shadow maps anti-aliasing algorithms based on pre-filtering, but exponential shadow maps will cause light bleeding, float overflow and penumbra discontinuity problems and variance shadow maps also have serious light bleeding problem. All these prevent their individual use in practice especially in large virtual environment such as games. This paper presents a hybrid method by storing three values in the depth texture: depth, squared depth and exponential depth. From these values we calculate two occlusion values using exponential and variance shadow maps respectively. A minimum of the two occlusion values is taken as the final result. This can efficiently eliminate the problems of the two methods and can be widely used in practice.

In this article, we propose a physically based novel scheme on the bubble creation. By the scheme, the phase change of the liquid-gas interaction and the effect of volume expansion are simulated by altering the divergence of the velocity field. Because of the little mass of the gas and the great difference in density between liquid and vapor, the volume of the liquid is considered to be constant. We construct the multi-material interface using the volume of fluid (VOF) and piecewise linear interface construction (PLIC) methods and maintain the volume conservation for each material.

We present a new approach for quickly designing 3D models of botanical trees using an iterative addition of new nodes to the tree branch structure. This process is guided by the proximity of points marking volume density data captured from photographs. Numerical parameters provide the user controls that are consistent with the characteristics of trees in landscaping and make it possible to generate a wide variety of tree styles. Meanwhile we synthesize visually believable motions for the generated tree models affected by a wind field. Our system enables the simulation of tree animation, by introducing physically-based transformation matrix calculations for hierarchical branch patterns. The system also supports the tree-shaping modes in which many branches and leaves are generated by interactively designed their distribution density. Experimental results show that our approach can design a variety of reasonably natural-looking trees and their motions.

In computer graphics, animated models are widely used to represent time-varying data. In this paper, we propose an improved method to generate multiresolution animation models. We use a curvature sensitive quadric error metric (QEM) criterion as our basic measurement, which can preserve local features on the surface. We append a deformation weight to the aggregated edge contraction cost for the whole animation to preserve areas with large deformation. At last, we introduce a mesh optimization method to deal with the animation sequence, which can efficiently improve the temporal coherence and reduce visual artifacts. The results show our approach is efficient, easy to implement, and good quality progressive animation models can be generated at any level of detail.

The paper presents a novel approach for simulating realistic time-varied carpets. By the approach, a 3D carpet is constructed first by image-based techniques from a single photo input, through a texel structure, established to generate realistic carpet with its pattern guided by the captured image. Secondly, a time-varying simulation model is proposed to capture various aspects of the time-dependent appearance of carpets such as dust accumulation, color fading and fiber change. Additionally, a time-varying map is provided to control the specific weathering degrees at different voxels in time through a hierarchal model. Experimental results show that the realistic time-varying simulation is successfully achieved with the proposed techniques.

Grass is one of the crucial elements in representing real nature scenes. Various methods have been proposed for grass simulation, for example, Boulanger and his colleagues render realistic grass in real-time with dynamic lighting, shadows and animations. Till now, few approaches have investigated the realistic simulation of withering grassland which is still a big challenge in computer graphics, due to the great complexity of both geometry and withering mechanism. In our work, a novel concept called Time-Varying Texels (TVT) is proposed to extend the static texel[Kajiya and Kay 1989] and make it time-serialized. In a grass TVT structure, time-dependent texels are arranged hierarchically in a Time-Space Partitioning (TSP) tree, so that the grass withering process can be efficiently approximated, with acceptable spatial and temporal errors. In this way we facilitate LOD rendering. Traditional texel structures could hardly undertake physical based calculation in each grass blade, so we introduce a point based structure (PBS) to increase the flexibility. Dynamic processes such as geometric deformation and material transformation can be achieved on PBS during the whole withering procedure. Additionally, by clustering the pre-computed TVT samples in low densities using a mingling algorithm, we obtain high-density grass TVT so as to significantly improve the efficiency of TVT generation.

Modern GPUs provide gradually increasing programmability on vertex shader, geometry shader and fragment shader in the past decade. However, many classical problems such as order-independent transparency (OIT), occlusion culling have not yet been efficiently solved using the traditional graphics pipeline. The main reason is that the behavior of the current stage of the pipeline is hard to be determined due to the unpredictable future data. Since the rasterization and blending stage are still largely fixed functions on chip, previous improvements on these problems always require hardware modifications thus remain on the theoretical level. In this paper we propose CUDA Renderer, a fully programmable graphics pipeline using compute unified device architecture (CUDA) [NVIDIA 2008] which can completely run on current graphics hardware. Our experimental results have demonstrated significant speedup to traditional graphics pipeline especially on OIT. We believe many other problems can also benefit from this flexible architecture.

We present a particle-based method for viscoelastic fluids simulation. In the method, based on the traditional Navier-Stokes equation, an additional elastic stress term is introduced to achieve viscoelastic flow behaviors, which have both fluid and solid features. Benefiting from the Lagrangian nature of Smoothed Particle Hydrodynamics, large flow deformation can be handled more easily and naturally. And also, by changing the viscosity and elastic stress coefficient of the particles according to the temperature variation, the melting and flowing phenomena, such as lava flow and wax melting, are achieved. The temperature evolution is determined with the heat diffusion equation. The method is effective and efficient, and has good controllability. Different kinds of viscoelastic fluid behaviors can be obtained easily by adjusting the very few experimental parameters.

The paper presents a novel approach for simulating the weathering fur. Dusty effects on fur is generated by volumetric γ-ton tracing method and the geometry deformation is modeled through a dynamic PBS. The proposed approach can efficiently simulate the weathering effects of fur.

Abstract We present a new model, called the dual‐microfacet, for those materials such as paper and plastic formed by a thin, transparent slab lying between two surfaces of spatially varying roughness. Light transmission through the slab is represented by a microfacet‐based BTDF which tabulates the microfacet's normal distribution (NDF) as a function of surface location. Though the material is bounded by two surfaces of different roughness, we approximate light transmission through it by a virtual slab determined by a single spatially‐varying NDF. This enables efficient capturing of spatially variant transparent slices. We describe a device for measuring this model over a flat sample by shining light from a CRT behind it and capturing a sequence of images from a single view. Our method captures both angular and spatial variation in the BTDF and provides a good match to measured materials.

Abstract We present a novel pressure correction scheme for the Smoothed Particle Hydrodynamics (SPH) for fluid animation. In the conventional SPH method, equations of state (EOS) are employed to relate the pressure to the particle density. To enforce the volume conservation, high speeds of sound are usually required, which leads to very small time steps and noisy pressure distribution. The problem remains one of the main reasons of numerical instability in SPH. In the paper, a new extra pressure correction scheme is proposed to transport the local pressure disturbance to the neighboring area and no solution of the Poisson equation is required. As a result, smoother pressure distribution and more efficient simulation are achieved. The proposed method has been used to simulate free surface problems. The results demonstrate the validation of the present SPH method. Surface tension and fluid fragmentation can be well handled. Copyright © 2009 John Wiley & Sons, Ltd.

Abstract This paper presents a new method for modeling and rendering fur with a wide variety of furstyles. We simulate virtual fur using shell textures—a multiple layers of textured slices for its generality and efficiency. As shell textures usually suffer from the inherent visual gap errors due to the uniform discretization nature, we present theangle‐split shell textures(ASST) approach, which classifies the shell textures into different types with different numbers of texture layers, by splitting the angle space of the viewing angles between fur orientation and view direction. Our system can render the fur with biological patterns, and utilizes vector field and scalar field on ASST to control the geometric variations of the furry shape. Users can intuitively shape the fur by applying the combing, blowing, and interpolating effects in real time. Our approach is intuitive to implement without using complex data structures, with real‐time performance for dynamic fur appearances. Copyright © 2009 John Wiley & Sons, Ltd.

Many graphics applications represent deformable surfaces through dynamic meshes. Such models often contain redundant details, which can be difficult for processing and transmission. In this paper, we propose an efficient method to generate multiresolution models for deforming surface sequence based on triangle optimization. We use an improved quadric error metric (QEM) criterion as our basic measurement, which can preserve more local features on the surface. We define a deformation weight to be appended to the aggregated edge contraction cost for the whole animation. This new metric can preserve not only the modelpsilas individual geometric features but also features only appeared during the deformation animation. At last, we propose an intrinsic Laplacian mesh smoothing method to improve the triangle shape and further reduce the visual distortion. Our approach is efficient, easy to implement, and as a result good quality approximations with optimized triangles can be generated at any given frame.

This paper presents the layer-based representation of polyhedrons and its use for point-in-polyhedron tests. In the representation, the facets and edges of a polyhedron are sequentially arranged, and so, the binary search algorithm is efficiently used to speed up inclusion tests. In comparison with conventional representation for polyhedrons, the layer-based representation we propose greatly reduces the storage requirement because it represents much information implicitly, though it still has a storage complexity O(n). It is simple to implement, and robust for inclusion tests because many singularities are erased in constructing the layer-based representation. By incorporating an octree structure for organizing polyhedrons, our approach can run at a speed comparable with Binary Space Partitioning (BSP)-based inclusion tests, and at the same time greatly reduce storage and preprocessing time in treating large polyhedrons. We have developed an efficient solution for point-in-polyhedron tests with the time complexity varying between O(n) and O(logn), depending on the polyhedron shape and the constructed representation, and less than O(log3n) in most cases. The time complexity of preprocess is between O(n) and O(n2), varying with polyhedrons, where n is the edge number of a polyhedron.

Most texture synthesis algorithms explicitly or implicitly adopt Markov random field or similar distribution as their basic model to guide the synthesis process. However, MRF-like models can 't handle textures well with large scale structure or unstable structure due to their inherent local and stable assumptions. To make improvement in this regard, we propose a new texture analysis/synthesis framework that combines two main ideas. Firstly, in material space we decompose the texture contents into units with "basic shape " and "feature vector". Based on this, the space spanned by a set of sampled textons is constructed to help introduce additional changes upon textons. Secondly, in pattern space, using the idea of "feature texture " acquired from texture swatch for different properties especially for distribution rules of textons, we may capture and manipulate the global structure flexibly. By this formulization, we are able to obtain a satisfactory texture appearance, and also a rich controlability as well.

It is a challenge work to render natural lighting phenomena in real-time. A major reason is due to high computational expense to simulate the physical model of atmosphere scattering. Another is due to the lack of power and programmability in the graphic hardware. In this paper, we propose unified volumes representation for light shaft and shadow, which is an efficient method of simulating natural light shafts and shadows with atmospheric scattering effect. We give the analytic formula of light shaft without numerical integration and then make use of the current graphic hardware to implement the integral computation on each volume surface for scattering. Our approach can not only simulate the lighting effect with single light source but also multiple parallel light sources according to the physical model of skylight and sunlight. With acceleration of the GPU, we can generate realistic appearance with high frame rate satisfying real time application. It can possibly be used in current commercial game or other virtual reality systems.

In this article, we present an efficient algorithm for encoding the connectivity information of triangular meshes. By the method, all triangles are traversed first to obtain operator series. Then an arithmetic coder based on variable code-mode is applied to encode the operator series. According to the operator last encoded, the property of triangular mesh and the method of mesh traversal, a code-mode is calculated for each operator being encoded currently, where the operator with higher prediction probability is given a shorter binary strand. Then we can obtain the binary strand according to its code-mode and encode every bit of this binary strand by adaptive arithmetic coding method. Testing result shows that the compression ratio of our algorithm is very high and even higher than the compression ratio by using TG algorithm, which is commonly regarded as one of the best in terms of compression ratio.

In our work, we mainly use a lattice Boltzmann method (LBM) to simulate the underlying dynamics of miscible mixtures in binary fluid simulation called TFLBM. However, it suffers from the limitation of only resolving mixture flows with low Reynolds number, and it would blow up when the Reynolds number gets higher. In order to resolve such mixture flows with higher Reynolds number, by further investigation, we proposed to use a subgrid method to stabilize the computation of two fluid mixtures. The idea of the subgrid method is to split the actual velocity field into large-scale (resolved) and small-scale (unresolved) components. The effect of the unresolved motion on the resolved one is included by introducing a so-called eddy viscosity, and the method is also referred to as large eddy simulation (LES).

With in skeleton based deformation, two issues are waiting for investigation, large angle rotation and artifacts near joint regions. In this paper we propose rotation angle decomposition and detail preserving method to conquer them. A skeleton with multiple connected bones is prior computed for a triangle mesh. While deforming, large angle rotation is simulated by several progressive small angle rotations. We also introduce a distortion metric function E to quantify the difference between mesh edge vectors before and after deformation, preserving the detail of mesh surfaces. Nearby vertex position blending is enforced. We use sparse conjugate gradient method for the minimization of metric function E. Results show our method is suitable for 3D modeling and character animation applications, making the deformation near joint regions more natural.

In this paper, we propose a novel method for recovering the background in an image. Our method can firstly identify the foreground objects, and then the foreground can be replaced by painting with the background information. Specifically, the foreground objects are detected and removed by applying image segmentation based on graph cuts. Then we can paint the holes by employing a modified texture synthesis method based on texton mask. In the first step, the user could mark some pixels as "object" or "background", called the "seeds" in the algorithm. In particular, the region penalties in our method are defined in CIELab color space, meanwhile the pairwise similarity is based on color distance in the perceptual color space. Relying on the principle of energy minimization, the segmentation process partitions an image into two sets: background and object. The topology of our segmentation is unrestricted and both "object" and "background" can contain several isolated parts. After the "object" parts are removed, based on the surrounding information and background structure, we can find the matching patches from the texton mask obtained from the source image. After that, we select the best matching patch to fill the hole and blend the boundary areas to improve the final result. The experimental results have shown that the system can reconstruct the background of an image with complex structure.

In this paper, we propose a novel method that represents the highly-complex point sets by clustering the points to normal-mapped B-spline surfaces (NBSs). The main idea is to construct elaborate normal maps on simple surfaces for the realistic rendering of complex point-set models. Based on this observation, we developed the coarse, normal-mapped B-spline surfaces to approximate the original point datasets with fine surface details. In our algorithm, a genetic clustering algorithm is proposed to automatically segment the point samples into several clusters according to their statistical properties, and a network of B-spline patches with normal maps are constructed according to the clustering results. Our experimental results show that this representation facilitates the modeling and rendering of complex point sets without losing the visual qualities.

An effective method of depth image based rendering is proposed by applying texture mapping onto virtual but pre-defined multiple planes in the scene, called virtual planes. The method allows the viewpoint either static or moving around, including crossing the plane of the source images. By this approach, the relief textures from depth images are mapped onto the virtual planes, and through the pre-warping process, the virtual planes are converted into standard polygonal textures. After the virtual plane mosaics, the resultant image that supports 3D objects and immersive scenes can be generated by polygonal texture mapping. In addition, both hardware and software implementation of the method can increase the power of conventional texture mapping in image based rendering. In particular, the scope of the viewpoint could be extended into the inner space of depth images, and as a result, a novel solution is provided for constructing real-time walkthrough systems as well as for panoramic modeling from an arbitrary viewpoint in the depth image space

Many mesh streaming algorithms have focused on the transmission order of the polygon data with respect to the current viewpoint. In contrast to the conventional progressive streaming where the resolution of a model changes in the geometry space, we present an new approach which firstly partitions a mesh into several patches, then converts these patch into multi-chart geometry images(MCGIM). After all the MCGIM and normal map atlas are obtained by regular re-sampling, we could construct the regular quadtree-based hierarchical representation based on MCGIM. Experimental results have shown the effectiveness of our approach where one server streams the MCGIM texture atlas to the clients.

In our surrounding environment, we may often see many various miscible liquid-liquid mixture phenomena, like pouring honey or ink into water, Coca Cola into strong wine etc., while few papers have devoted to the simulation of the phenomena. In this paper, we use a two-fluid lattice Boltzmann method (TFLBM) to simulate the underlying dynamics of miscible mixtures. By the method, a subgrid model is applied to improve its numerical stability so that the free surface of the mixture, accompanying with higher Reynolds number, can be simulated. We also apply control forces to the mixture with interesting animation created. By optimizing the memory structure and taking the advantage of dual-core or multi-core systems, we achieve real time computation for a domain in 643 cells full of fluid mixtures.

Through interaction with surroundings, the fluids may change their properties such as shapes, temperature vastly, and the same would happen to the surroundings simultaneously. On the other hand, different surroundings characterize different interactions, and may change the shapes and motions of the fluids in different ways. Therefore, it is of importance in physically-based simulation of fluids to build physically correct models to represent the varying interactions between fluids and the environments. In this paper, we make a simple summation on the interactions, and in particular focus on those most interesting to us, and model them with various physical solutions. In some of the methods, advantage is taken with the graphics processing unit (GPU) to achieve real-time computation for medial-scale simulation

This paper presents a novel scheme for accelerating ray traversal computation in ray tracing. By the scheme, a pre-computed stage is applied to constructing what is called a traversal field for each rigid object that records the destinations for all possible incoming rays. The field data, which could be efficiently compressed offline, is stored in a small number of big rectangles called ray-relays that enclose each approximate convex segment of an object. In the ray-tracing stage, the records on relays are retrieved in a constant time, so that a ray traversal is implemented as a simple texture lookup on GPU. Thus, the performance of our approach is only related to the number of relays rather than scene size, while the number of relays is quite small. In addition, because the traversal fields only depend on the internal construction of each convex segment, they can be used to ray trace objects undergoing rigid motions at a negligible extra cost. Experimental results show that interactive rates could be achieved for dynamic scenes with the effects of specular reflections and refractions on an ordinary desk PC with GPU.

Abstract Miscible fluid mixtures, like pouring honey into water, Coca Cola into strong wine, are common phenomena in our daily life. While two miscible fluids are mixed together, their appearances in terms of colors and shapes will change due to their mixing interaction. The interaction between the mixture components could be regarded as a combination of the diffusing process and demixing process. If the former dominates the interaction, it is miscible; otherwise, it is immiscible. The complex microscopic interplay between the mixture components makes the simulation highly challenging. So far, there have been some dedicated research in computer graphics dealing with immiscible mixtures, but few works have been done focusing on miscible mixtures. In this paper, for the first time, we introduce a two‐fluid lattice Boltzmann method (LBM), called TFLBM, applied to miscible binary mixtures. Different from other similar methods, the viscous and diffusing properties of the fluid in our work are considered separately, so that the physical insight is exposed more clearly and rationally. In addition, the operation of LBM is mostly a linear local computation, and graphics processing unit (GPU) has been utilized to achieve real‐time simulation. Copyright © 2006 John Wiley & Sons, Ltd.

In this paper we present a novel method of hybrid-scheme for texture synthesis. By the scheme, we capture the feature of textures in two levels. At the macro-structure level, we first generate a texton mask on the pattern from the original source texture, employed in the synthesis as a primary guidance for the patch-based sampling. And in the micro-structure level, we perform blending using a pixel-based synthesis method based on the grey-level texton mask within the boundary zones of the patches. As a result, our method may not only quickly capture the macro-structure of the source texture, with little overhead from simple interactive operation,, and at the same time, the micro-structure may be also nicely preserved in the synthesis by a pixel-based blending procedure. Experiments show that our solution in combining patch-based sampling and pixel-based sampling method based on texton masks is highly efficient for synthesizing large amount of textures with comprehensive structures.

This paper presents a novel approach to implementing dynamic LOD on GPU. For our purpose, a quadtree structure is created based on seamless geometry image atlas, which is a 3D surface representation in parameter space by combining the features of geometry images and poly-cube maps. All the nodes in the quadtree are packed into the atlas textures. There are two rendering passes in our approach. In the first pass, the LOD selection is performed in the fragment shaders. The resultant buffer is taken as the input texture to the second rendering pass by vertex texturing, and thus the node culling and triangulation can be performed in the vertex shaders. Our LOD algorithm can generate adaptive meshes dynamically, and can be fully implemented on GPU. It improves the efficiency of LOD selection, and alleviates the computing load on CPU.

Point and polygon representations have their respective merits in rendering objects. In this paper, we propose a novel hybrid multi-resolution approach, PQuadtrees, to efficiently render highly detailed objects. PQuadtrees are constructed from geometry images. Both point and polygon are tightly integrated into a uniform structure. While traversing the P-Quadtrees in rendering, the part of surface that face the viewer can be rendered by coarser quad mesh to reduce the numbers of rendering primitives. The shading details can be enhanced by hardware accelerated normal mapping. The view dependent LOD selects the finer hierarchy on silhouette, which is rendered by points. The rendering of large-scale model is greatly accelerated while the visual effect both at the surfaces and the silhouette is guaranteed.

In this paper, we solve the 3D fluid dynamics problem in a complex environment by taking advantage of the parallelism and programmability of GPU. In difference from other methods, innovation is made in two aspects. Firstly, more general boundary conditions could be processed on GPU in our method. By the method, we generate the boundary from a 3D scene with solid clipping, making the computation run on GPU despite of the complexity of the whole geometry scene. Then by grouping the voxels into different types according to their positions relative to the obstacles and locating the voxel that determines the value of the current voxel, we modify the values on the boundaries according to the boundary conditions. Secondly, more compact structure in data packing with flat 3D textures is designed at the fragment processing level to enhance parallelism and reduce execution passes. The scalar variables including density and temperature are packed into four channels of texels to accelerate the computation of 3D Navier-Stokes equations (NSEs). The test results prove the efficiency of our method, and as a result, it is feasible to run middle-scale problems of 3D fluid dynamics in an interactive speed for more general environment with complex geometry on PC platform.

Abstract Taking advantage of the parallelism and programmability of GPU, we solve the fluid dynamics problem completely on GPU. Different from previous methods, the whole computation is accelerated in our method by packing the scalar and vector variables into four channels of texels. In order to be adaptive to the arbitrary boundary conditions, we group the grid nodes into different types according to their positions relative to obstacles and search the node that determines the value of the current node. Then we compute the texture coordinates offsets according to the type of the boundary condition of each node to determine the corresponding variables and achieve the interaction of flows with obstacles set freely by users. The test results prove the efficiency of our method and exhibit the potential of GPU for general‐purpose computations. Copyright © 2004 John Wiley & Sons, Ltd.

View-dependent level-of-detail (LOD) and visibility culling are two powerful tools for accelerating the rendering of very large models in a real-time visualization, especially in walkthrough of a large-scale terrain environment. In this paper, we propose a visibility-driven Continuous LOD (CLOD) framework for terrain, which takes advantage of both techniques. The visibility determination is based on the well-known occlusion horizon algorithm. By making use of the features of the terrain extracted in pre-processing stage, a new cascading occlusion culling (COC) algorithm is proposed to cull those regions classified as invisible to current viewpoint in real time. The time consumption and storage overheads that we spend on visibility preprocessing are quite small. Visibility-driven CLOD enhances culling efficiency and improves the frame rates significantly for walkthrough of a terrain environment.

Abstract Realistic image synthesis of highly complex, natural scenes has been a challenging topic in computer graphics over the years. Generation of snowy scenery could be even more difficult due to the difficulty of using common modelling primitives such as polygons and curved surfaces to express the snow‐like shapes. In this paper, we propose a hybrid multi‐mapping method to tackle the snowy scenery problem, in which a displacement map is utilized to model snowy blocks on near objects based on proximate polygons, and a volumetric texture map is combined to handle distant objects such as bushes and trees. Our experimental results showed that the hybrid method is viable in producing realistic snowy scenery with comprehensive complex environments, and in a favourable requirement of storage usage and rendering computation. Copyright © 2003 John Wiley & Sons, Ltd.

In this paper we propose a hybrid representation of environment models by a point representation augmented to geometric representation of surface polygons reconstructed from multiple reference images, through which a real time walkthrough of a complex environment can be achieved. By the method, we start from classification of pixels of the source images into two categories, corresponding respectively to the planar and non-planar surfaces in 3D space. For the pixels corresponding to the planar surfaces, the plane coefficients are reconstructed and all their appearances in the reference images are merged to form uniformly sampled texture images by a comparison of sampling rate and resampling. For the pixels corresponding to the non-planar surfaces, a point representation is applied and the redundant pixels are removed again through sampling rate comparison. The remained pixels are organized by OBB-tree according to their space coordinates. At the same time, the holes that are unable to be captured by all the reference images are pre-filled in the preprocessing phase so that the probability of hole appearance in walkthrough is greatly reduced. Under this hybrid representation, texture mapping and point warping are employed to render the novel views, to take full advantages of the acceleration utility of graphics hardware.

In this paper, a hybrid representation of environment models using a combination of points and polygons is proposed. Through the model, a real time walkthrough of a complex environment can be achieved. We start from multiple depth reference images and classify the pixels of images into two categories, corresponding respectively to planar and non-planar surfaces in 3D space, then all the redundant points are removed by a comparison algorithm of sampling rate. For pixels corresponding to nonplanar surfaces, the point representation is maintained, and a local reconstruction and resampling process is employed to re-sample a set of points that distribute more uniformly on the surfaces. The re-sampled points are organized by an OBB-tree according to their space coordinates and a multiresolution structure is built to improve the rendering efficiency. For pixels corresponding to planar surfaces, the plane coefficients are reconstructed and all their appearances in the reference images are merged to texture maps. Under this hybrid representation, texture mapping and point-based rendering are employed to render the novel views, to take full advantage of the acceleration utility of graphics hardware. The algorithm demonstrates the combined advantages of approaches in traditional computer graphics and PBR/IBR, texture mapping for plane surfaces and point-based rendering for high detail surfaces and shapes.

Current out‐of‐core simplification algorithms can efficiently simplify large models that are too complex to be loaded in to the main memory at one time. However, these algorithms do not preserve surface details well since adaptive sampling, a typical strategy for detail preservation, remains to be an open issue for out‐of‐core simplification. In this paper, we present an adaptive sampling scheme, called the balanced retriangulation (BR), for out‐of‐core simplification. A key idea behind BR is that we can use Garland's quadric error matrix to analyze the global distribution of surface details. Based on this analysis, a local retriangulation achieves adaptive sampling by restoring detailed areas with cell split operations while further simplifying smooth areas with edge collapse operations. For a given triangle budget, BR preserves surface details significantly better than uniform sampling algorithms such as uniform clustering. Like uniform clustering, our algorithm has linear running time and small memory requirement.

This paper presents an efficient inverse warping algorithm for generating novel views by combining multiple reference images taken from different viewpoints. The method proceeds in three steps. Firstly, the reference images are preprocessed for extracting edge pixels. Secondly, an inverse warping is performed to render the desired image from one primary reference image. By taking advantages of epipolar line features and depth discontinuities in reference images, the inverse warping can be efficiently applied by segments, to accelerate the rendering substantially. Finally, holes in the desired image are filled up through searching the corresponding points in other reference images. At this stage, two accelerating techniques are presented. By using the proposed algorithm we can navigate a virtual environment at interactive rate.

We present a method for efficient synthesis of photorealistic free-form knitwear. Our approach is motivated by the observation that a single cross-section of yarn can serve as the basic primitive for modeling entire articles of knitwear. This primitive, called the lumislice, describes radiance from a yarn cross-section based on fine-level interactions — such as occlusion, shadowing, and multiple scattering — among yarn fibers. By representing yarn as a sequence of identical but rotated cross-sections, the lumislice can effectively propagate local microstructure over arbitrary stitch patterns and knitwear shapes. This framework accommodates varying levels of detail and capitalizes on hardware-assisted transparency blending. To further enhance realism, a technique for generating soft shadows from yarn is also introduced.

Due to visibility change and surface enlargement in producing a novel view from a new viewpoint, 3D re-projection from one reference image in IBMR inevitably produces holes in the destination image. Even worse, exposure errors occur when a background region occluded is visible in a desired image because of the absence of some background elements in the reference image. The general solution to this kind of problems is to use multiple images from different viewpoints as input source. By doing so however, the rendering cost would increase with the number of reference images and the composition algorithm has to rely on the z-buffer processing.In fact, plenty of redundant information exists among different reference images. Seeking for a nice way to extract the information needed in the novel view from the reference images is the key issue in solving the problem. In this paper, we propose a new method of image synthesis from multiple reference images. The method combines forward warping and backward warping to fulfil the image composition task for a novel viewpoint. The primary inspiration behind the development of our image synthesis method comes from a fact that the polygon edge geometry may indicate where an exposure and, possible an exposure error occur in the destination image if object silhouettes are prior known. The feature that intersection between scanline and polygons must be in pairs is employed to distinguish holes caused by enlargement of surfaces from those by visibility change. Different heuristic methods are used to choose one image as a primary reference image which shares the most resemblance with the destination image, and other reference images for filling different kinds of holes. Depth continuity along scanline and the depth information already present in the destination image are employed to accelerate the searching process of the corresponding pixels for filling holes.

This paper presents an efficient method for real-time generation of progressive mesh. In our algorithm, only the current mesh and local information of each vertex, such as vertex position, edge length, neighboring vertices, adjacent faces, and face normals are considered at each simplification step. All edge collapse costs are calculated and sorted into a binary tree using heap sort algorithm at initial stage. The whole complexity of the algorithm is O(n*lg(n)). Other properties such as vertex color and face texture are processed in the same way as geometry. Test shows that the algorithm is viable in real-time simplification for medium scale virtual models on PC platforms.

By employment of a footprint table in conducting intensity integration, splatting method has been very successful in rendering regular data volumes. Recently, the method has also been extended to render irregular data volumes. However, since samples in irregular volumes vary greatly in size and shape, the footprint table is unable to be employed in an efficient manner. This hinders the application of splatting approach from being used in the irregular volume case. In this paper, an adaptable splatting method is proposed, which provides an efficient way to integrate color intensity in terms of footprint table for the samples in various sizes. Experiments show that the new method may be used to produce better images without extra expense.

Selective visualization is a solution for visualizing data of large size and dimensionality. In this paper a new method is proposed for effectively rendering certain chosen parts among the full set of data in terms of a colour buffer, referred to as the virtual plane, for storing intermediate results. By this method, scientists may concentrate their attention on the contents of data in which they are interested. Besides, the method could be easily integrated with all the current direct volume rendering techniques, especially progressive refinement methods and selective methods. Copyright © 1999 John Wiley & Sons, Ltd.

Article A representation for composition of virtual indoor environment Share on Authors: Hongju Li Computer Science Lab., Institute of Software, Chinese Academy of Science, Beijing 100080, China Computer Science Lab., Institute of Software, Chinese Academy of Science, Beijing 100080, ChinaView Profile , Enhua Wu Computer Science Lab., Institute of Software, Chinese Academy of Science, Beijing 100080, China and Faculty of Science and Technology, University of Macau, P.O.Box 3001, Macau Computer Science Lab., Institute of Software, Chinese Academy of Science, Beijing 100080, China and Faculty of Science and Technology, University of Macau, P.O.Box 3001, MacauView Profile Authors Info & Claims VRST '98: Proceedings of the ACM symposium on Virtual reality software and technologyNovember 1998 Pages 171–178https://doi.org/10.1145/293701.293727Online:02 November 1998Publication History 0citation361DownloadsMetricsTotal Citations0Total Downloads361Last 12 Months2Last 6 weeks0 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access

Though radiosity method is an advanced rendering technique of global illumination, it is still not sufficiently to cope with many natural phenonmena such as furry surfaces. The rendering of furry surfaces has been a long outstanding problem in image synthesis. Therefore, it is of significance to develop an approach to incorporate the rendering of furry surfaces into radiosity method. By combining the concept of texel with the furry radiosity map , established from the radiosity calculation based on the proposed furry form-factor in the paper, a new radiosity algorithm has been developed to produce the images of furry surfaces.

The development of global illumination and texture generation makes it possible to produce the most realistic images. However, it is still difficult or deficient so far to simulate bump texture effects while the interreflection of light being modeled by the present ray tracing or radiosity methods. A method of bump texture generation, being incorporated into the process of radiosity solution, is presented in the paper. The method is characterized by introduction of a perturbed radiosity map, established in the context of either progressive radiosity or standard radiosity solution. To calculate the perturbed radiosity, a concept of perturbed form-factors is proposed, and the algorithms for evaluating the perturbed form-factors are also given. As a result, a bilinear-interpolation shading scheme for perturbed surfaces is provided, and the texturing method is easily added to a newly improved solution of progressive refinement radiosity for non-diffuse environment.