The success of current deep saliency models heavily depends on large amounts of annotated human fixation data to fit the highly non-linear mapping between the stimuli and visual saliency. Such fully supervised data-driven approaches are annotation-intensive and often fail to consider the underlying mechanisms of visual attention. In contrast, in this paper, we introduce a model based on various cognitive theories of visual saliency, which learns visual attention patterns in a weakly supervised manner. Our approach incorporates insights from cognitive science as differentiable submodules, resulting in a unified, end-to-end trainable framework. Specifically, our model encapsulates the following important components motivated from biological vision. (a) As scene semantics are closely related to visually attentive regions, our model encodes discriminative spatial information for scene understanding through spatial visual semantics embedding. (b) To model the objectness factors in visual attention deployment, we incorporate object-level semantics embedding and object relation information. (c) Considering the "winner-take-all" mechanism in visual stimuli processing, we model the competition mechanism among objects with softmax based neural attention. (d) Lastly, a conditional center prior is learned to mimic the spatial distribution bias of visual attention. Furthermore, we propose novel loss functions to utilize supervision cues from image-level semantics, saliency prior knowledge, and self-information compression. Experiments show that our method achieves promising results, and even outperforms many of its fully supervised counterparts. Overall, our weakly supervised saliency method makes an essential step towards reducing the annotation budget of current approaches, as well as providing a more comprehensive understanding of the visual attention mechanism. Our code is available at: https://github.com/ashleylqx/WeakFixation.git.

The selective visual attention mechanism in the human visual system (HVS) restricts the amount of information to reach visual awareness for perceiving natural scenes, allowing near real-time information processing with limited computational capacity. This kind of selectivity acts as an ‘Information Bottleneck (IB)’, which seeks a trade-off between information compression and predictive accuracy. However, such information constraints are rarely explored in the attention mechanism for deep neural networks (DNNs). In this paper, we propose an IB-inspired spatial attention module for DNN structures built for visual recognition. The module takes as input an intermediate representation of the input image, and outputs a variational 2D attention map that minimizes the mutual information (MI) between the attention-modulated representation and the input, while maximizing the MI between the attention-modulated representation and the task label. To further restrict the information bypassed by the attention map, we quantize the continuous attention scores to a set of learnable anchor values during training. Extensive experiments show that the proposed IB-inspired spatial attention mechanism can yield attention maps that neatly highlight the regions of interest while suppressing backgrounds, and bootstrap standard DNN structures for visual recognition tasks (e.g., image classification, fine-grained recognition, cross-domain classification). The attention maps are interpretable for the decision making of the DNNs as verified in the experiments. Our code is available at this https URL.

Human visual system can selectively attend to parts of a scene for quick perception, a biological mechanism known asHuman attention. Inspired by this, recent deep learning models encode attention mechanisms to focus on the most task-relevant parts of the input signal for further processing, which is calledMachine/Neural/Artificial attention. Understanding the relation between human and machine attention is important for interpreting and designing neural networks. Many works claim that the attention mechanism offers an extra dimension of interpretability by explaining where the neural networks look. However, recent studies demonstrate that artificial attention maps do not always coincide with common intuition. In view of these conflicting evidence, here we make a systematic study on using artificial attention and human attention in neural network design. With three example computer vision tasks (i.e., salient object segmentation, video action recognition, and fine-grained image classification), diverse representative backbones (i.e., AlexNet, VGGNet, ResNet) and famous architectures (i.e., Two-stream, FCN), corresponding real human gaze data, and systematically conducted large-scale quantitative studies, we quantify the consistency between artificial attention and human visual attention and offer novel insights into existing artificial attention mechanisms by giving preliminary answers to several key questions related to human and artificial attention mechanisms. Overall results demonstrate that human attention can benchmark the meaningful ‘ground-truth’ in attention-driven tasks, where the more the artificial attention is close to human attention, the better the performance; for higher-level vision tasks, it is case-by-case. It would be advisable for attention-driven tasks to explicitly force a better alignment between artificial and human attention to boost the performance; such alignment would also improve the network explainability for higher-level computer vision tasks.

We propose an interactive contour extraction method inspired by a skill often adopted in sketching: an artist usually sketches an object by first drawing lots of short, directional, and redundant strokes, then following these small strokes to draw the final outline of the object. Our method simulates this process. To extract a contour, our method relies on user interaction, which provides us with a narrow band containing the target contour. Then, we densely sample sub-bands from the whole band, with each sub-band containing a local segment of the target contour. We design a curve-centered coordinate system in which a dynamic programming algorithm is proposed to extract the local segment in each sub-band. The local segment is guaranteed to be as evident and smooth as possible, to mimic the strokes sketched by the artist. Finally, we integrate all local segments of all sub-bands together to obtain the whole target contour based on the weighted principal component analysis. Our method can extract high-quality object contours due to the dense validations among local segments. That is, even if one segment deviates from the right location, several other segments in its local neighborhood can correct it in the integration stage. Both quantitative experiments and a user study demonstrate the effectiveness of the proposed method.

In this paper, we propose a novel superpixel segmentation method using an adaptive nonlocal random walk (ANRW) algorithm. There are three main steps in our image superpixel segmentation algorithm. Our method is based on the random walk model, in which the seed points are produced to generate the initial superpixels by a gradient-based method in the first step. In the second step, the ANRW is proposed to get the initial superpixels by adjusting the NRW to obtain a better image and superpixel segmentation. In the last step, these small superpixels are merged to get the final regular and compact superpixels. The experimental results demonstrate that our method achieves a better superpixel performance than the state-of-the-art methods. Our source code will be available at: http://github.com/shenjianbing/ANRW.

This paper proposes a novel residual attentive learning network architecture for predicting dynamic eye-fixation maps. The proposed model emphasizes two essential issues, i.e, effective spatiotemporal feature integration and multi-scale saliency learning. For the first problem, appearance and motion streams are tightly coupled via dense residual cross connections, which integrate appearance information with multi-layer, comprehensive motion features in a residual and dense way. Beyond traditional two-stream models learning appearance and motion features separately, such design allows early, multi-path information exchange between different domains, leading to a unified and powerful spatiotemporal learning architecture. For the second one, we propose a composite attention mechanism that learns multi-scale local attentions and global attention priors end-to-end. It is used for enhancing the fused spatiotemporal features via emphasizing important features in multi-scales. A lightweight convolutional Gated Recurrent Unit (convGRU), which is flexible for small training data situation, is used for long-term temporal characteristics modeling. Extensive experiments over four benchmark datasets clearly demonstrate the advantage of the proposed video saliency model over other competitors and the effectiveness of each component of our network. Our code and all the results will be available at https://github.com/ashleylqx/STRA-Net.

This paper presents a new surveillance video synopsis method which performs much better than previous approaches in terms of both compression ratio and artifact. Previously, a surveillance video was usually compressed by shifting the moving objects of that video forward along the time axis, which inevitably yielded serious collision and chronological disorder artifacts between the shifted objects. The main observation of this paper is that these artifacts can be alleviated by changing the speed or size of the objects, since with varied speed and size the objects can move more flexibly to avoid collision points or to keep chronological relationships. Based on this observation, we propose a video synopsis method that performs object shifting, speed changing, and size scaling simultaneously. We show how to integrate the three heterogeneous operations into a single optimization framework and achieve high-quality synopsis results. Unlike previous approaches that usually use alternative optimization strategies to solve synopsis optimizations, we develop a Metropolis sampling algorithm to find the solution for our three-variable optimization problem. A variety of experiments demonstrate the effectiveness of our method.

We present a method for synopsizing multiple videos captured by a set of surveillance cameras with some overlapped field-of-views. Currently, object-based approaches that directly shift objects along the time axis are already able to compute compact synopsis results for multiple surveillance videos. The challenge is how to present the multiple synopsis results in a more compact and understandable way. Previous approaches show them side by side on the screen, which however is difficult for user to comprehend. In this paper, we solve the problem by joint object-shifting and camera view-switching. Firstly, we synchronize the input videos, and group the same object in different videos together. Then we shift the groups of objects along the time axis to obtain multiple synopsis videos. Instead of showing them simultaneously, we just show one of them at each time, and allow to switch among the views of different synopsis videos. In this view switching way, we obtain just a single synopsis results consisting of content from all the input videos, which is much easier for user to follow and understand. To obtain the best synopsis result, we construct a simultaneous object-shifting and view-switching optimization framework instead of solving them separately. We also present an alternative optimization strategy composed of graph cuts and dynamic programming to solve the unified optimization. Experiments demonstrate that our single synopsis video generated from multiple input videos is compact, complete, and easy to understand.

Video stabilization is usually composed of three stages: feature trajectory extraction, trajectory smoothing, and frame warping. Most previous approaches view them as three separate stages. This paper proposes a method combining the last two stages, namely the trajectory smoothing and frame warping stages, into a single optimization framework. The novelty exists in the way of how we combine them: the trajectory smoothing part plays a major role while the frame warping part plays an auxiliary role. With this kind of design, we can conveniently increase the strength of the trajectory smoothing part by a robust first-order derivative term, which makes it possible to produce very aggressive stabilization effects. On the other hand, we adopt adaptive weighting mechanisms in the frame warping part, to follow the smoothed trajectories as much as possible while regularizing other places as similar as possible. Our method is robust to utilize both foreground and background features, and very short trajectories. The utilization of all these information in turn increases the accuracy of the proposed method. We also provide a simplified implementation of our method, which is less accurate but more efficient. Experiments on various kinds of videos demonstrate the effectiveness of our method.

Currently, the most widely used point trajectories generation methods estimate the trajectories from the dense optical flow, by using a consistency check strategy to detect the occluded regions. However, these methods will miss some important trajectories, thus resulting in breaking smooth areas without any structure especially around the motion boundaries (MBs). We suggest exploring MBs in video to generate more accurate dense point trajectories. Estimating MBs from the video improves the point trajectory accuracy of the discontinuity or occluded areas. Then, we obtain trajectories by tracking the initial feature points through all frames. The experimental results demonstrate that our method outperforms the state-of-the-art methods on the challenging benchmark.

In this paper, we propose a new multiobject visual tracking algorithm by submodular optimization. The proposed algorithm is composed of two main stages. At the first stage, a new selecting strategy of tracklets is proposed to cope with occlusion problem. We generate low-level tracklets using overlap criteria and min-cost flow, respectively, and then integrate them into a candidate tracklets set. In the second stage, we formulate the multiobject tracking problem as the submodular maximization problem subject to related constraints. The submodular function selects the correct tracklets from the candidate set of tracklets to form the object trajectory. Then, we design a connecting process which connects the corresponding trajectories to overcome the occlusion problem. Experimental results demonstrate the effectiveness of our tracking algorithm. Our source code is available at https://github.com/shenjianbing/submodulartrack.

In visual tracking, usually only a small number of samples are labeled, and most existing deep learning based trackers ignore abundant unlabeled samples that could provide additional information for deep trackers to boost their tracking performance. An intuitive way to explain unlabeled data is to incorporate manifold regularization into the common classification loss functions, but the high computational cost may prohibit those deep trackers from practical applications. To overcome this issue, we propose a two-stage approach to a deep tracker that takes into account both labeled and unlabeled samples. The annotation of unlabeled samples is propagated from its labeled neighbors first by exploring the manifold space that these samples are assumed to lie in. Then, we refine it by training a deep convolutional neural network using both labeled and unlabeled data in a supervised manner. Online visual tracking is further carried out under the framework of particle filters with the presented manifold regularized deep model being updated every few frames. Experimental results on different tracking datasets demonstrate that our tracker outperforms most existing tracking approaches. The source code and results are available at: https://github.com/shenjianbing/MRCNNTracking.

This paper presents a non-photorealistic rendering technique for stylizing a photograph in the pencil drawing style, which can well preserve the fine structure of the original image. We first construct a structure map from the anti-color image of the original image to model the detailed underlying fine structure of the original image, and generate a coarse pencil drawing image with line integral convolution. Then we refine the coarse pencil drawing image with the structure map for enrich the structure information. The presented algorithm is highly parallel allowing a real-time performance with GPU implementation. Experimental results show that our approach can produce more attractive and impressive pencil drawing effects with a variety of photographs.

In this paper, we extend the sparse representation based image super-resolution method to process videos, mainly aiming at obtaining temporally consistent consecutive high-resolution (HR) video frames. In our formulation, the previous estimated HR frame is used to guide the sparse reconstruction of current low-resolution (LR) frame, which is able to obtain more consistent representations. We show that such guidance is robust and effective by incorporating with a non-rigid dense correspondence based motion compensation schema. We also propose a dictionary updating strategy which regularly updates the dictionaries that are critical for the sparse representation procedure using the newly reconstructed HR frames. To further preserve sharp edges and remove reconstruction errors, once a HR image is recovered, we refine it with a L0-norm based optimization that constrains the final HR output with relatively sparse gradients. Experimental results on natural videos demonstrated the effectiveness of our proposed method.

Abstract In this paper, we present a novel real‐time approach to generate high‐quality stippling on 3D scenes. The proposed method is built on a precomputed 2D sample sequence called incremental Voronoi set with blue‐noise properties. A rejection sampling scheme is then applied to achieve tone reproduction, by thresholding the sample indices proportional to the inverse target tonal value to produce a suitable stipple density. Our approach is suitable for stippling large‐scale or even dynamic scenes because the thresholding of individual stipples is trivially parallelizable. In addition, the static nature of the underlying sequence benefits the frame‐to‐frame coherence of the stippling. Finally, we propose an extension that supports stipples of varying sizes and tonal values, leading to smoother spatial and temporal transitions. Experimental results reveal that the temporal coherence and real‐time performance of our approach are superior to those of previous approaches.

Abstract In this paper, a novel method with good numerical stability is proposed from the perspective of energy preserving to alleviate the numerical dissipations in the advection step of Eulerian fluid simulation. The main idea is to measure the vorticity loss during advection, calculate the lost angular kinetic energy with a proposed scheme, and then synthesize a high‐frequency incompressible details field to compensate the lost energy in a way that is consistent with Kolmogorov's theory, which prevents the synthetic details from interfering with the existing fluid flow. The method works independently of the advection scheme and can be easily combined with other advection schemes to enhance the effect. It adds only 5% to 10% of the computational overhead while producing convincing fluid details without changing the overall behavior of the original flow.

We propose a new metric to predict perceived crosstalk using the original images rather than both the original and ghosted images. The proposed metrics are based on color information. First, we extract a disparity map, a color difference map, and a color contrast map from original image pairs. Then, we use those maps to construct two new metrics (Vdispc and Vdlogc). Metric Vdispc considers the effect of the disparity map and the color difference map, while Vdlogc addresses the influence of the color contrast map. The prediction performance is evaluated using various types of stereoscopic crosstalk images. By incorporating Vdispc and Vdlogc, the new metric Vpdlc is proposed to achieve a higher correlation with the perceived subject crosstalk scores. Experimental results show that the new metrics achieve better performance than previous methods, which indicate that color information is one key factor for crosstalk visible prediction. Furthermore, we construct a new data set to evaluate our new metrics.

We introduce the term video co-saliency to denote the task of extracting the common noticeable, or salient, regions from multiple relevant videos. The proposed video co-saliency approach accounts for both inter-video foreground correspondences and intra-video saliency stimuli to emphasize the salient foreground regions of video frames and, at the same time, disregard irrelevant visual information of the background. Compared with image co-saliency, it is more reliable due to the utilization of temporal information of video sequence. Benefiting from the discriminability of video co-saliency, we present a unified framework for segmenting out the common salient regions of relevant videos, guided by video co-saliency prior. Unlike naive video co-segmentation approaches employing simple color differences and local motion features, the presented video co-saliency provides a more powerful indicator for the common salient regions, thus conducting video co-segmentation efficiently. Extensive experiments show that the proposed method successfully infers video co-saliency and extracts the common salient regions, outperforming the state-of-the-art methods.

Stitching videos captured by hand-held mobile cameras can essentially enhance entertainment experience of ordinary users. However, such videos usually contain heavy shakiness and large parallax, which are challenging to stitch. In this paper, we propose a novel approach of video stitching and stabilization for videos captured by mobile devices. The main component of our method is a unified video stitching and stabilization optimization that computes stitching and stabilization simultaneously rather than does each one individually. In this way, we can obtain the best stitching and stabilization results relative to each other without any bias to one of them. To make the optimization robust, we propose a method to identify background of input videos, and also common background of them. This allows us to apply our optimization on background regions only, which is the key to handle large parallax problem. Since stitching relies on feature matches between input videos, and there inevitably exist false matches, we thus propose a method to distinguish between right and false matches, and encapsulate the false match elimination scheme and our optimization into a loop, to prevent the optimization from being affected by bad feature matches. We test the proposed approach on videos that are causally captured by smartphones when walking along busy streets, and use stitching and stability scores to evaluate the produced panoramic videos quantitatively. Experiments on a diverse of examples show that our results are much better than (challenging cases) or at least on par with (simple cases) the results of previous approaches.Stitching videos captured by hand-held mobile cameras can essentially enhance entertainment experience of ordinary users. However, such videos usually contain heavy shakiness and large parallax, which are challenging to stitch. In this paper, we propose a novel approach of video stitching and stabilization for videos captured by mobile devices. The main component of our method is a unified video stitching and stabilization optimization that computes stitching and stabilization simultaneously rather than does each one individually. In this way, we can obtain the best stitching and stabilization results relative to each other without any bias to one of them. To make the optimization robust, we propose a method to identify background of input videos, and also common background of them. This allows us to apply our optimization on background regions only, which is the key to handle large parallax problem. Since stitching relies on feature matches between input videos, and there inevitably exist false matches, we thus propose a method to distinguish between right and false matches, and encapsulate the false match elimination scheme and our optimization into a loop, to prevent the optimization from being affected by bad feature matches. We test the proposed approach on videos that are causally captured by smartphones when walking along busy streets, and use stitching and stability scores to evaluate the produced panoramic videos quantitatively. Experiments on a diverse of examples show that our results are much better than (challenging cases) or at least on par with (simple cases) the results of previous approaches.

In[1], the biographies of Hanqiu Sun and Guiqing Li included incorrect information. The correct biographies are as follows.

Turbulent vortices in smoke flows are crucial for a visually interesting appearance. Unfortunately, it is challenging to efficiently simulate these appealing effects in the framework of vortex filament methods. The vortex filaments in grids scheme allows to efficiently generate turbulent smoke with macroscopic vortical structures, but suffers from the projection-related dissipation, and thus the small-scale vortical structures under grid resolution are hard to capture. In addition, this scheme cannot be applied in wall-bounded turbulent smoke simulation, which requires efficiently handling smoke-obstacle interaction and creating vorticity at the obstacle boundary. To tackle above issues, we propose an effective filament-mesh particle-particle (FMPP) method for fast wall-bounded turbulent smoke simulation with ample details. The Filament-Mesh component approximates the smooth long-range interactions by splatting vortex filaments on grid, solving the Poisson problem with a fast solver, and then interpolating back to smoke particles. The Particle-Particle component introduces smoothed particle hydrodynamics (SPH) turbulence model for particles in the same grid, where interactions between particles cannot be properly captured under grid resolution. Then, we sample the surface of obstacles with boundary particles, allowing the interaction between smoke and obstacle being treated as pressure forces in SPH. Besides, the vortex formation region is defined at the back of obstacles, providing smoke particles flowing by the separation particles with a vorticity force to simulate the subsequent vortex shedding phenomenon. The proposed approach can synthesize the lost small-scale vortical structures and also achieve the smoke-obstacle interaction with vortex shedding at obstacle boundaries in a lightweight manner. The experimental results demonstrate that our FMPP method can achieve more appealing visual effects than vortex filaments in grids scheme by efficiently simulating more vivid thin turbulent features.

We present an interactive and stable multi-contact dynamic simulation algorithm for rigid bodies. Our approach is based on fast frictional dynamics (FFD) [14], which is designed for large sets of non-convex rigid bodies. We use a new friction model that performs velocity-level multi-contact simulation using impulse decomposition. Moreover, we accurately handle friction at each contact point using contact distribution and frictional impulse solvers, which also account for relative motion. We evaluate our algorithm's performance on many complex multi-body benchmarks with thousands of contacts. In practice, our dynamics simulation algorithm takes a few milliseconds per timestep and exhibits more stable behaviors.

Previous works on palette-based color manipulation typically fail to produce visually pleasing results with vivid color and natural appearance. In this paper, we present an approach to edit colors of an image by adjusting a compact color palette. Different from existing methods that fail to preserve inherent color characteristics residing in the source image, we propose a color decomposition optimization for flexible recoloring while retaining these characteristics. For an input image, we first employ a variant of the k -means algorithm to create a palette consisting of a small set of most representative colors. Next, we propose a color decomposition optimization to decompose colors of the entire image into linear combinations of basis colors in the palette. The captured linear relationships then allow us to recolor the image by recombining the coding coefficients with a user-modified palette. Qualitative comparisons with existing methods show that our approach can more effectively recolor images. Further user study quantitatively demonstrates that our method is a good candidate for color manipulation tasks. In addition, we showcase some applications enabled by our method, including pattern colorings suggesting, color transfer, tissue staining analysis and color image segmentation.

Wide-baseline street image interpolation is useful but very challenging. Existing approaches either rely on heavyweight 3D reconstruction or computationally intensive deep networks. We present a lightweight and efficient method which uses simple homography computing and refining operators to estimate piecewise smooth homographies between input views. To achieve the goal, we show how to combine homography fitting and homography propagation together based on reliable and unreliable superpixel discrimination. Such a combination, other than using homography fitting only, dramatically increases the accuracy and robustness of the estimated homographies. Then, we integrate the concepts of homography and mesh warping, and propose a novel homography-constrained warping formulation which enforces smoothness between neighboring homographies by utilizing the first-order continuity of the warped mesh. This further eliminates small artifacts of overlapping, stretching, etc. The proposed method is lightweight and flexible, allows wide-baseline interpolation. It improves the state of the art and demonstrates that homography computation suffices for interpolation. Experiments on city and rural datasets validate the efficiency and effectiveness of our method.

Cloud simulation is important for creating images of outdoor scenes. However, the complexity of this natural phenomenon makes the simulation of large-scale clouds difficult in real time. In this paper, we present a new method for 3D cloud simulation in which cloud animation is simplified and simulated by approximating Lennard-Jones Potential. To solve the N-body problem in Lennard-Jones Potential, we minimized the interaction between particles by dividing the simulation space into many cells and we defined a cutoff distance to perform calculation between neighboring particles. Additionally, a separate distance is introduced between particles to maintain the stability in the Lennard-Jones system. Our experimental results demonstrate that our method is computationally inexpensive and suitable for real time applications where large-scale simulation of clouds is required.

Abstract Image mosaic effects are wildly applied in print media, domestic decoration, and many image beautification applications. However, the current image mosaic methods are mostly based on fixed‐size image tiles, simple color adjustment, and irregular image segmentation, which are inaccurate and very time‐consuming. In this paper, we present a graphics processing unit‐accelerated perceptual mosaic using density tiles replacement and brightness lighting optimization, keeping original image structure details and providing more expressive visual effects. Automatic density replacement map segmentation and color‐based region tiles replacement are performed to facilitate the mosaic. Delicate brightness optimization and perceptual color correction are further applied to enhance expressive lighting effects. We also consider the salience perception of images and similarity correlation among neighboring tiles for our perceptual mosaic. The experimental results have shown the efficiency and high‐quality performance of our density‐enhanced perceptual mosaic on graphics processing unit. Copyright © 2016 John Wiley & Sons, Ltd.

Supervoxels have been widely used as a preprocessing step to exploit object boundaries to improve the performance of video processing tasks. However, most of the traditional supervoxel algorithms do not perform well in regions with complex textures or weak boundaries. These methods may generate supervoxels with overlapping boundaries. In this paper, we present the novel video supervoxel generation algorithm using partially absorbing random walks to get more accurate supervoxels in these regions. Our spatial-temporal framework is introduced by making full use of the appearance and motion cues, which effectively exploits the temporal consistency in video sequence. Moreover, we build a novel Laplacian optimization structure using two adjacent frames to make our approach more efficient. Experimental results demonstrated that our method achieved better performance than the state-of-the-art supervoxel algorithms.

In this paper, we propose a novel approach to generate accurate soft shadows for complex virtual scenes in real time. A Multi-Layer Occluder buffer (MLO-buffer) is used in our algorithm, and it is determined by rendering the scene from the center of area light source. MLO-buffer records all the potential occluders in the scene. When calculating shadow factors, area light source is represented by a set of point samples, and MLO-buffer is traversed to figure out the visibility relations between an occludee and each light sample. Finally, the accurate soft shadows can be produced. The experimental results show that our algorithm can generate high-quality soft shadows in real time without making any assumption, like shape orientation or density distribution, on the area light source.

We introduce an adaptive hybrid level set surface tracking for fluid simulation in this paper. To improve the accuracy of the convectional level set method, we propose a method that combines the higher accuracy interpolation and octree grid to improve the accuracy while still maintain good efficiency. Adaptive gradient-augmented level set is adopted for higher accuracy interpolation and two strategies are proposed to guide the building of the octree structure. We demonstrate our method with several level set tests, and couple it to several existing fluid simulators. The results show that our method facilitates significant improvements in accuracy and efficiency, and is effective for free surface fluid simulation in capturing fluid details.

Solid textures exhibit several benefits over traditional texture mapping, such as no demanding for parameterization and providing internal information. However, the difficulty in accessing and high memory consumption seriously limit the usage of solid textures. We present an efficient approach to directly synthesize vector solid textures comprising 3D particles from 2D examples. Two-scale shaping model is introduced to automatically and accurately reconstruct 3D particle outlines from 2D cross sections. Low frequency spherical harmonics are used to morph smooth outlines while high frequency ones model particle surface details. These outlines are further converted to signed distance field grids for vector representation. Particle volume color is represented by radial basis functions. We arrange particles using simplified particle proxies to improve efficiency. Particles' position offsets are calculated according to the particle itself and one-ring neighbor particles' shapes iteratively. Finally, our cell-building algorithm constructs containers and computes scale for each particle instance. Experiments show that our algorithm can generate vector solid textures with plausible 3D particles.

Image-based rendering (IBR) can render novel views from several existing photographs of the same scene. Current IBR methods can be roughly classified into reconstruction-based methods and warping-based methods. Warping-based methods [Liu et al. 2009; Chaurasia et al. 2011] directly warped the triangle mesh overlaid on the source image to make the source image match with the target one. Then the novel views between the two given images can be computed by interpolating and retexturing the intermediate mesh.

Simultaneous multi-impact simulation is a challenging problem that frequently arises in physically-based modeling of rigid bodies. There are several physical criteria that should be satisfied for rigid body collision handling, but existing methods generally fail to meet one or more of them. In order to capture the inner process of potential energy variation, which is the physical foundation of collisions in a multi-impact system, we present a novel quadratic contact energy model for rigid body simulation. By constructing quadratic energy functions with respect to the impulses, post-impact reactions of rigid bodies can be computed efficiently. Our model can satisfy the physical criteria and can simulate various natural phenomena including the wave effect. Also, our model can be easily combined with Linear Complementary Problem (LCP) and can provide feasible results with any restitution coefficient. In practice, our model can solve the simultaneous multi-impact problem efficiently and robustly, and we highlight its performance on different benchmarks.

Abstract In virtual environments, real‐time simulation and rendering of dynamic fluids have always been the pursuit for virtual reality research. In this paper, we present a real‐time framework for realistic fluid simulation and rendering on graphics processing unit. Because of the high demand for interactive fluids with larger particle set, the computational need is becoming higher. The proposed framework can effectively reduce the computational burden through avoiding the computation in inactive areas, where many particles with similar properties and low local pressure cluster together. While in active areas, the computation is fully carried out; thus, the fluid dynamics are largely preserved. Here, a robust particle classification technique is introduced to classify particles into either active or inactive. The test results have shown that the technique improves the time performance of fluid simulation largely. We then incorporate parallel surface reconstruction technique using marching cubes to extract the surfaces of the fluid. The introduced histogram pyramid‐based marching cubes technique is fast and memory efficiency. As a result, we are able to produce plausible and interactive fluids with the proposed framework for large‐scale virtual environments. Copyright © 2013 John Wiley & Sons, Ltd.

We present a new scheme for video colorization using optimization in rotation-aware Gabor feature space. Most current methods of video colorization incur temporal artifacts and prohibitive processing costs, while this approach is designed in a spatiotemporal manner to preserve temporal coherence. The parallel implementation on graphics hardware is also facilitated to achieve realtime performance of color optimization. By adaptively clustering video frames and extending Gabor filtering to optical flow computation, we can achieve real-time color propagation within and between frames. Temporal coherence is further refined through user scribbles in video frames. The experimental results demonstrate that our proposed approach is efficient in producing high-quality colorized videos.

Video synopsis aims at removing video's less important information, while preserving its key content for fast browsing, retrieving, or efficient storing. Previous video synopsis methods, including frame-based and object-based approaches that remove valueless whole frames or combine objects from time shots, cannot handle videos with redundancies existing in the movements of video object. In this paper, we present a novel part-based object movements synopsis method, which can effectively compress the redundant information of a moving video object and represent the synopsized object seamlessly. Our method works by part-based assembling and stitching. The object movement sequence is first divided into several part movement sequences. Then, we optimally assemble moving parts from different part sequences together to produce an initial synopsis result. The optimal assembling is formulated as a part movement assignment problem on a Markov Random Field (MRF), which guarantees the most important moving parts are selected while preserving both the spatial compatibility between assembled parts and the chronological order of parts. Finally, we present a non-linear spatiotemporal optimization formulation to stitch the assembled parts seamlessly, and achieve the final compact video object synopsis. The experiments on a variety of input video objects have demonstrated the effectiveness of the presented synopsis method.

In dynamic fluid simulation, the user often need to control the local regions of the entire fluid scene for the special design effects, such as flowing marks and icons in dynamic fluids. Currently, the research aimed at such an issue is rare. It is rather difficult to realize the local feature effects using the global control methods. This paper proposes a novel local-control method for dynamic fluid simulation. Specifically, our method enables the generation vivid local features according to the pattern image specified by the user. We first search the whole fluid space to find the local areas best matched to the input pattern image. The special driving force is calculated to shape the local areas into the target pattern. In order to ensure the continuity of the whole velocity field, we mix the velocity fields of the local area and the one of its corresponding position in the global zone. Our method can provide the user or designers with the local-control tool for editing the dynamic fluid scenes intuitively. The dynamic fluid simulations with the local features have been experimented, which showed the validity and effectiveness of our method.

Video synopsis aims at providing condensed representations of video data sets that can be easily captured from digital cameras nowadays, especially for daily surveillance videos. Previous work in video synopsis usually moves active objects along the time axis, which inevitably causes collisions among the moving objects if compressed much. In this paper, we propose a novel approach for compact video synopsis using a unified spatiotemporal optimization. Our approach globally shifts moving objects in both spatial and temporal domains, which shifting objects temporally to reduce the length of the video and shifting colliding objects spatially to avoid visible collision artifacts. Furthermore, using a multilevel patch relocation (MPR) method, the moving space of the original video is expanded into a compact background based on environmental content to fit with the shifted objects. The shifted objects are finally composited with the expanded moving space to obtain the high-quality video synopsis, which is more condensed while remaining free of collision artifacts. Our experimental results have shown that the compact video synopsis we produced can be browsed quickly, preserves relative spatiotemporal relationships, and avoids motion collisions.

Recently, image registration approaches based on local feature matching have revealed promising results in the fields of natural and medical image processing. However, the performance becomes worse when it is applied to high-resolution satellite image. This arises because the complex nature of this kind of image, which results in a lot of outliers in the initial matches. Therefore, how to select reliable matches from low-precision initial matches becomes very crucial. In this paper, we modify the traditional SIFT descriptor and propose a novel algorithm to identify outliers from initial SIFT matches, which is based on the consistence of local contour fragments between two matched regions. The experimental results show that the presented approach can extract high-precision matches from low-precision initial SIFT matches, and is efficient for high-resolution satellite image registration.

In this paper, we present a novel color-mood-aware technique to re-texture clothing in a photograph. An efficient classification algorithm is developed to classify clothing textures using color mood scheme. To re-texture the clothing, our approach first computes the gradient maps for the cloth region to be replaced and then calculates the texture distortion coordinates on the projected cloth region according to the gradient maps. After the user selects a target clothing texture from the classified clothing texture database, the lighting and shading effects on the original photograph is transferred using the HSV color space. Experimental results show that the proposed approach successfully re-textures the clothes in photographs while preserving the geometry and lighting features.

This paper presents a method for abstracting multi-keyframe from video datasets. Existing video abstraction methods focused on simple view videos, and the results will be unacceptable if applied to overlapping views directly due to limitations like unavoidable redundancy and complicated inner correlations. We propose a correlation map to naturally model the correlations with various attributes among multi-keyframe, keyframe importance and weighted correlations are then computed to construct the map. The weighted correlations, unlike the unweighted ones, not only model probabilistic relationship among keyframes but also address the temporal and visual similarity. We facilitate the abstraction process via SVM classification and keyframes reduction using rough set. The multi-keyframe correlation map, which serially assembles event-centered keyframes in temporal order, is presented for displaying the abstraction, which shows the correlations and improves the browsability of video datasets.

Abstract Physics‐based simulation is usually complex and time consuming, and consequently not suitable for real‐time applications. In this paper, we propose the efficient dynamics models for the real‐time simulation of soft fabrics interacting with water, including multi‐soaking effect and underwater dynamics. The multi‐soaking effect of soft fabrics is modeled based on the physics processes. Further, we develop the optimized mass–spring model that supports the large flow forces interacting with the fabrics underwater. The fabric spring forces are linearly derived and integrated with GPU–CUDA acceleration, feasible for real‐time VR applications with large set of fabric particles. Copyright © 2011 John Wiley & Sons, Ltd.

Abstract In recent years, some wavelets constructed via local lifting have been proposed to accelerate the compression of geometric models. Unlike the traditional wavelet‐based compression with global optimisation, their compression ratios are still not sufficient, particularly at low‐bit rates. In this paper, we present a novel efficient wavelet‐based compression approach using the matrix‐valued Loop subdivision on triangular meshes. Based on the local lifting scheme, the computation of the wavelet transform is highly efficient and requires low memory usage. With the shape controls introduced by the matrix‐valued subdivision, our approach can greatly improve the fitting quality of surfaces while keeping the efficient data compression. The experimental results showed that our wavelet transform had better performance on balancing the efficiency and compression ratios. Copyright © 2011 John Wiley & Sons, Ltd.

Multiresolution rendering of deformable models, performing fast rendering of global deformations while preserving local surface details, is usually a computation-costly and time-consuming process, because two non-trivial operators are involved. We propose a novel GPU-based adaptive rendering of deforming mesh sequence on sole-cube maps (SCM), which is a variant of geometry images built upon spherical parameterizations. We also introduce the differential coordinates to bound the local resampling error for supporting details preservation and view-dependent visualization. By precomputing the adaptive SCM texture atlas of deforming mesh sequences as well as their differential coordinates, we can map both the deformation and the level-of-details (LOD) operator to the GPU. The proposed algorithm enables us to reconstruct the deformed positions and sufficiently fine-scale approximations of deforming mesh sequences for efficient GPU processing. The GPU-friendly data structure and process allow us to render dynamically deforming 3D models with GPU parallelization, also our system improves the efficiency of the manipulations of node selection and boundary stitching, significantly alleviating the computing load on CPU.

Abstract Adding interactive haptic‐constraint sensations is important in interactive computer gaming and 3D shape design. Usually constraints are set on vertices of the object to drive the deformation. How to simulate dynamic force constraints in interactive design is still a challenging task. In this paper, we propose a novel haptic‐constraint modeling method based on interactive metaballs, during which the haptic‐constraint tools are attracted to the target location and then control the touch‐enabled deformation within the constrained areas. The interactive force feedbacks facilitate designers to accurately deform the target regions and fine carve the details as their intention on the objects. Our work studies how to apply touch sensation in such constrained deformations using interactive metaballs, thus users can truly feel and control the soft‐touch objects during the deforming interactions. Experimental results show that the dynamic sense of touch during the haptic manipulation is intuitively simulated to users, via the interacting interface we have developed. Copyright © 2010 John Wiley & Sons, Ltd.

This paper presents a novel bilateral filtering using fuzzy-median for image manipulations such as denoising and tone mapping. Our proposed bilateral filtering consists of the standard bilateral filter and the estimation of the pixel values by the fuzzy median filter. We have applied the proposed fuzzy filtering for image denoising with both the impulse and Gaussian random noise, which achieves better results than the bilateral filtering based denoising approaches, the Perona-Maliks anisotropic diffusion filter, the fuzzy vector median filter and the non-local means filter. Further, we develop the tone mapping algorithm of high dynamic range images incorporating the proposed fuzzy filtering, which does not introduce unpleasant visual halo artifacts.

We present a new approach for quickly designing 3D models of botanical trees using an iterative addition of new nodes to the tree branch structure. This process is guided by the proximity of points marking volume density data captured from photographs. Numerical parameters provide the user controls that are consistent with the characteristics of trees in landscaping and make it possible to generate a wide variety of tree styles. Meanwhile we synthesize visually believable motions for the generated tree models affected by a wind field. Our system enables the simulation of tree animation, by introducing physically-based transformation matrix calculations for hierarchical branch patterns. The system also supports the tree-shaping modes in which many branches and leaves are generated by interactively designed their distribution density. Experimental results show that our approach can design a variety of reasonably natural-looking trees and their motions.

This paper presents a new GPU-based implementation of fast non-local means (NLM) filtering for material restyling. Our fast NLM filtering algorithm is able to achive realtime feedback of interactive image editing. Furthermore a novel material editing method based on our fast NLM filtering is proposed to change the material appearance of image-based objects. Given an input image, an alpha matte is created to differentiate the object from its background. After the automatic matting process, the object is removed from the background, and the 3D shape of the object is recovered. We use the fast non-local means (NLM) filtering to process the luminance channel of an image and obtain a pseudo-depth map that is sufficient for altering the material appearance of the observed object. We recover the gradient luminance maps for the region to be material-restyled; and change the material property in the region-of-interest by solving the Poisson equation. In this way, the new material is mapped onto the 3D shape, resulting in an object which appears to be made of a different material. Our new NLM filtering-based material editing is easy to implement in parallel with graphics hardware. The experimental results have demonstrated the satisfactory performance of our method.

In this paper we present a novel tone mapping algorithm for high dynamic range (HDR) images using the retinal adaptation model (RAM). The physiological evidence suggests that the RAM is obtained by measuring intensity-response functions to flashes of light presented under varying adaptation conditions, which leads to a theoretic-sound model that can be flexibly adapted for tone reproduction. The multiplicative-subtractive process of the model can provide high quality tone mapping results for rendering the HDR images. The experimental results demonstrate that our RAM-based tone mapping approach is effective to produce pleasing results on HDR images in a wide range of real-world scenarios.

Abstract Embedding geometric textures in a shell space around an arbitrary surface has been a popular way to add highly detailed geometric details and enhance visual richness in graphics community, but the dynamic effects of geometric textures have not been modeled and simulated. In this paper, we introduce an efficient algorithm for deforming geometric textures with dynamic effects. The algorithm consists of two steps. First, it computes a deformed shell space by optimizing a material related energy function, which is then used to evaluate the equilibrium position of the geometric texture. Second, an explicit time integration scheme is applied for vibrating the geometric texture around its equilibrium position. Users can deform the geometric textures by dragging its vertices directly, and the dynamic behavior of the geometric textures can be changed by adjusting several material parameters. The dynamic simulation of geometric textures can be easily implemented on GPU and runs at real‐time rates. Copyright © 2009 John Wiley & Sons, Ltd.

Abstract This paper presents a new method for modeling and rendering fur with a wide variety of furstyles. We simulate virtual fur using shell textures—a multiple layers of textured slices for its generality and efficiency. As shell textures usually suffer from the inherent visual gap errors due to the uniform discretization nature, we present theangle‐split shell textures(ASST) approach, which classifies the shell textures into different types with different numbers of texture layers, by splitting the angle space of the viewing angles between fur orientation and view direction. Our system can render the fur with biological patterns, and utilizes vector field and scalar field on ASST to control the geometric variations of the furry shape. Users can intuitively shape the fur by applying the combing, blowing, and interpolating effects in real time. Our approach is intuitive to implement without using complex data structures, with real‐time performance for dynamic fur appearances. Copyright © 2009 John Wiley & Sons, Ltd.

Abstract In this paper a biorthogonal wavelet approach based on Doo‐Sabin subdivision is presented. In the dual subdivision like Doo‐Sabin scheme, all the old control vertices disappear after one subdivision step, which is a big challenge to the biorthogonal wavelet construction. In our approach, the barycenters of the V‐faces corresponding to the old vertices are selected as the vertices associated with the scaling functions to construct the scaling space. The lifting scheme is used to guarantee the fitting quality of the wavelet transform, and a local orthogonalization is introduced with a discrete inner product operation to improve the computation efficiency. Sharp feature modeling based on extended Doo‐Sabin subdivision rules is also discussed in the framework of our wavelet construction. The presented wavelet construction is proven to be stable and effective by the experimental results.

Abstract Dynamic textures are composed of spatially multi‐scale and temporally coherent visual signals. In this paper, we propose a novel approach for multi‐resolution parametric synthesis (MPS) of manipulative dynamic textures, including learning, analysis, and motion manipulation of layered feature spaces. The proposed approach separates the motions into different scales or frequencies, and the dynamic feature spaces are analyzed in modeling and then mapped to a global coordinate system to expose the multi‐scale structures. Our approach emphasizes the importance of fine‐scale data appearing more informative, while coarse signals are less focused. The synthesized dynamic textures using our MPS approach are strongly enhanced in dynamic appearance than the single‐scale LDS model, and also flexible in manipulating the decomposed feature spaces for creating different dynamic textures of the real world. Our experimental results showed the more faithful synthesized dynamic textures from input short videos, and their manipulative controls by simple editing of feature space basis, not only the whole video data. Copyright © 2008 John Wiley & Sons, Ltd.

Abstract Shadow mapping has been extensively used for real‐time shadow rendering in 3D computer games, though it suffers from the inherent aliasing problems due to its image‐based nature. This paper presents an enhanced variant of light space perspective shadow maps to optimize perspective aliasing distribution in possible general cases where the light and view directions are not orthogonal. To be mathematically sound, the generalized representation of perspective aliasing errors has been derived in detail. Our experiments have shown the enhanced shadow quality using our algorithm in dynamic scenes. Copyright © 2008 John Wiley & Sons, Ltd.

This paper presents an approach of replacing textures of specified regions in the input image and video using stretch-based mesh optimization.. The retexturing results have the similar distortion and shading effects conforming to the underlying geometry and lighting conditions. For replacing textures in single image,two important steps are developed: the stretch-based mesh parametrization incorporating the recovered normal information is deduced to imitate perspective distortion of the region of interest; the Poisson-based refinement process is exploited to account for texture distortion at fine scale. The luminance of the input image is preserved through color transfer in YCbCr color space. Our approach is independent of the replaced textures. Once the input image is processed, any new texture can be applied to efficiently generate the retexturing results. For video retexturing, we propose key-frame-based texture replacement extended and generalized from the image retexturing. Our approach repeatedly propagates the replacement result of key frame to the rest of the frames. We develop the local motion optimization scheme to deal with the inaccuracies and errors of robust optical flow when tracking moving objects. Visibility shifting and texture drifting are effectively alleviated using graphcut segmentation algorithm and the global optimization to smooth trajectories of the tracked points over temporal domain. Our experimental results showed that the proposed approach can generate visually pleasing results for both image and video.

This paper presents the layer-based representation of polyhedrons and its use for point-in-polyhedron tests. In the representation, the facets and edges of a polyhedron are sequentially arranged, and so, the binary search algorithm is efficiently used to speed up inclusion tests. In comparison with conventional representation for polyhedrons, the layer-based representation we propose greatly reduces the storage requirement because it represents much information implicitly, though it still has a storage complexity O(n). It is simple to implement, and robust for inclusion tests because many singularities are erased in constructing the layer-based representation. By incorporating an octree structure for organizing polyhedrons, our approach can run at a speed comparable with Binary Space Partitioning (BSP)-based inclusion tests, and at the same time greatly reduce storage and preprocessing time in treating large polyhedrons. We have developed an efficient solution for point-in-polyhedron tests with the time complexity varying between O(n) and O(logn), depending on the polyhedron shape and the constructed representation, and less than O(log3n) in most cases. The time complexity of preprocess is between O(n) and O(n2), varying with polyhedrons, where n is the edge number of a polyhedron.

In this paper, we propose a novel method that represents the highly-complex point sets by clustering the points to normal-mapped B-spline surfaces (NBSs). The main idea is to construct elaborate normal maps on simple surfaces for the realistic rendering of complex point-set models. Based on this observation, we developed the coarse, normal-mapped B-spline surfaces to approximate the original point datasets with fine surface details. In our algorithm, a genetic clustering algorithm is proposed to automatically segment the point samples into several clusters according to their statistical properties, and a network of B-spline patches with normal maps are constructed according to the clustering results. Our experimental results show that this representation facilitates the modeling and rendering of complex point sets without losing the visual qualities.

Abstract It is crucial for the users to touch, grasp and manipulate the interested objects through our sense of touch in many interactive applications, such as on‐line computer games, interactive cartoon design, and virtual prototyping. In this paper, we propose an interactive haptic deformation approach which incorporates the dynamic simulation of mass–spring systems and flexible control of free‐form deformation in the touch‐enabled soft‐object deformation. Through distributing mass, spring and damping coefficients of the object to the bounded Bezier volume lattice, the deformation of the object related to the haptic avatar follows the physical laws and has high working rate. Both homogenous and inhomogenous materials are simulated. The anchor nodes of haptic input are specified to create amazing special effects during the interactive haptic deformation. Interactive haptic deformations of three‐type tropic fishes, Angel, Demekin, and GuppyBlueGrass, have been experimented to simulate vivid fish swimming processes in the virtual ocean scene. Our proposed approach provides touch‐enabled input and efficient performance in the flexible deforming controls, letting the objects move in a dynamic, cartoon‐style deforming manner. Copyright © 2007 John Wiley & Sons, Ltd.

Abstract Palpation is an important method of feeling with hands during a physical examination, in which the doctor presses on the surface of the patient body to feel the organs or tissues underneath. In current surgical simulation systems, the lack of an effective sense of touch is still a major problem. In this paper, a dynamic touch‐enabled virtual palpation model is proposed. The palpation force sensing between the index finger and virtual tissues is simulated through a body‐based haptic interaction model. Both contact and frictional forces are evaluated based on Hertz's contact theory, and the press distribution within the contact area is also specified. The non‐linear viscoelastic behavior of typical tissues is mimicked via a volumetric tetrahedral mass‐spring system. Reaction during the palpation is restricted to a local area to highly reduce the order of the dynamic equation of the entire system to guarantee a fast working rate. Mechanical tests have been performed to evaluate the palpation force perception and the realistic behavior of typical human tissues. Copyright © 2007 John Wiley & Sons, Ltd.

Abstract In computer graphics community, face model is one of the most useful entities. The automatic detection of 3D face model has special significance to computer graphics, vision, and human‐computer interaction. However, few methods have been dedicated to this task. This paper proposes a machine learning approach for fully automatic 3D face detection. To exploit the facial features, we introducegeometric context, a novel shape descriptor which can compactly encode the distribution of local geometry and can be evaluated efficiently by using a new volume encoding form, namedintegral volume. Geometric contexts over 3D face offer the rich and discriminative representation of facial shapes and hence are quite suitable to classification. We adopt an AdaBoost learning algorithm to select the most effective geometric context‐based classifiers and to combine them into a strong classifier. Given an arbitrary 3D model, our method first identifies the symmetric parts as candidates with a new reflective symmetry detection algorithm. Then uses the learned classifier to judge whether the face part exists. Experiments are performed on a large set of 3D face and non‐face models and the results demonstrate high performance of our method. Copyright © 2007 John Wiley & Sons, Ltd.

A new efficient biorthogonal wavelet analysis based on the principal square root of subdivision is proposed in the paper by using the lifting scheme. Since the principal square root of subdivision is of the slowest topological refinement among the traditional triangular subdivisions, the multiresolution analysis based on the principal square root of subdivision is more balanced than the existing wavelet analyses on triangular meshes, and accordingly offers more levels of detail for processing polygonal models. In order to optimize the multiresolution analysis process, the new wavelets, no matter whether they are interior or on boundaries, are orthogonalized with the local scaling functions based on a discrete inner product with subdivision masks. Because the wavelet analysis and synthesis algorithms are actually composed of a series of local lifting operations, they can be performed in linear time. The experiments demonstrate the efficiency and stability of the wavelet analysis for both closed and open triangular meshes with principal square root of subdivision connectivity. The principal square root of -subdivision-based biorthogonal wavelets can be used in many applications such as progressive transmission, shape approximation, multiresolution editing and rendering of 3D geometric models.

Most applications in virtual and augmented reality usually need to visualize large amounts of extra text/image/video annotations associated with objects to assist user's navigation. Labeling is one of the most intuitive techniques to achieve this goal, which attaches specific texts to related objects to provide users a clear environmental clue. The basic problem of labeling is the clutters due to unexpected overlapping and occlusion among labels and objects, which makes the environment ambiguous and obscure. In this paper, we present the interactive system of dynamic labeling management to address this problem for virtual environment navigation applications. Our system integrates the techniques of labeling with adaptive placement, view-driven label filtering and structured label searching. These features have been developed in our system with well display of augmented labels in VR applications.

Abstract In this paper, we propose a novel mesh morphing approach based on polycubic cross‐parameterization. We compose parameterizations over the surfaces of the polycubes whose shape is similar to that of the given meshes. Because the polycubes capture the large‐scale features, we can easily preserve the shape of the models, mapping legs to legs, head to head, and so on. For the finer features that are not reflected by the shape of the polycubes, we split the polycubes into matching patches and optimize them to get a low‐distortion bijection that satisfies user‐prescribed constraints. Our approach works well for meshes with arbitrary genus as long as the polycubes capture this feature and transfers texture seamlessly. We can also build maps with singularities between models with different genus. Copyright © 2005 John Wiley & Sons, Ltd.

Abstract We propose a novel image/video retexturing approach that preserves the original shading effects without knowing the underlying surface and lighting conditions. For static images, we first introduce the Poisson equation‐based algorithm to simulate the texture distortion on the projected interest region of the underlying surface, while preserving the shading effect of the original image. We further work on videos by retexturing the key frame as static image and then propagating the results onto the other frames. In video retexturing, we have introduced the mesh based optimization for object tracking to avoid texture drifting, and the graph cut algorithm to effectively deal with visibility shift between frames. The graph cut algorithm is applied on a trimap along the boundary of the object to extract the textured part inside the trimap. The proposed approach is developed in image/video retexturing at nearly interactive rate, and our experimental results have showed the satisfactory performance of our approach. Copyright © 2005 John Wiley & Sons, Ltd.

Abstract In this paper, we propose a novel practical method for blob‐based liquid 3D morphing. Firstly, blobby objects are employed to approximate a given polygonal surface. The primitives in the medial axis sphere‐tree of a polygonal model are utilized as initial blobs—this greatly improves the robustness and efficiency of the blob‐based approximation. Secondly, we establish the blob correspondences between two models by sphere cellular matching and hierarchical matching. Finally, we interpolate the parameters of the implicit representation to get the intermediate shapes. Experiments show our method can produce visually pleasing liquid morphing effects. Copyright © 2005 John Wiley & Sons, Ltd.

In this paper, we propose a new wavelet scheme for loop subdivision surfaces. The main idea enabling our wavelet construction is to extend the subdivision rules to be invertible, thus executing each inverse subdivision step in the reverse order makes up the wavelet decomposition rule. As opposed to other existing wavelet schemes for loop surfaces, which require solving a global sparse linear system in the wavelet analysis process, our wavelet scheme provides efficient (linear time and fully in-place) computations for both forward and backward wavelet transforms. This characteristic makes our wavelet scheme extremely suitable for applications in which the speed for wavelet decomposition is critical. We also describe our strategies for optimizing free parameters in the extended subdivision steps, which are important to the performance of the final wavelet transform. Our method has been proven to be effective, as demonstrated by a number of examples.

Our sense of touch is spatially focused and has a far lower bandwidth in comparison with visual sense that has the largest bandwidth. While most haptic studies utilize point interactions, resulting in a conflict between this low information bandwidth and relative complexity of virtual scene. In this paper, we investigate a novel multi-resolution force evaluation scheme of hybrid virtual models in a unified haptic interaction framework. Force contact model of tool-object interaction based on Hertz's contact theory is integrated in our work. Physical properties with different object materials expressed by Poisson's Ratio and Young's Modulus are involved to investigate most realistic perception during multi-resolution haptic interaction. The hierarchical impostor representation of surface and volumetric models are constructed, and the optimal performance based on cost and benefit evaluation at run time is employed to meet both the visual and hapic perceptual qualities. During this multi-resolution haptic interaction, our scheme adaptively determines the rendering mode using graphics and haptics impostors represented in the virtual environments. Our experimental results have demonstrated the satisfactory performance of proposed multi-resolution haptic interaction framework applicable for hybrid virtual models.

Abstract Realistic image synthesis of highly complex, natural scenes has been a challenging topic in computer graphics over the years. Generation of snowy scenery could be even more difficult due to the difficulty of using common modelling primitives such as polygons and curved surfaces to express the snow‐like shapes. In this paper, we propose a hybrid multi‐mapping method to tackle the snowy scenery problem, in which a displacement map is utilized to model snowy blocks on near objects based on proximate polygons, and a volumetric texture map is combined to handle distant objects such as bushes and trees. Our experimental results showed that the hybrid method is viable in producing realistic snowy scenery with comprehensive complex environments, and in a favourable requirement of storage usage and rendering computation. Copyright © 2003 John Wiley & Sons, Ltd.

An effective deformable model based on a successive force propagation process is proposed. It avoids the laborious stiffness matrix formulation and is scalable simply by controlling the penetration depth. Mechanical tests are performed to evaluate its feasibility for modeling real tissues. An interactive system is developed using a commercial haptic device.

Virtual sculpture is a modeling technique for computer graphics based on the notion of sculpting a solid material with tools. Currently, most interactive sculpture is mainly focused on vision-based sensory channel. With visual feedback alone virtual sculpture cannot simulate the realistic sculpting operations in the physical world. The sense of touch, in combination with our kinesthetic sense, is capable of adding a new modality to virtual sculpture, especially in presenting complex geometry & material properties. In this paper, we propose a virtual haptic sculpting (VHS) system in the volume space, which supports real-time melting, burning, stamping, painting, constructing and peeling interactions. Based on the constructive volume methodology, we have developed sculpting tools as volumes, with each properties and size, distribution for elements, and rules of the interaction between the volumetric data and the tools. The sculpting tools are controlled directly by the 6-DOF haptic input to simulate realistic sculpting operations, in applying the computed model and tool dynamics while interacting with the volume. Both synthetic volumetric data and medical scan volumes are experimented using the 6-DOF PHANToM Desktop haptic interface.

Real-time rendering for large-scale, complex dynamic virtual scenes is a challenging problem in computer graphics. In this paper, we propose a hybrid rendering algorithm of dynamic virtual environments that seamlessly fuses the point-based scheme and polygon-based scheme. In our algorithm, the scene is organized into a BSP-tree. Objects in the leaf-nodes of the BSP tree are further subdivided into a quad-tree hierarchy, which contains both the sample points and polygon rendering information at each level. The accelerated rendering algorithm integrates the hierarchical occlusion map technique, image caching technique and BSP technique to fast render complex dynamic scenes. During navigation, our system adaptively determines the rendering mode and the level of details of objects, and achieves smooth transition between the two rendering modes by effectively controlling the rendering precision. The dynamic objects can be processed in the system uniformly. Our experimental results have demonstrated the satisfactory performance of the proposed hybrid-rendering scheme for the dynamic virtual environments.

A novel progressive geometry compression scheme is presented in this paper. In this scheme, a mesh is represented as a base mesh followed by some groups of vertex split operations using an improved simplification method in which each level of the mesh can be refined into the next level by carrying out a group of vertex split operations in any order. Consequently, the progressive mesh (PM) representation can be effectively encoded by permuting the vertex split operations in each group. Meanwhile, a geometry predictor using the Laplacian operator is designed to predict each new vertex position using its neighbours. The correction is quantized and encoded using a Huffman coding scheme. Experimental results show that our algorithm obtains higher compression ratios than previous work. It is very suitable for the progressive transmission of geometric models over the Internet.

With the development of volume visualization methods, we can easily extract meaningful information from volumetric data using interactive graphics and imaging. Haptic interaction of volumetric data adds a new modality to volume visualization that has an advantage in presenting complex attributes of local region. However, the benefits of haptic rendering of volumetric data have only been recognized recently. Most traditional haptic rendering methods are developed to compute realistic interaction force with geometric primitives. Direct volume haptic rendering allows haptic palpation of volumetric data, but lacks of the ability of simulating the contact sensation of stiff embedded implicit surface.In this paper, we propose a direct haptic rendering method for isosurface in volumetric data using a point-based haptic feedback device, without the extraction of the isosurface to geometric representations such as polygons. Our algorithm extends the intermediate representation approach that had been introduced for dealing with complex virtual environment, to haptically render volumetric data. The algorithm uses a virtual plane as an intermediate representation of the implicit isosurface, and computes the point interaction force applied to the haptic interface based on this virtual plane. Using this approach, we are able to gain higher haptic servo rate for volumetric data. It makes maintenance of the stability of the simulation easier, and applicable to noisy data without preprocessing. We have developed our algorithm and tested with synthetic data and medical data, using the PHANToMTM haptic interface.

Due to visibility change and surface enlargement in producing a novel view from a new viewpoint, 3D re-projection from one reference image in IBMR inevitably produces holes in the destination image. Even worse, exposure errors occur when a background region occluded is visible in a desired image because of the absence of some background elements in the reference image. The general solution to this kind of problems is to use multiple images from different viewpoints as input source. By doing so however, the rendering cost would increase with the number of reference images and the composition algorithm has to rely on the z-buffer processing.In fact, plenty of redundant information exists among different reference images. Seeking for a nice way to extract the information needed in the novel view from the reference images is the key issue in solving the problem. In this paper, we propose a new method of image synthesis from multiple reference images. The method combines forward warping and backward warping to fulfil the image composition task for a novel viewpoint. The primary inspiration behind the development of our image synthesis method comes from a fact that the polygon edge geometry may indicate where an exposure and, possible an exposure error occur in the destination image if object silhouettes are prior known. The feature that intersection between scanline and polygons must be in pairs is employed to distinguish holes caused by enlargement of surfaces from those by visibility change. Different heuristic methods are used to choose one image as a primary reference image which shares the most resemblance with the destination image, and other reference images for filling different kinds of holes. Depth continuity along scanline and the depth information already present in the destination image are employed to accelerate the searching process of the corresponding pixels for filling holes.

The availability of high-performance 3D workstations has increased the range of application for interactive real-time animation. In these applications the user can directly interact with the objects in the animation and direct the evolution of their motion, rather than simply watching a pre-computed animation sequence. Interactive real-time animation has fast-growing applications in virtual reality, scientific visualization, medical training and distant learning. Traditional approaches to computer animation have been based on the animator having complete control over all aspects of the motion. In interactive animation the user can interact with any of the objects, which changes the current motion path or behaviour in real time. The objects in the animation must be capable of reacting to the user's actions and not simply replay a canned motion sequence. This paper presents a framework for interactive animation that allows the animator to specify the reactions of objects to events generated by other objects and the user. This framework is based on the concept of relations that describe how an object reacts to the influence of a dynamic environment. Each relation specifies one motion primitive triggered by either its enabling condition or the state of the environment. A collection of the relations is structured through several hierarchical layers to produce responsive behaviours and their variations. This framework is illustrated by several room-based dancing examples that are modelled by relations. Copyright © 2000 John Wiley & Sons, Ltd.

A distributed virtual reality system based on a hybrid rendering scheme is presented in this paper. Unlike the traditional image based rending algorithm, our approach divides the local scene within the frustum of the current viewpoint into two layers, i.e., the front layer and the back layer. Objects in the front layer are rendered in real time with a geometry based hardware z-buffer algorithm. To generate the image of back layer, we first perform a nonlinear transformation to all visible points of a keyframe adjacent to the current viewpoint to reveal their coplanar characteristics. A hierarchical quadrangle approximation of the local scene is then set up, the intermediate image of the back layer at the current viewpoint is then obtained by re-projecting these quadrangles onto the current image plane, gaps that might appear on the intermediate image am filled by bi-directional backward warping. As our approach consists of two independent phases, it can be easily implemented in parallel or distribution mode. Experimental results show that our approach has a satisfactory performance.

We propose a task planner that incorporates with a virtual reality interface for 3D immersive interaction of CAD models and high-level task planning of mating processes. The planner is constructed with the following objectives: 3D immersive VR interface, assembly analysis, feature/constraint update, and assembly path planning.

Object interactions are ubiquitous in interactive computer graphics, 3D object motion simulations, virtual reality and robotics applications. Most collision detection algorithms are based on geometrical object-space interference tests. Some algorithms have employed an image-space approach to the collision detection problem. In this paper we demonstrate an image-space collision detection process that allows substantial computational savings during the image-space interference test. This approach makes efficient use of the graphics rendering hardware for real-time complex object interactions. Copyright © 1999 John Wiley & Sons, Ltd.

Object interactions are ubiquitous in interactive computer graphics, 3D object motion simulations, virtual reality and robotics applications. Most collision detection algorithms are based on geometrical object space interference tests. Some algorithms have employed an image space approach to the collision detection problem. We demonstrate an image space collision detection process that allows substantial computational savings during the image space interference test. This approach makes efficient use of the graphics rendering hardware for real time complex object interactions.

In a virtual environment as well as in practice, collisions between complex surfaces and/or articulated bodies may occur simultaneously at multiple points of interference. Previous solutions to the collision response problem have been formulated based on pairwise rigid body dynamic interactions. These give only a local view of the dynamic response of each body. We present a new solution to the global interactions and dynamic response between multiple structures in a three-dimensional environment. This is based on a graph-theoretical framework that allows a systematic formulation of the equations of motion, gives a global view of all the interactions in a system, and suggests other approaches to the computation of dynamic response of a system.

The HolorGraphic modelling system (HGMS) is a general computational framework for the algebraic manipulation of complex mathematical models of both physical and non-physical systems. The lack of constraints on the operational algebra in HGMS provides both flexibility and expressive power for manipulating variables and functions. The underlying influence of the HolorGraphic framework is now being independently observed in seemingly unrelated fields of studies, from theoretical physics to the construction and analysis of curves and surfaces in computer aided geometric design. As it takes its own form above and beyond tensor theory, the HolorGraphic framework provides both a consistent mathematical basis of formulation as well as a visual understanding of the object of study. In this article, we illustrate the use of HGMS in the symbolic computation procedures using general holors and show the basic operations on these structures.

Although object manipulation is an important issue in VR, most applications fail to provide the natural interface of a dextrous hand that interacts with the objects in a physics-based realism. The paper presents a hybrid control approach that uses both kinematics and dynamics methods at different stages of picking to generate physically-accurate hand interactions in real time. This approach can simulate the natural picking interactions in various hand applications, including 3D mechanical design, assembly, surgical simulation, and telerobotics.

To provide fast display and manipulation of the 3D reconstruction of volumetric data, our system uses surface rendering which reduces 3D data sets to geometric primitives that have been implemented in the SGIs hardware, so a better performance of rendering speed is obtained. Our system supports both basic and advanced navigation features of medical data sets, including translating, rotating, zooming in/out, advanced cutting functions (transparent alignment, multi-plane cut, cut in any orientation and depth), classification of tissues, and centralization function. The 3D interface of our system provides users realistic view and feature-based manipulation of the medical data sets to extract useful information for medical training and education.

Article Free Access Share on Hardware-assisted virtual collisions Authors: George Baciu Computer Science Department, The Hong Kong University of Science and Technology, Clear Water Bay, Kowloon, Hong Kong Computer Science Department, The Hong Kong University of Science and Technology, Clear Water Bay, Kowloon, Hong KongView Profile , Wingo Sai-Keung Wong View Profile , Hanqiu Sun View Profile Authors Info & Claims VRST '98: Proceedings of the ACM symposium on Virtual reality software and technologyNovember 1998 Pages 145–151https://doi.org/10.1145/293701.293720Published:02 November 1998Publication History 3citation300DownloadsMetricsTotal Citations3Total Downloads300Last 12 Months25Last 6 weeks5 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteeReaderPDF

Visualizing architectural designs with t,he aid of Virtual Reality (VR) technology is becoming a common task among architects, design engineers, contractors and customers due to a more realistic resemblance to the real look of the final constructions.In order to preview virtual buildings, one needs to reconstruct a VR model from the two-dimensional architectural drawings.In the conventional approach, a large amount of manual tasks are necessary while using some common 3D authoring tools avaliable in the market.This gives us the incentive to examine the process of modelling and reconstruction of 2D designs for use in practical applications.In this paper, we identify the main tasks in the conventional reconstruction process and give a semi-aut,omated solution that aids in the process of ext,rusion and modelling of large building complexes.

There is no doubt that 3D interfaces edged 2D ones in 3D modeling applications. 2D input devices cannot provide 3D information directly and are less efficient. Direct manipulation, which is highly desirable in 3D modeling, is also impossible. This paper describes the design and implementation of a virtual hand interface for a 3D modeler, which can be used to creating VRML worlds. The virtual hand interface allows direct 3D interaction of objects. Users can grab, move or resize the object easily with their hands in a more natural and intuitive way. A fuzzy logic posture recognition system is incorporated with the modeler to accept posture commands. Fuzzy logic is applied to cope with imprecise data stored in the sample posutre database and the ones to be recognized in real-time. A posture database editor is developed for managing the sample posture database, whose functions include posture insertion, deletion, testing and viewing the records in the posture database. The 3D virtual worl...

Over the past few decades many advances have been made in computer animation, which has largely replaced traditional hand-drawn animation. However, the traditional path model used in animation has not changed. This model as based on predefined paths does not effectively address the needs of scene animation, which demands unpredictable environmental influences, multiple motion interactions, and personal behaviors. Some of the suggested approaches to scene animation are sensor-effector model, rule-based model, and predefined environment model, but these approaches don't completely solve the problem. This paper presents a relation-based model for dealing with the important issues of scene animation. Two fundamental principles of our model are: the specification of atomic units of motion (relations), and the ability to combine these atomic units to produce sophisticated behaviors. Each relation models a simple responsive behavior between two objects, and has its own sensor, response, duration, strength, and state. The dynamic states of relations can be interactively structured into hierarchical layers that produce complex scene behaviors. During the animation, structured relations are dynamically triggered by either its sensing condition(s) or the state of the environment. The variable state-control hierarchies facilitate on-line scene behavior animations. Several dancing examples for the use of our model are illustrated.

A scene animation involving a limited environmental boundary, obstacles, other static and dynamic objects, and scene events is more interdependent, variable, and personal than a single object's animation. Traditional path specification does not directly address the special issues in this problem domain, and its use can be very costly and time consuming. In addition to this approach, three other models are proposed for scene animations. These are the sensor-effector model, the rule-based model, and the predefined environment model. These models, however, are either incomplete or limited to certain behaviours or simple environments. This paper analyses the complexities of modular specification and processing in a general scene context, based on the concept of relations, its modelling framework, and state control hierarchies. These estimated complexities outline the general specification and system processing interfaces that can be used effectively in various scene applications.