Recent work in 3D deep learning has demonstrated that unsigned distance functions (UDFs) are a useful representation for 3D reconstruction and shape generation because they can represent surfaces with arbitrary topology. However, extracting meshes that preserve the intended topology, especially in the presence of non-manifold structures, remains challenging. We present DCx , an extension of the standard Dual Contouring (DC) method which was originally proposed for isosurface extraction from signed distance functions (SDFs). Standard DC operates on individual voxels and inserts one vertex per active cube, where activation is determined by detecting sign changes. To address the lack of sign information in UDFs, DCx adopts an optimization-based strategy for determining active cubes. It operates on each 2 × 2 × 2 voxel block, referred to as an expanded cube, and introduces a voxel-to-mesh lookup table that stores connectivity patterns based on local voxel configurations. This enables efficient triangle extraction using predefined templates. These changes improve upon DC by avoiding failure cases caused by unreliable active-cube detection in UDFs and by correcting mesh connections in non-manifold regions. As a result, DCx supports the extraction of both manifold and non-manifold surfaces from neural UDFs. DCx is conceptually simple and easy to implement. Experimental results show that DCx produces meshes with higher accuracy in a more robust way than existing methods, particularly on shapes with complex geometry or non-manifold structures. The source code is available at http://github.com/jjjkkyz/DCx.

Speech-driven 3D facial animation has notable applications in the VR domain, including virtual anchors and digital avatars, etc. However, producing facial animations that convey complex emotional expressions remains a substantial challenge. Existing methods struggle to simultaneously achieve accurate lip synchronization, natural facial expressions, and realistic emotional representation. Significantly, the impact of head pose on boosting facial emotional expressiveness has not been thoroughly investigated. To address these issues, we propose EmoPoseFace, a novel Diffusion-based network to generate speech-driven 3D emotional facial animations with synchronized head poses. Our method employs a dual-branch conditional generation architecture to separately model facial expressions and head poses, integrating emotion and head-pose conditions for coherent facial expression-pose control. In addition, we design the Global-local Facial Fine-grained Editing Module (GL-FFE), which achieves emotional enhancement of facial expressions and fine-grained facial modification, while maintains the naturalness and authenticity of facial movements. Extensive experiments demonstrate that our approach outperforms existing methods in lip-sync accuracy and emotional detail preservation. The introduction of head pose control and GL-FFE significantly expands the expressiveness of emotional virtual facial animation, and the fine-grained editing is widely approved in perceptual user studies.

While Signed Distance Fields (SDF) are well-established for modeling watertight surfaces, Unsigned Distance Fields (UDF) broaden the scope to include open surfaces and models with complex inner structures. Despite their flexibility, UDFs encounter significant challenges in high-fidelity 3D reconstruction, such as non-differentiability at the zero level set, difficulty in achieving the exact zero value, numerous local minima, vanishing gradients, and oscillating gradient directions near the zero level set. To address these challenges, we propose Details Enhanced UDF (DEUDF) learning that integrates normal alignment and the SIREN network for capturing fine geometric details, adaptively weighted Eikonal constraints to address vanishing gradients near the target surface, unconditioned MLP-based UDF representation to relax non-negativity constraints, and DCUDF for extracting the local minimal average distance surface. These strategies collectively stabilize the learning process from unoriented point clouds and enhance the accuracy of UDFs. Our computational results demonstrate that DEUDF outperforms existing UDF learning methods in both accuracy and the quality of reconstructed surfaces.

Efficiently modeling large 2D contexts is essential for various fields including Giga-Pixel Whole Slide Imaging (WSI) and remote sensing. Transformer-based models offer high parallelism but face challenges due to their quadratic complexity for handling long sequences. Recently, Mamba introduced a selective State Space Model (SSM) with linear complexity and high parallelism, enabling effective and efficient modeling of wide context in 1D sequences. However, extending Mamba to vision tasks, which inherently involve 2D structures, results in spatial discrepancies due to the limitations of 1D sequence processing. On the other hand, current 2D SSMs inherently model 2D structures but they suffer from prohibitively slow computation due to the lack of efficient parallel algorithms. In this work, we propose 2DMamba, a novel 2D selective SSM framework that incorporates the 2D spatial structure of images into Mamba, with a highly optimized hardware-aware operator, adopting both spatial continuity and computational efficiency. We validate the versatility of our approach on both WSIs and natural images. Extensive experiments on 10 public datasets for WSI classification and survival analysis show that 2DMamba improves up to 2.48% in AUC, 3.11% in F1 score, 2.47% in accuracy and 5.52% in C-index. Additionally, integrating our method with VMamba for natural imaging yields 0.5 to 0.7 improvements in mIoU on the ADE20k semantic segmentation dataset, and 0.2% accuracy improvement on ImageNet-1K classification dataset. Our code is available at https://github.com/AtlasAnalyticsLab/2DMamba.

Unsigned distance fields (UDFs) are widely used in 3D deep learning due to their ability to represent shapes with arbitrary topology. While prior work has largely focused on learning UDFs from point clouds or multi-view images, extracting meshes from UDFs remains challenging, as the learned fields rarely attain exact zero distances. A common workaround is to reconstruct signed distance fields (SDFs) locally from UDFs to enable surface extraction via Marching Cubes. However, this often introduces topological artifacts such as holes or spurious components. Moreover, local SDFs are inherently incapable of representing non-manifold geometry, leading to complete failure in such cases. To address this gap, we propose MIND ($\mathrm{\underline{M}aterial}$ $\mathrm{\underline{I}nterface}$ $\mathrm{from}$ $\mathrm{\underline{N}on}$-$\mathrm{manifold}$ $\mathrm{\underline{D}istance}$ $\mathrm{fields}$), a novel algorithm for generating material interfaces directly from UDFs, enabling non-manifold mesh extraction from a global perspective. The core of our method lies in deriving a meaningful spatial partitioning from the UDF, where the target surface emerges as the interface between distinct regions. We begin by computing a two-signed local field to distinguish the two sides of manifold patches, and then extend this to a multi-labeled global field capable of separating all sides of a non-manifold structure. By combining this multi-labeled field with the input UDF, we construct material interfaces that support non-manifold mesh extraction via a multi-labeled Marching Cubes algorithm. Extensive experiments on UDFs generated from diverse data sources, including point cloud reconstruction, multi-view reconstruction, and medial axis transforms, demonstrate that our approach robustly handles complex non-manifold surfaces and significantly outperforms existing methods. The source code is available at https://github.com/jjjkkyz/MIND.

Most of the existing panoramic video navigation approaches are saliency-driven, whereby off-the-shelf saliency detection tools are directly employed to aid the navigation approaches in localizing video content that should be incorporated into the navigation path. In view of the dilemma faced by our research community, we rethink if the "saliency clues" are really appropriate to serve the panoramic video navigation task. According to our in-depth investigation, we argue that using "saliency clues" cannot generate a satisfying navigation path, failing to well represent the given panoramic video, and the views in the navigation path are also low aesthetics. In this paper, we present a brand-new navigation paradigm. Although our model is still trained on eye-fixations, our methodology can additionally enable the trained model to perceive the "meaningful" degree of the given panoramic video content. Outwardly, the proposed new approach is saliency-free, but inwardly, it is developed from saliency but biasing more to be "meaningful-driven"; thus, it can generate a navigation path with more appropriate content coverage. Besides, this paper is the first attempt to devise an unsupervised learning scheme to ensure all localized meaningful views in the navigation path have high aesthetics. Thus, the navigation path generated by our approach can also bring users an enjoyable watching experience. As a new topic in its infancy, we have devised a series of quantitative evaluation schemes, including objective verifications and subjective user studies. All these innovative attempts would have great potential to inspire and promote this research field in the near future.

To date, the widely adopted way to perform fixation collection in panoptic video is based on a head-mounted display (HMD), where users' fixations are collected while wearing a HMD to explore the given panoptic scene freely. However, this widely-used data collection method is insufficient for training deep models to accurately predict which regions in a given panoptic are most important when it contains intermittent salient events. The main reason is that there always exist "blind zooms" when using HMD to collect fixations since the users cannot keep spinning their heads to explore the entire panoptic scene all the time. Consequently, the collected fixations tend to be trapped in some local views, leaving the remaining areas to be the "blind zooms". Therefore, fixation data collected using HMD-based methods that accumulate local views cannot accurately represent the overall global importance - the main purpose of fixations - of complex panoptic scenes. To conquer, this paper introduces the auxiliary window with a dynamic blurring (WinDB) fixation collection approach for panoptic video, which doesn't need HMD and is able to well reflect the regional-wise importance degree. Using our WinDB approach, we have released a new PanopticVideo-300 dataset, containing 300 panoptic clips covering over 225 categories. Specifically, since using WinDB to collect fixations is blind zoom free, there exists frequent and intensive "fixation shifting" - a very special phenomenon that has long been overlooked by the previous research - in our new set. Thus, we present an effective fixation shifting network (FishNet) to conquer it. All these new fixation collection tool, dataset, and network could be very potential to open a new age for fixation-related research and applications in 360o environments.

The rapid, accurate, and robust computation of virtual human figures' "in-between" pose transitions from available and sometimes sparse inputs is of fundamental significance to 3D interactive graphics and computer animation. Various methods have been proposed to produce natural lifelike transitions of human pose automatically in recent decades. Nevertheless, conventional pure model-driven methods require heuristic knowledge (e.g., least motion guided by physics laws) and ad-hoc clues (e.g., splines with non-uniform time warp) that are difficult to obtain, learn, and infer. With the fast emergence of large-scale datasets readily available to animators in the most recent years, deep models afford a powerful alternative to tackle the aforementioned challenges. However, pure data-driven methods still suffer from the remaining challenges such as unseen data in practice and less generative power in model/domain/data transfer, and the measurement of the generative power has always been omitted in these works. In essence, data-driven methods solely rely on the qualities and quantities of training datasets. In this paper, we propose a hybrid approach built upon the seamless integration of data-driven and model-driven methods, called Dynamic Motion Transition (DMT), with the following salient modeling advantages: (1) The data augmentation capability based on the limited human locomotion data capture and the concept of force-derived directly from physical laws; (2) Force learning by which skeleton joints are driven to move, and the Conditional Temporal Transformer (CTT) being trained to learn the force change in the local range, both at the fine level; and (3) At the coarse level, the effective and flexible creation of the subsequent step motion using Dynamic Movement Primitives (DMP) until the target is reached. Our extensive experiments have confirmed that our model can outperform the state-of-the-art methods under the newly devised metric by virtue of the least action loss function. In addition, our novel method and system are of immediate benefit to many other animation tasks such as motion synthesis and control, and motion tracking and prediction in this bigdata graphics era.

Single-view 3D object reconstruction (SVOR) aims to recover the 3D shape of an object from a single 2D image. Despite advances in deep learning (DL), challenges such as incomplete image information, scarce 3D data annotation, and highly variable object shapes still limit the performance of SVOR. Meanwhile, with the rapid development of novel view synthesis (NVS) techniques, the SVOR field has received significant advancements. However, existing reviews have not comprehensively covered the rapid developments in NVS-based approaches. This article aims to fill this gap by highlighting the latest progress in SVOR, particularly advancements related to NVS-based methods. Additionally, we observed discrepancies between existing quality evaluation metrics in SVOR and human visual perception. This is because some critical object parts are essential to consider during the evaluation. For example, when reconstructing airplanes, critical parts like the empennage and wings are often overlooked in evaluation metrics due to their smaller size compared to the fuselage. Consequently, poor reconstruction of these parts may not significantly affect overall evaluation scores. To address this issue, we propose a more comprehensive evaluation method that reflects human visual perception accurately. To achieve this, we introduce a weighted evaluation method that considers part saliency and proposes a novel technique for automatically perceiving reconstruction discrepancies. This study effectively enhances the accuracy and consistency of evaluations through these approaches, offering new insights and methodologies, filling a void in the existing literature, and providing valuable contributions to both research and practical applications in SVOR.

Real-time subsurface scattering techniques are widely used in translucent material rendering. Among advanced methods that rely on the bidirectional scattering-surface reflectance distribution function (BSSRDF), screen space algorithms exhibit limited translucency, while existing large-distance methods are inefficient and yield poor illumination details. To address these limitations for better large-distance scattering, we develop a novel algorithm by extending the photon beam diffusion (PBD) model within the light view and screen space. Unlike surface irradiance in prior methods, we incorporate the refracted beam in the medium into real-time scattering estimation, presenting a new consideration for photon beam utilization. Concretely, we store all photon beam samples in light view textures and utilize an adaptive sampling pattern for beam sample selection in large filtering kernel sizes. This can reduce the sample count based on surface attributes. In screen space, virtual sources are derived from samples to estimate PBD contributions, with an approximation that preserves boundary conditions. To avoid possible overestimation, we implement correction factors that scale contributions, effectively aligning our results with path-tracing references. Through these reformulations, our efficient PBD generates results closest to references among existing methods. The experiments accurately represent better front-face illumination details and backlit translucency effects, while significantly accelerating performance compared to previous large-distance methods.

In this study, we devise a framework for volumetrically reconstructing fluid from observable, measurable free surface motion. Our innovative method amalgamates the benefits of deep learning and conventional simulation to preserve the guiding motion and temporal coherence of the reproduced fluid. We infer surface velocities by encoding and decoding spatiotemporal features of surface sequences, and a 3D CNN is used to generate the volumetric velocity field, which is then combined with 3D labels of obstacles and boundaries. Concurrently, we employ a network to estimate the fluid's physical properties. To progressively evolve the flow field over time, we input the reconstructed velocity field and estimated parameters into the physical simulator as the initial state. Our approach yields promising results for both synthetic fluid generated by different fluid solvers and captured real fluid. The developed framework naturally lends itself to a variety of graphics applications, such as 1) effective reproductions of fluid behaviors visually congruent with the observed surface motion, and 2) physics-guided re-editing of fluid scenes. Extensive experiments affirm that our novel method surpasses state-of-the-art approaches for 3D fluid inverse modeling and animation in graphics.

The accurate and diversified generation of motion sequences for virtual characters poses both an enticing and challenging task within the domain of 3D animation and game content production. To achieve a natural and realistic full-body motion, the movements of virtual characters must adhere to a set of constraints, promoting reliable and seamless pose-changing. This study presents a two-stage model specifically designed to learn Inverse Kinematics (IK) constraints from the representative quadruped character poses. In the first stage, we employ frequency analysis to decompose motion poses into the base-level and style-level components. The base-level content encapsulates the global correlations in the dataset, while the style-level variation centers on distinguishing the local attributes in similar data elements. In order to construct data correlations among poses, we embed the decomposed pose feature into a latent space in the second stage. The kernel matrix of the embedding, which is refined from the original joint angles to the decomposed representation and the IK constraints, creates a more compact distribution of the pose similarity and also guarantees a plausible sampling result with certain IK constraints. Moreover, new motions from the edited IK constraints can also be generated by proposing a searching strategy to adapt to our latent embedding. Experimental results reveal that our method is competitive with the state-of-the-art synthetic approaches in terms of accuracy, highlighting our considerable potential for high efficiency in the animation production.

Traditional dental plaque detection relies on medical staining reagents and professional intervention. Deep learning-based automatic staining-free dental plaque segmentation provides an alternative for patients to perform plaque detection at home without staining reagents. However, existing methods still struggle with low-contrast visual features between unstained plaque and healthy teeth. To address this, we propose a Frequency-Guided Network (FGN) for low-contrast staining-free dental plaque segmentation. We observe that dental plaque tends to concentrate specifically near the junction between the teeth and the gingiva. This junction demonstrates abrupt changes in pixel values, indicating high-frequency regions in the image. In other words, dental plaque tends to appear near the high-frequency regions of oral endoscope images. Exploiting this characteristic, we employ a frequency-guided decoupling module to separate the image into high-frequency and low-frequency regions automatically and expand the high-frequency region to encompass nearby potential dental plaque. Then we supervise two regions individually to specifically focus on the expended high-frequency region for localizing nearby dental plaque. Additionally, we propose a high-to-low frequency multiple tasks framework. In the first phase, the network segments the teeth region, and then we input the teeth mask into the second phase. In the second stage, the teeth mask allows us to have a higher frequency at the junction between the teeth and gums, thereby enhancing the effectiveness of frequency-guided decoupling. Furthermore, FGN integrates a frequency-driven refinement module to enhance the guidance quality of the teeth mask for the second phase. Extensive evaluations of the oral endoscope dataset demonstrate that our method outperforms existing high-performance segmentation methods. User studies also confirm that our approach achieves superior results to experienced dentists. https://frequency-guided-network.github.io/

Computer animation's quest to bridge content and style has historically been a challenging venture, with previous efforts often leaning toward one at the expense of the other. This paper tackles the inherent challenge of content-style duality, ensuring a harmonious fusion where the core narrative of the content is both preserved and elevated through stylistic enhancements. We propose a novel Multi-condition Motion Latent Diffusion Model (MCM-LDM) for Arbitrary Motion Style Transfer (AMST). Our MCM-LDM significantly emphasizes preserving trajectories, recognizing their fundamental role in defining the essence and fluidity of motion content. Our MCM-LDM's cornerstone lies in its ability first to disentangle and then intricately weave together motion's tripartite components: motion trajectory, motion content, and motion style. The critical insight of MCM-LDM is to embed multiple conditions with distinct priorities. The content channel serves as the primary flow, guiding the overall structure and movement, while the trajectory and style channels act as auxiliary components and synchronize with the primary one dynamically. This mechanism ensures that multi-conditions can seamlessly integrate into the main flow, enhancing the overall animation without overshadowing the core content. Empirical evaluations underscore the model's proficiency in achieving fluid and authentic motion style transfers, setting a new benchmark in the realm of computer animation. The source code and model are available at https://github.com/XingliangJin/MCM-LDM.git.

To date, the quest to rapidly and effectively produce human-object interaction (HOI) animations directly from textual descriptions stands at the forefront of computer vision research. The underlying challenge demands both a discriminating interpretation of language and a comprehen-sive physics-centric model supporting real-world dynamics. To ameliorate, this paper advocates HOIAnimator, a novel and interactive diffusion model with perception ability and also ingeniously crafted to revolutionize the animation of complex interactions from linguistic narratives. The effectiveness of our model is anchored in two ground-breaking innovations: (1) Our Perceptive Diffusion Models (PDM) brings together two types of models: one focused on hu-man movements and the other on objects. This combination allows for animations where humans and objects move in concert with each other, making the overall motion more realistic. Additionally, we propose a Perceptive Message Passing (PMP) mechanism to enhance the communication bridging the two models, ensuring that the animations are smooth and unified; (2) We devise an Interaction Contact Field (ICF), a sophisticated model that implicitly captures the essence of HOls. Beyond mere predictive contact points, the ICF assesses the proximity of human and object to their respective environment, informed by a probabilistic distribution of interactions learned throughout the denoising phase. Our comprehensive evaluation showcases HOlani-mator's superior ability to produce dynamic, context-aware animations that surpass existing benchmarks in text-driven animation synthesis.

Numerical solvers of Partial Differential Equations (PDEs) are of fundamental significance to science and engineering. To date, the historical reliance on legacy techniques has circumscribed possible integration of big data knowledge and exhibits sub-optimal efficiency for certain PDE formulations, while data-driven neural methods typically lack mathematical guarantee of convergence and correctness. This paper articulates a mathematically rigorous neural solver for linear PDEs. The proposed UGrid solver, built upon the principled integration of U-Net and MultiGrid, manifests a mathematically rigorous proof of both convergence and correctness, and showcases high numerical accuracy, as well as strong generalization power to various input geometry/values and multiple PDE formulations. In addition, we devise a new residual loss metric, which enables unsupervised training and affords more stability and a larger solution space over the legacy losses.

Traditional 3D shape reconstruction techniques from multi-view images, such as structure from motion and multi-view stereo, face challenges in reconstructing transparent objects. Recent advances in neural radiance fields and its variants primarily address opaque or transparent objects, encountering difficulties to reconstruct both transparent and opaque objects simultaneously. This paper introduces $\alpha$-NeuS$\textemdash$an extension of NeuS$\textemdash$that proves NeuS is unbiased for materials from fully transparent to fully opaque. We find that transparent and opaque surfaces align with the non-negative local minima and the zero iso-surface, respectively, in the learned distance field of NeuS. Traditional iso-surfacing extraction algorithms, such as marching cubes, which rely on fixed iso-values, are ill-suited for such data. We develop a method to extract the transparent and opaque surface simultaneously based on DCUDF. To validate our approach, we construct a benchmark that includes both real-world and synthetic scenes, demonstrating its practical utility and effectiveness. Our data and code are publicly available at https://github.com/728388808/alpha-NeuS.

Abstract Neural Radiance Fields (NeRFs) have achieved significant recognition for their proficiency in scene reconstruction and rendering by utilizing neural networks to depict intricate volumetric environments. Despite considerable research dedicated to reconstructing physical scenes, rare works succeed in challenging scenarios involving dynamic, multi‐material objects. To alleviate, we introduce CoupNeRF, an efficient neural network architecture that is aware of multiple material properties. This architecture combines physically grounded continuum mechanics with NeRF, facilitating the identification of motion systems across a wide range of physical coupling scenarios. We first reconstruct specific‐material of objects within 3D physical fields to learn material parameters. Then, we develop a method to model the neighbouring particles, enhancing the learning process specifically in regions where material transitions occur. The effectiveness of CoupNeRF is demonstrated through extensive experiments, showcasing its proficiency in accurately coupling and identifying the behavior of complex physical scenes that span multiple physics domains.

Abstract Sub‐surface scattering is always an important feature in translucent material rendering. When light travels through optically thick media, its transport within the medium can be approximated using diffusion theory, and is appropriately described by the bidirectional scattering‐surface reflectance distribution function (BSSRDF). BSSRDF methods rely on assumptions about object geometry and light distribution in the medium, which limits their applicability to general participating media problems. However, despite the high computational cost of path tracing, BSSRDF methods are often favoured due to their suitability for real‐time applications. We review these methods and discuss the most recent breakthroughs in this field. We begin by summarizing various BSSRDF models and then implement most of them in a 2D searchlight problem to demonstrate their differences. We focus on acceleration methods using BSSRDF, which we categorize into two primary groups: pre‐computation and texture methods. Then we go through some related topics, including applications and advanced areas where BSSRDF is used, as well as problems that are sometimes important yet are ignored in sub‐surface scattering estimation. In the end of this survey, we point out remaining constraints and challenges, which may motivate future work to facilitate sub‐surface scattering.

Learning and inferring underlying motion patterns of captured 2D scenes and then re-creating dynamic evolution consistent with the real-world natural phenomena have high appeal for graphics and animation. To bridge the technical gap between virtual and real environments, we focus on the inverse modeling and reconstruction of visually consistent and property-verifiable oceans, taking advantage of deep learning and differentiable physics to learn geometry and constitute waves in a self-supervised manner. First, we infer hierarchical geometry using two networks, which are optimized via the differentiable renderer. We extract wave components from the sequence of inferred geometry through a network equipped with a differentiable ocean model. Then, ocean dynamics can be evolved using the reconstructed wave components. Through extensive experiments, we verify that our new method yields satisfactory results for both geometry reconstruction and wave estimation. Moreover, the new framework has the inverse modeling potential to facilitate a host of graphics applications, such as the rapid production of physically accurate scene animation and editing guided by real ocean scenes.

The authors apologize for a hidden error in the article. It is that the images in Figs. 14(a) and 14(d) were mistakenly presented as left–right mirror images. The authors have flipped them to ensure that the figures now correspond correctly with others in the subfigures (b, c, e, f). The accurate version of Fig. 14 is provided as below.

Abstract Road network design, as an important part of landscape modeling, shows a great significance in automatic driving, video game development, and disaster simulation. To date, this task remains labor‐intensive, tedious and time‐consuming. Many improved techniques have been proposed during the last two decades. Nevertheless, most of the state‐of‐the‐art methods still encounter problems of intuitiveness, usefulness and/or interactivity. As a rapid deviation from the conventional road design, this paper advocates an improved road modeling framework for automatic and interactive road production driven by geographical maps (including elevation, water, vegetation maps). Our method integrates the capability of flexible image generation models with powerful transformer architecture to afford a vectorized road network. We firstly construct a dataset that includes road graphs, density map and their corresponding geographical maps. Secondly, we develop a density map generation network based on image translation model with an attention mechanism to predict a road density map. The usage of density map facilitates faster convergence and better performance, which also serves as the input for road graph generation. Thirdly, we employ the transformer architecture to evolve density maps to road graphs. Our comprehensive experimental results have verified the efficiency, robustness and applicability of our newly‐proposed framework for road design.

Skeleton-based action recognition is crucial for natural human-computer interaction, dynamic behavior analysis, and behavior surveillance. The key challenge is to effectively capture the intrinsic local-global clues of the activity. However, it remains challenging to efficiently leverage multidimensional information related to joints' local visual appearances, global spatial relationships, and coherent temporal cues. To address this challenge, we propose a joints-centered spatial-temporal feature-fused framework for action recognition, which exploits skeleton-based graph diffusion and convolution. Specifically, we employ Partial Differential Equation (PDE) based skeleton graph diffusion to automatically activate and diffuse the salient appearance features of joints. This approach simultaneously integrates the joints' appearance clues and their hierarchical relationships at both the super-pixel level and structure level. The diffused appearance-related features of the joints are further fused with skeleton-related spatial-temporal features, and the resulting fused features are fed into a skeleton convolution network for action recognition. Our method was extensively evaluated on two public datasets (NTU-RGBD and UWA3D), and the results demonstrate the improved accuracy and effectiveness of our approach. Our code will be public.

Recent years have witnessed the rapid deployment of numerous physics-based modeling and simulation algorithms and techniques for fluids, solids, and their delicate coupling in computer animation. However, it still remains a challenging problem to model the complex elastic-viscoplastic behaviors during fluid–solid phase transitions and facilitate their seamless interactions inside the same framework. In this article, we propose a practical method capable of simulating granular flows, viscoplastic liquids, elastic-plastic solids, rigid bodies, and interacting with each other, to support novel phenomena all heavily involving realistic phase transitions, including dissolution, melting, cooling, expansion, shrinking, and so on. At the physics level, we propose to combine and morph von Mises with Drucker–Prager and Cam–Clay yield models to establish a unified phase-field-driven EVP model, capable of describing the behaviors of granular, elastic, plastic, viscous materials, liquid, non-Newtonian fluids, and their smooth evolution. At the numerical level, we derive the discretization form of Cahn–Hilliard and Allen–Cahn equations with the material point method to effectively track the phase-field evolution, so as to avoid explicit handling of the boundary conditions at the interface. At the application level, we design a novel heuristic strategy to control specialized behaviors via user-defined schemes, including chemical potential, density curve, and so on. We exhibit a set of numerous experimental results consisting of challenging scenarios to validate the effectiveness and versatility of the new unified approach. This flexible and highly stable framework, founded upon the unified treatment and seamless coupling among various phases, and effective numerical discretization, has its unique advantage in animation creation toward novel phenomena heavily involving phase transitions with artistic creativity and guidance.

High-accuracy, high-efficiency physics-based fluid-solid interaction is essential for reality modeling and computer animation in online games or real-time Virtual Reality (VR) systems. However, the large-scale simulation of incompressible fluid and its interaction with the surrounding solid environment is either time-consuming or suffering from the reduced time/space resolution due to the complicated iterative nature pertinent to numerical computations of involved Partial Differential Equations (PDEs). In recent years, we have witnessed significant growth in exploring a different, alternative data-driven approach to addressing some of the existing technical challenges in conventional model-centric graphics and animation methods. This article showcases some of our exploratory efforts in this direction. One technical concern of our research is to address the central key challenge of how to best construct the numerical solver effectively and how to best integrate spatiotemporal/dimensional neural networks with the available MPM's pressure solvers. In particular, we devise the MPMNet, a hybrid data-driven framework supporting the popular and powerful MPM, to combine the comprehensive properties of MPM in numerically handling physical behaviors ranging from fluid to deformable solids and the high efficiency of data-driven models. At the architectural level, our MPMNet comprises three primary components: A data processing module to describe the physical properties by way of the input fields; A deep neural network group to learn the spatiotemporal features; And an iterative refinement process to continue to reduce possible numerical errors. The goal of these special technical developments is to aim at involved numerical acceleration while preserving physical accuracy, realizing efficient and accurate fluid-solid interactions in a data-driven fashion. The extensive experimental results verify that our MPMNet can tremendously speed up the computation compared with the popular numerical methods as the complexity of interaction scenes increases while better retaining the numerical accuracy.

In this article, we present a unified framework to simulate non-Newtonian behaviors. We combine viscous and elasto-plastic stress into a unified particle solver to achieve various non-Newtonian behaviors ranging from fluid-like to solid-like. Our constitutive model is based on a Generalized Maxwell model, which incorporates viscosity, elasticity and plasticity in one non-linear framework by a unified way. On the one hand, taking advantage of the viscous term, we construct a series of strain-rate dependent models for classical non-Newtonian behaviors such as shear-thickening, shear-thinning, Bingham plastic, etc. On the other hand, benefiting from the elasto-plastic model, we empower our framework with the ability to simulate solid-like non-Newtonian behaviors, i.e., visco-elasticity/plasticity. In addition, we enrich our method with a heat diffusion model to make our method flexible in simulating phase change. Through sufficient experiments, we demonstrate a wide range of non-Newtonian behaviors ranging from viscous fluid to deformable objects. We believe this non-Newtonian model will enhance the realism of physically-based animation, which has great potential for computer graphics.

Cognitive control is often perplexing to elucidate and can be easily influenced by emotions. Understanding the individual cognitive control level is crucial for enhancing VR interaction and designing adaptive and self-correcting VR/AR applications. Emotions can reallocate processing resources and influence cognitive control performance. However, current research has primarily emphasized the impact of emotional valence on cognitive control tasks, neglecting emotional arousal. In this study, we comprehensively investigate the influence of emotions on cognitive control based on the arousal-valence model. A total of 26 participants are recruited, inducing emotions through VR videos with high ecological validity and then performing related cognitive control tasks. Leveraging physiological data including EEG, HRV, and EDA, we employ classification techniques such as SVM, KNN, and deep learning to categorize cognitive control levels. The experiment results demonstrate that high-arousal emotions significantly enhance users' cognitive control abilities. Utilizing complementary information among multi-modal physiological signal features, we achieve an accuracy of 84.52% in distinguishing between high and low cognitive control. Additionally, time-frequency analysis results confirm the existence of neural patterns related to cognitive control, contributing to a better understanding of the neural mechanisms underlying cognitive control in VR. Our research indicates that physiological signals measured from both the central and autonomic nervous systems can be employed for cognitive control classification, paving the way for novel approaches to improve VR/AR interactions.

3D Facial animations, crucial to augmented and mixed reality digital media, have evolved from mere aesthetic elements to potent storytelling media. Despite considerable progress in facial animation of neutral emotions, existing methods still struggle to capture the authenticity of emotions. This paper introduces a novel approach to capture fine facial expressions and generate facial animations using audio synchronization. Our method consists of two key components: First, the Local-to-global Latent Diffusion Model (LG-LDM) tailored for authentic facial expressions, which can integrate audio, time step, facial expressions, and other conditions towards possible encoding of emotionally rich yet latent features in response to possibly noisy raw audio signals. The core of LG-LDM is our carefully designed Facial Denoiser Model (FDM) for aligning the local-to-global animation feature with audio. Second, we redesign an Emotion-centric Vector Quantized-Variational AutoEncoder framework (EVQ-VAE) to finely decode the subtle differences under different emotions and reconstruct the final 3D facial geometry. Our work significantly contributes to the key challenges of emotionally realistic 3D facial animation for audio synchronization and enhances the immersive experience and emotional depth in augmented and mixed reality applications. We provide a reproducibility kit including our code, dataset, and detailed instructions for running the experiments. This kit is available at https://github.com/wangxuanx/Face-Diffusion-Model.

It is widely acknowledged that selecting appropriate training data is crucial for obtaining good results in real-world testing, more so than utilizing complex network architectures. However, in the field of RGB-D SOD research, researchers have primarily focused on enhancing network architectures and have given less consideration to the choice of training and testing datasets, which may not translate well in practical applications. This paper aims to address an existing issue - how can we automatically generate a data-driven RGB-D SOD training dataset? We propose that in addition to scene similarity, the concept of "modality profile'' should be taken into account. The term "modality profile'' refers to the complementary status of modalities within a given dataset. A training dataset with a modality profile similar to the test dataset can significantly improve performance. To address this, we present a viable solution for automatically generating a training dataset with any desired modality profile in a weakly supervised manner. Our method also provides high-quality pseudo-GTs for all RGB-D images obtained from the web, making it suitable for training RGB-D SOD models. Extensive quantitative evaluations demonstrate the significance of the proposed "modality profile'' and confirm the superiority of the newly constructed training set guided by our "modality profile''. All codes, datasets, and results are available at this link.

Abstract Desiccation cracking of soil‐like materials is a common phenomenon in natural dry environment, however, it remains a challenge to model and simulate complicated multi‐physical processes inside the porous structure. With the goal of tracking such physical evolution accurately, we propose an MPM based method to simulate volumetric shrinkage and crack during moisture diffusion. At the physical level, we introduce Richards equations to evolve the dynamic moisture field to model evaporation and diffusion in unsaturated soils, with which a elastoplastic model is established to simulate strength changes and volumetric shrinkage via a novel saturation‐based hardening strategy during plastic treatment. At the algorithmic level, we develop an MPM‐fashion numerical solver for the proposed physical model and achieve stable yet efficient simulation towards delicate deformation and fracture. At the geometric level, we propose a correlating stretching criteria and a saturation‐aware extrapolation scheme to extend existing surface reconstruction for MPM, producing visual compelling soil appearance. Finally, we manifest realistic simulation results based on the proposed method with several challenging scenarios, which demonstrates usability and efficiency of our method.

Graph Convolutional Networks (GCNs) have successfully boosted skeleton-based human action recognition. However, existing GCN-based methods mostly cast the problem as separated person's action recognition while ignoring the interaction between the action initiator and the action responder, especially for the fundamental two-person interactive action recognition. It is still challenging to effectively take into account the intrinsic local-global clues of the two-person activity. Additionally, message passing in GCN depends on adjacency matrix, but skeleton-based human action recognition methods tend to calculate the adjacency matrix with the fixed natural skeleton connectivity. It means that messages can only travel along a fixed path at different layers of the network or in different actions, which greatly reduces the flexibility of the network. To this end, we propose a novel graph diffusion convolutional network for skeleton based semantic recognition of two-person actions by embedding the graph diffusion into GCNs. At technical fronts, we dynamically construct the adjacency matrix based on practical action information, so that we can guide the message propagation in a more meaningful way. Simultaneously, we introduce the frame importance calculation module to conduct dynamic convolution, so that we can avoid the negative effect caused by the traditional convolution, wherein the shared weights may fail to capture key frames or be affected by noisy frames. Besides, we comprehensively leverage the multidimensional features related to joints' local visual appearances, global spatial relationship and temporal coherency, and for different features, different metrics are designed to measure the similarity underlying the corresponding real physical law of the motions. Moreover, extensive experiments and comprehensive evaluations on four public large-scale datasets (NTU-RGB+D 60, NTU-RGB+D 120, Kinetics-Skeleton 400, and SBU-Interaction) demonstrate that our method outperforms the state-of-the-art methods.

In this paper, we propose a new method, called DoubleCoverUDF, for extracting the zero level-set from unsigned distance fields (UDFs). DoubleCoverUDF takes a learned UDF and a user-specified parameter r (a small positive real number) as input and extracts an iso-surface with an iso-value r using the conventional marching cubes algorithm. We show that the computed iso-surface is the boundary of the r -offset volume of the target zero level-set S , which is an orientable manifold, regardless of the topology of S. Next, the algorithm computes a covering map to project the boundary mesh onto S , preserving the mesh's topology and avoiding folding. If S is an orientable manifold surface, our algorithm separates the double-layered mesh into a single layer using a robust minimum-cut post-processing step. Otherwise, it keeps the double-layered mesh as the output. We validate our algorithm by reconstructing 3D surfaces of open models and demonstrate its efficacy and effectiveness on synthetic models and benchmark datasets. Our experimental results confirm that our method is robust and produces meshes with better quality in terms of both visual evaluation and quantitative measures than existing UDF-based methods. The source code is available at https://github.com/jjjkkyz/DCUDF.

Non-blind deblurring (NBD) is a modeling method of the image deblurring problem in computer vision, where the blurring kernel is known or can be externally estimated. In this paper, we attempt to solve a parametric NBD problem, inspired by the simultaneous acquisition of ptychography and fluorescent imaging (FI). Ptychography is an imaging method that favors larger probes, i.e. convolutional kernels, while FI relies on a small probe for high resolution. Also, the kernel can be solved during ptychographic reconstruction. With Ptycho-FI using the same larger kernel, we can perform NBD on the blurred fluorescent images to achieve high-resolution FI, and thus speed up the experiments. To this end, we design a deep latent space deformation network that is directly parameterized by the kernel. The network consists of three components: encoder, deformer, and decoder, where the deformer is specifically meant to rectify the latent space representations of blurred images to a standard latent space, regardless of the kernel. The deformation network is trained with a two-stage training scheme. We conduct extensive experiments to confirm that our parametric model can adapt to drastically different blurring kernels and perform robust deblurring.

Abstract As a usual component in virtual scenes, water surface plays an important role in various graphical applications, including special effects, video games, and virtual reality. Although recent years have witnessed significant progress based on Navier–Stokes equations and simplified water models, large‐scale water surface waves with high‐frequency visual details remain computationally expensive for interactive applications. This article proposes a novel frequency‐aware neural network to synthesize consistent and detailed water surface waves from low‐resolution input. At its core, our approach leverage the wavelet transformation theory over space, frequency and direction, and incremental supervision to decompose the 4D amplitude function into multiple smaller subproblems. Specifically, we first customize four subnetworks and corresponding loss functions for super‐resolution of spatial resolution, temporal evolution, wave direction subdivision, and wave number, respectively. Then, to enforce the upsampling along each dimension orthogonal to each other, we introduce a cooperative training scheme to fine‐tune and integrate the proposed subnetworks with carefully designed training dataset. Our method can visually enhance high‐resolution spatial details, temporal coherence, interactions with complex boundaries, and various wave patterns with flexible control along multiple dimensions. Through extensive experiments, our method arrives at 13 speedup for 32 upsampling of various simulation scenarios. We also validate the effectiveness and robustness of our method to produce realistic water surface waves toward artistic innovation.

Recent research advances in salient object detection (SOD) could largely be attributed to ever-stronger multi-scale feature representation empowered by the deep learning technologies. The existing SOD deep models extract multi-scale features via the off-the-shelf encoders and combine them smartly via various delicate decoders. However, the kernel sizes in this commonly-used thread are usually "fixed". In our new experiments, we have observed that kernels of small size are preferable in scenarios containing tiny salient objects. In contrast, large kernel sizes could perform better for images with large salient objects. Inspired by this observation, we advocate the "dynamic" scale routing (as a brand-new idea) in this paper. It will result in a generic plug-in that could directly fit the existing feature backbone. This paper's key technical innovations are two-fold. First, instead of using the vanilla convolution with fixed kernel sizes for the encoder design, we propose the dynamic pyramid convolution (DPConv), which dynamically selects the best-suited kernel sizes w.r.t. the given input. Second, we provide a self-adaptive bidirectional decoder design to accommodate the DPConv-based encoder best. The most significant highlight is its capability of routing between feature scales and their dynamic collection, making the inference process scale-aware. As a result, this paper continues to enhance the current SOTA performance. Both the code and dataset are publicly available at https://github.com/wuzhenyubuaa/DPNet.

The accurate detection of dental plaque at an early stage will definitely prevent periodontal diseases and dental caries. However, it remains difficult for the current dental examination to accurately recognize dental plaque without using medical dyeing reagent due to the low contrast between dental plaque and healthy teeth. To combat this problem, this paper proposes a novel network enhanced by a self-attention module for intelligent dental plaque segmentation. The key motivation is to directly utilize oral endoscope images (bypassing the need for dyeing reagent) and get accurate pixel-level dental plaque segmentation results. The algorithm needs to conduct self-attention at the super-pixel level and fuse the super-pixels' local-to-global features. Our newly-designed network architecture will afford the simultaneous fusion of multiple-scale complementary information guided by the powerful deep learning paradigm. The critical fused information includes the statistical distribution of the plaques color, the heat kernel signature (HKS) based local-to-global structure relationship, and the circle-LBP based local texture pattern in the nearby regions centering around the plaque area. To further refine the fuzed multiple-scale features, we devise an attention module based on CNN, which could focalize the regions of interest in plaque more easily, especially for many challenging cases. Extensive experiments and comprehensive evaluations confirm that, for a small-scale training dataset, our method could outperform the state-of-the-art methods. Meanwhile, the user studies verify the claim that our method is more accurate than conventional dental practice conducted by experienced dentists.

Compared with the conventional hand-crafted approaches, the deep learning based ISOD (image salient object detection) models have achieved tremendous performance improvements by training exquisitely crafted fancy networks over large-scale training sets. However, do we really need large-scale training set for ISOD? In this article, we provide a deeper insight into the interrelationship between the ISOD performance and the training data. To alleviate the conventional demands for large-scale training data, we provide a feasible way to construct a novel small-scale training set, which only contains 4 K images. To take full advantage of this new set, we propose a novel bi-stream network consisting of two different feature backbones. Benefit from the proposed gate control unit, this bi-stream network is able to achieve complementary fusion status for its subbranches. To our best knowledge, this is the first attempt to use a small-scale training set to compete with other large-scale ones; nevertheless, our method can still achieve the leading SOTA performance on all tested benchmark datasets. Both the code and dataset are publicly available athttps://github.com/wuzhenyubuaa/TSNet.

Poisson surface reconstruction (PSR) remains a popular technique for reconstructing watertight surfaces from 3D point samples thanks to its efficiency, simplicity, and robustness. Yet, the existing PSR method and subsequent variants work only for oriented points. This paper intends to validate that an improved PSR, called iPSR, can completely eliminate the requirement of point normals and proceed in an iterative manner. In each iteration, iPSR takes as input point samples with normals directly computed from the surface obtained in the preceding iteration, and then generates a new surface with better quality. Extensive quantitative evaluation confirms that the new iPSR algorithm converges in 5--30 iterations even with randomly initialized normals. If initialized with a simple visibility based heuristic, iPSR can further reduce the number of iterations. We conduct comprehensive comparisons with PSR and other powerful implicit-function based methods. Finally, we confirm iPSR's effectiveness and scalability on the AIM@SHAPE dataset and challenging (indoor and outdoor) scenes. Code and data for this paper are at https://github.com/houfei0801/ipsr.

The ecological validity of emotion-inducing scenarios is essential for emotion research. In contrast to the classical passive induction paradigm, immersive VR fully engages the psychological and physiological components of the subject, which is considered an ecologically valid paradigm for studying emotion. Several studies investigate the emotional responses to different VR tasks or games using subjective scales. However, little research regards VR as an eliciting material, especially when systematically analyzing emotional processes in VR from a neurophysiological perspective. To fill this gap and scientifically evaluate VR's ability to be used as an active method for emotion elicitation, we investigate the dynamic relationship between explicit information (subjective evaluations) and implicit information (objective neurophysiological data). A total of 28 participants are enlisted to watch eight VR videos while their SAM/IPQ scores and EEG data are recorded simultaneously. In ecologically valid scenarios, the subjective results demonstrate that VR has significant advantages for evoking emotion in arousal-valence. This conclusion is backed by our examination of objective neurophysiological evidence that VR videos effectively induce high-arousal emotions. In addition, we obtain features of critical channels and frequency oscillations associated with emotional valence, thereby validating previous research in more lifelike circumstances. In particular, we discover hemispheric asymmetry in the occipital region under high and low emotional arousal, which adds to our understanding of neural features and the dynamics of emotional arousal. As a result, we successfully integrate EEG and VR to demonstrate that VR is more pragmatic for evoking natural feelings and is beneficial for emotional research. Our research has set a precedent for new methodologies of using VR induction paradigms to acquire a more reliable explanation of affective computing.

In this paper, we devise a novel semantic completion network, called point cloud semantic scene completion network (PCSSC-Net), for indoor scenes solely based on point clouds. Existing point cloud completion networks still suffer from their inability of fully recovering complex structures and contents from global geometric descriptions neglecting semantic hints. To extract and infer comprehensive information from partial input, we design a patch-based contextual encoder to hierarchically learn point-level, patch-level, and scene-level geometric and contextual semantic information with a divide-and-conquer strategy. Consider that the scene semantics afford a high-level clue of constituting geometry for an indoor scene environment, we articulate a semantics-guided completion decoder where semantics could help cluster isolated points in the latent space and infer complicated scene geometry. Given the fact that real-world scans tend to be incomplete as ground truth, we choose to synthesize scene dataset with RGB-D images and annotate complete point clouds as ground truth for the supervised training purpose. Extensive experiments validate that our new method achieves the state-of-the-art performance, in contrast with the current methods applied to our dataset.

Thanks to the rapid advances in the deep learning techniques and the wide availability of large-scale training sets, the performances of video saliency detection models have been improving steadily and significantly. However, the deep learning based visual-audio fixation prediction is still in its infancy. At present, only a few visual-audio sequences have been furnished with real fixations being recorded in the real visual-audio environment. Hence, it would be neither efficiency nor necessary to re-collect real fixations under the same visual-audio circumstance. To address the problem, this paper advocate a novel approach in a weakly-supervised manner to alleviating the demand of large-scale training sets for visual-audio model training. By using the video category tags only, we propose the selective class activation mapping (SCAM), which follows a coarse-to-fine strategy to select the most discriminative regions in the spatial-temporal-audio circumstance. Moreover, these regions exhibit high consistency with the real human-eye fixations, which could subsequently be employed as the pseudo GTs to train a new spatial-temporal-audio (STA) network. Without resorting to any real fixation, the performance of our STA network is comparable to that of the fully supervised ones. Our code and results are publicly available at https://github.com/guotaowang/STANet.

Prior knowledge integration helps identify semantic entities and their relationships in a graphical representation, however, its meaningful abstraction and intervention remain elusive. This paper advocates a knowledge-inspired 3D scene graph prediction method solely based on point clouds. At the mathematical modeling level, we formulate the task as two sub-problems: knowledge learning and scene graph prediction with learned prior knowledge. Unlike conventional methods that learn knowledge embedding and regular patterns from encoded visual information, we propose to suppress the misunderstandings caused by appearance similarities and other perceptual confusion. At the network design level, we devise a graph auto-encoder to automatically extract class-dependent representations and topological patterns from the one-hot class labels and their intrinsic graphical structures, so that the prior knowledge can avoid perceptual errors and noises. We further devise a scene graph prediction model to predict credible relationship triplets by incorporating the related prototype knowledge with perceptual information. Comprehensive experiments confirm that, our method can successfully learn representative knowledge embedding, and the obtained prior knowledge can effectively enhance the accuracy of relationship predictions. Our thorough evaluations indicate the new method can achieve the state-of-the-art performance compared with other scene graph prediction methods.

Abstract Despite the rapid development and proliferation of computer graphics hardware devices for scene capture in the most recent decade, the high‐resolution 3D/4D acquisition of gaseous scenes (e.g., smokes) in real time remains technically challenging in graphics research nowadays. In this paper, we explore a hybrid approach to simultaneously taking advantage of both the model‐centric method and the data‐driven method. Specifically, this paper develops a novel conditional generative model to rapidly reconstruct the temporal density and velocity fields of gaseous phenomena based on the sequence of two projection views. With the data‐driven method, we can achieve the strong coupling of density update and the estimation of flow motion, as a result, we can greatly improve the reconstruction performance for smoke scenes. First, we employ a conditional generative network to generate the initial density field from input projection views and estimate the flow motion based on the adjacent frames. Second, we utilize the differentiable advection layer and design a velocity estimation network with the long‐term mechanism to help achieve the end‐to‐end training and more stable graphics effects. Third, we can re‐simulate the input scene with flexible coupling effects based on the estimated velocity field subject to artists' guidance or user interaction. Moreover, our generative model could accommodate single projection view as input. In practice, more input projection views are enabling and facilitating the high‐fidelity reconstruction with more realistic and finer details. We have conducted extensive experiments to confirm the effectiveness, efficiency, and robustness of our new method compared with the previous state‐of‐the‐art techniques.

With the rapid development of deep learning techniques, image saliency deep models trained solely by spatial information have occasionally achieved detection performance for video data comparable to that of the models trained by both spatial and temporal information. However, due to the lesser consideration of temporal information, the image saliency deep models may become fragile in the video sequences dominated by temporal information. Thus, the most recent video saliency detection approaches have adopted the network architecture starting with a spatial deep model that is followed by an elaborately designed temporal deep model. However, such methods easily encounter the performance bottleneck arising from the single stream learning methodology, so the overall detection performance is largely determined by the spatial deep model. In sharp contrast to the current mainstream methods, this paper proposes a novel plug-and-play scheme to weakly retrain a pretrained image saliency deep model for video data by using the newly sensed and coded temporal information. Thus, the retrained image saliency deep model will be able to maintain temporal saliency awareness, achieving much improved detection performance. Moreover, our method is simple yet effective for adapting any off-the-shelf pre-trained image saliency deep model to obtain high-quality video saliency detection. Additionally, both the data and source code of our method are publicly available.

We have witnessed a growing interest in video salient object detection (VSOD) techniques in today's computer vision applications. In contrast with temporal information (which is still considered a rather unstable source thus far), the spatial information is more stable and ubiquitous, thus it could influence our vision system more. As a result, the current main-stream VSOD approaches have inferred and obtained their saliency primarily from the spatial perspective, still treating temporal information as subordinate. Although the aforementioned methodology of focusing on the spatial aspect is effective in achieving a numeric performance gain, it still has two critical limitations. First, to ensure the dominance by the spatial information, its temporal counterpart remains inadequately used, though in some complex video scenes, the temporal information may represent the only reliable data source, which is critical to derive the correct VSOD. Second, both spatial and temporal saliency cues are often computed independently in advance and then integrated later on, while the interactions between them are omitted completely, resulting in saliency cues with limited quality. To combat these challenges, this paper advocates a novel spatiotemporal network, where the key innovation is the design of its temporal unit. Compared with other existing competitors (e.g., convLSTM), the proposed temporal unit exhibits an extremely lightweight design that does not degrade its strong ability to sense temporal information. Furthermore, it fully enables the computation of temporal saliency cues that interact with their spatial counterparts, ultimately boosting the overall VSOD performance and realizing its full potential towards mutual performance improvement for each. The proposed method is easy to implement yet still effective, achieving high-quality VSOD at 50 FPS in real-time applications.

The existing fusion-based RGB-D salient object detection methods usually adopt the bistream structure to strike a balance in the fusion trade-off between RGB and depth (D). While the D quality usually varies among the scenes, the state-of-the-art bistream approaches are depth-quality-unaware, resulting in substantial difficulties in achieving complementary fusion status between RGB and D and leading to poor fusion results for low-quality D. Thus, this paper attempts to integrate a novel depth-quality-aware subnet into the classic bistream structure in order to assess the depth quality prior to conducting the selective RGB-D fusion. Compared to the SOTA bistream methods, the major advantage of our method is its ability to lessen the importance of the low-quality, no-contribution, or even negative-contribution D regions during RGB-D fusion, achieving a much improved complementary status between RGB and D. Our source code and data are available online at https://github.com/qdu1995/DQSD.

Directly benefiting from the deep learning methods, object detection has witnessed a great performance boost in recent years. However, drone-view object detection remains challenging for two main reasons: (1) Objects of tiny-scale with more blurs w.r.t. ground-view objects offer less valuable information towards accurate and robust detection; (2) The unevenly distributed objects make the detection inefficient, especially for regions occupied by crowded objects. Confronting such challenges, we propose an end-to-end global-local self-adaptive network (GLSAN) in this paper. The key components in our GLSAN include a global-local detection network (GLDN), a simple yet efficient self-adaptive region selecting algorithm (SARSA), and a local super-resolution network (LSRN). We integrate a global-local fusion strategy into a progressive scale-varying network to perform more precise detection, where the local fine detector can adaptively refine the target's bounding boxes detected by the global coarse detector via cropping the original images for higher-resolution detection. The SARSA can dynamically crop the crowded regions in the input images, which is unsupervised and can be easily plugged into the networks. Additionally, we train the LSRN to enlarge the cropped images, providing more detailed information for finer-scale feature extraction, helping the detector distinguish foreground and background more easily. The SARSA and LSRN also contribute to data augmentation towards network training, which makes the detector more robust. Extensive experiments and comprehensive evaluations on the VisDrone2019-DET benchmark dataset and UAVDT dataset demonstrate the effectiveness and adaptivity of our method. Towards an industrial application, our network is also applied to a DroneBolts dataset with proven advantages. Our source codes have been available at https://github.com/dengsutao/glsan.

Human attention is an interactive activity between our visual system and our brain, using both low-level visual stimulus and high-level semantic information. Previous image salient object detection (SOD) studies conduct their saliency predictions via a multitask methodology in which pixelwise saliency regression and segmentation-like saliency refinement are conducted simultaneously. However, this multitask methodology has one critical limitation: the semantic information embedded in feature backbones might be degenerated during the training process. Our visual attention is determined mainly by semantic information, which is evidenced by our tendency to pay more attention to semantically salient regions even if these regions are not the most perceptually salient at first glance. This fact clearly contradicts the widely used multitask methodology mentioned above. To address this issue, this paper divides the SOD problem into two sequential steps. First, we devise a lightweight, weakly supervised deep network to coarsely locate the semantically salient regions. Next, as a postprocessing refinement, we selectively fuse multiple off-the-shelf deep models on the semantically salient regions identified by the previous step to formulate a pixelwise saliency map. Compared with the state-of-the-art (SOTA) models that focus on learning the pixelwise saliency in single images using only perceptual clues, our method aims at investigating the object-level semantic ranks between multiple images, of which the methodology is more consistent with the human attention mechanism. Our method is simple yet effective, and it is the first attempt to consider salient object detection as mainly an object-level semantic reranking problem.

Existing RGB-D salient object detection methods treat depth information as an independent component to complement RGB and widely follow the bistream parallel network architecture. To selectively fuse the CNN features extracted from both RGB and depth as a final result, the state-of-the-art (SOTA) bistream networks usually consist of two independent subbranches: one subbranch is used for RGB saliency, and the other aims for depth saliency. However, depth saliency is persistently inferior to the RGB saliency because the RGB component is intrinsically more informative than the depth component. The bistream architecture easily biases its subsequent fusion procedure to the RGB subbranch, leading to a performance bottleneck. In this paper, we propose a novel data-level recombination strategy to fuse RGB with D (depth) before deep feature extraction, where we cyclically convert the original 4-dimensional RGB-D into DGB, RDB and RGD. Then, a newly lightweight designed triple-stream network is applied over these novel formulated data to achieve an optimal channel-wise complementary fusion status between the RGB and D, achieving a new SOTA performance.

CNN based lung segmentation models in absence of diverse training dataset fail to segment lung volumes in presence of severe pathologies such as large masses, scars, and tumors. To rectify this problem, we propose a multi-stage algorithm for lung volume segmentation from CT scans. The algorithm uses a 3D CNN in the first stage to obtain a coarse segmentation of the left and right lungs. In the second stage, shape correction is performed on the segmentation mask using a 3D structure correction CNN. A novel data augmentation strategy is adopted to train a 3D CNN which helps in incorporating global shape prior. Finally, the shape corrected segmentation mask is up-sampled and refined using a parallel flood-fill operation. The proposed multi-stage algorithm is robust in the presence of large nodules/tumors and does not require labeled segmentation masks for entire pathological lung volume for training. Through extensive experiments conducted on publicly available datasets such as NSCLC, LUNA, and LOLA11 we demonstrate that the proposed approach improves the recall of large juxtapleural tumor voxels by at least 15% over state-of-the-art models without sacrificing segmentation accuracy in case of normal lungs. The proposed method also meets the requirement of CAD software by performing segmentation within 5 seconds which is significantly faster than present methods.

Screen content images (SCIs) usually comprise various content types with sharp edges, in which artifacts or distortions can be effectively sensed by a vanilla structure similarity measurement in a full-reference manner. Nonetheless, almost all of the current state-of-the-art (SOTA) structure similarity metrics are “locally” formulated in a single-level manner, while the true human visual system (HVS) follows the multilevel manner; such mismatch could eventually prevent these metrics from achieving reliable quality assessment. To ameliorate this issue, this article advocates a novel solution to measure structure similarity “globally” from the perspective of sparse representation. To perform multilevel quality assessment in accordance with the real HVS, the abovementioned global metric will be integrated with the conventional local ones by resorting to the newly devised selective deep fusion network. To validate its efficacy and effectiveness, we have compared our method with 12 SOTA methods over two widely used large-scale public SCI datasets, and the quantitative results indicate that our method yields significantly higher consistency with subjective quality scores than the current leading works. Both the source code and data are also publicly available to gain widespread acceptance and facilitate new advancement and validation.

Multi-scale granular materials, such as powdered materials and mudslides, are pretty common in nature. Modeling such materials and their phase transitions remains challenging since this task involves the delicate representations of various ranges of particles with multiple scales that cause their property variations among liquid, granular solid (i.e., particles), and smoke-like materials. To effectively animate the complicated yet intriguing natural phenomena involving multi-scale granular materials and their phase transitions in graphics with high fidelity, this article advocates a hybrid Euler-Lagrange solver to handle the behaviors of involved discontinuous fluid-like materials faithfully. At the algorithmic level, we present a unified framework that tightly couples the affine particle-in-cell (APIC) solver with density field to achieve the transformation spanning across granular particles, dust cloud, powders, and their natural mixtures. For example, a part of the granular particles could be transformed into dust cloud while interacting with air and being represented by density field. Meanwhile, the velocity decrease of the involved materials could also result in the transit from the density-field-driven dust to powder particles. Besides, to further enhance our modeling and simulation power to broaden the range of multi-scale materials, we introduce a moisture property for granular particles to control the transitions between particles and viscous liquid. At the geometric level, we devise an additional surface-tracking procedure to simulate the viscous liquid phase. We can arrive at delicate viscous behaviors by controlling the corresponding yield conditions. Through various experiments with the different scenes design being conducted in our unified framework, we can validate the mixed multi-scale materials' mutual transformation processes. Our unified framework furnished with a hybrid solver can significantly enhance the modeling flexibility and the animation potential of the particle-grid hybrid materials in graphics.

This article articulates a novel learning framework for both parameter estimation and detail enhancement for Eulerian gas based on data guidance. The key motivation of this article is to devise a new hybrid, grid-based simulation that could inherit modeling and simulation advantages from both physically-correct simulation methods and powerful data-driven methods, while combating existing difficulties exhibited in both approaches. We first employ a convolutional neural network (CNN) to estimate the physical parameters of gaseous phenomena in Eulerian settings, then we can use the just-learnt parameters to re-simulate (with or without artists' guidance) for specific scenes with flexible coupling effects. Next, a second CNN is adopted to reconstruct the high-resolution velocity field to guide a fast re-simulation on the finer grid, achieving richer and more realistic details with little extra computational expense. From the perspective of physics-based simulation, our trained networks respect temporal coherence and physical constraints. From the perspective of the data-driven machine-learning approaches, our network design aims at extracting a meaningful parameters and reconstructing visually realistic details. Additionally, our implementation based on parallel acceleration could significantly enhance the computational performance of every involved module. Our comprehensive experiments confirm the controllability, effectiveness, and accuracy of our novel approach when producing various gaseous scenes with rich details for widespread graphics applications.

This paper presents a vector painting system for digital artworks. We first propose Temporal Diffusion Curve (TDC), a new form of vector graphics, and a novel random-access solver for modeling the evolution of strokes. With the help of a procedural stroke processing function, the TDC strokes can achieve various shapes and effects for multiple art styles. Based on these, we build a painting system of great potential. Thanks to the random-access solver, our method has real-time performance regardless of the rendering resolution, provides straightforward editing possibilities on strokes both at runtime and afterward, and is effective and straightforward for art production. Compared with the previous Diffusion Curve, our method uses strokes as the basic graphics primitives, which are able to intersect each other and much more consistent with the intuition and painting habits of human. We finally demonstrate that professional artists can create multiple genres of artworks with our painting system.

Point cloud filtering is a fundamental problem in geometry modeling and processing. Despite of significant advancement in recent years, the existing methods still suffer from two issues: 1) they are either designed without preserving sharp features or less robust in feature preservation; and 2) they usually have many parameters and require tedious parameter tuning. In this article, we propose a novel deep learning approach that automatically and robustly filters point clouds by removing noise and preserving their sharp features. Our point-wise learning architecture consists of an encoder and a decoder. The encoder directly takes points (a point and its neighbors) as input, and learns a latent representation vector which goes through the decoder to relate the ground-truth position with a displacement vector. The trained neural network can automatically generate a set of clean points from a noisy input. Extensive experiments show that our approach outperforms the state-of-the-art deep learning techniques in terms of both visual quality and quantitative error metrics. The source code and dataset can be found at https://github.com/dongbo-BUAA-VR/Pointfilter.

In recent years, virtual reality (VR) based training has greatly changed surgeons learning mode. It can simulate the surgery from the visual, auditory, and tactile aspects. VR medical simulator can greatly reduce the risk of the real patient and the cost of hospitals. Laparoscopic cholecystectomy is one of the typical representatives in minimal invasive surgery (MIS). Due to the large incidence of cholecystectomy, the application of its VR-based simulation is vital and necessary for the residents' surgical training. In this paper, we present a VR simulation framework based on position-based dynamics (PBD) for cholecystectomy. To further accelerate the deformation of organs, PBD constraints are solved in parallel by a graph coloring algorithm. We introduce a bio-thermal conduction model to improve the realism of the fat tissue electrocautery. Finally, we design a hybrid multi-model connection method to handle the interaction and simulation of the liver-gallbladder separation. This simulation system has been applied to laparoscopic cholecystectomy training in several hospitals. From the experimental results, users can operate in real-time with high stability and fidelity. The simulator is also evaluated by a number of digestive surgeons through preliminary studies. They believed that the system can offer great help to the improvement of surgical skills.

Today convolutional neural networks (CNNs) have reached out to specialized applications in science communities that otherwise would not be adequately tackled. In this paper, we systematically study a multi-label annotation problem of x-ray scattering images in material science. For this application, we tackle an open challenge with training CNNs - identifying weak scattered patterns with diffuse background interference, which is common in scientific imaging. We articulate an Attentional Aggregation Module (AAM) to enhance feature representations. First, we reweight and highlight important features in the images using data-driven attention maps. We decompose the attention maps into channel and spatial attention components. In the spatial attention component, we design a mechanism to generate multiple spatial attention maps tailored for diversified multi-label learning. Then, we condense the enhanced local features into non-local representations by performing feature aggregation. Both attention and aggregation are designed as network layers with learnable parameters so that CNN training remains fluidly end-to-end, and we apply it in-network a few times so that the feature enhancement is multi-scale. We conduct extensive experiments on CNN training and testing, as well as transfer learning, and empirical studies confirm that our method enhances the discriminative power of visual features of scientific imaging.

Recently CNN-centric object tracking methods have been gaining tremendous success in ground-view videos, however, it remains hard to cope with vehicle tracking in unmanned aerial vehicle (UAV) videos. The key difficulties mainly stem from lacking large-scale well-labeled training datasets and view-invariant appearance model for fast-moving drone-view vehicles. We enhance the vehicle's cross-view feature by exploring relations between the pivotal context and the target to facilitate unsupervised vehicle tracking. The relation is modeled as the relevance of the target and its contextual regions in the tracking task. Specifically, we propose a contextual relation actor-critic (CRAC) framework integrates an actor-critic agent with a dual GAN learning mechanism, which aims to dynamically search the related contextual regions and transfer the relations from ground-view to drone-view videos while retaining the discriminative features. We demonstrate that CRAC could be applied to several state-of-the-art trackers by extensive experiments and ablation studies on four public benchmarks. All the experiments confirm that, our CRAC can improve the performance of state-of-the-art methods in terms of accuracy, robustness, and versatility.

Abstract In physics‐based liquid simulation for graphics applications, pressure projection consumes a significant amount of computational time and is frequently the bottleneck of the computational efficiency. How to rapidly apply the pressure projection and at the same time how to accurately capture the liquid geometry are always among the most popular topics in the current research trend in liquid simulations. In this paper, we incorporate an artificial neural network into the simulation pipeline for handling the tricky projection step for liquid animation. Compared with the previous neural‐network‐based works for gas flows, this paper advocates new advances in the composition of representative features as well as the loss functions in order to facilitate fluid simulation with free‐surface boundary. Specifically, we choose both the velocity and the level‐set function as the additional representation of the fluid states, which allows not only the motion but also the boundary position to be considered in the neural network solver. Meanwhile, we use the divergence error in the loss function to further emulate the lifelike behaviours of liquid. With these arrangements, our method could greatly accelerate the pressure projection step in liquid simulation, while maintaining fairly convincing visual results. Additionally, our neutral network performs well when being applied to new scene synthesis even with varied boundaries or scales.

Abstract In this paper, we articulate a novel plastic phase‐field (PPF) method that can tightly couple the phase‐field with plastic treatment to efficiently simulate ductile fracture with GPU optimization. At the theoretical level of physically‐based modeling and simulation, our PPF approach assumes the fracture sensitivity of the material increases with the plastic strain accumulation. As a result, we first develop a hardening‐related fracture toughness function towards phase‐field evolution. Second, we follow the associative flow rule and adopt a novel degraded von Mises yield criterion. In this way, we establish the tight coupling of the phase‐field and plastic treatment, with which our PPF method can present distinct elastoplasticity, necking, and fracture characteristics during ductile fracture simulation. At the numerical level towards GPU optimization, we further devise an advanced parallel framework, which takes the full advantages of hierarchical architecture. Our strategy dramatically enhances the computational efficiency of preprocessing and phase‐field evolution for our PPF with the material point method (MPM). Based on our extensive experiments on a variety of benchmarks, our novel method's performance gain can reach 1.56× speedup of the primary GPU MPM. Finally, our comprehensive simulation results have confirmed that this new PPF method can efficiently and realistically simulate complex ductile fracture phenomena in 3D interactive graphics and animation.

Abstract In recent years, the topic of skeleton‐based human action recognition has attracted significant attention from researchers and practitioners in graphics, vision, animation, and virtual environments. The most fundamental issue is how to learn an effective and accurate representation from spatiotemporal action sequences towards improved performance, and this article aims to address the aforementioned challenge. In particular, we design a novel method of hybrid features' extraction based on the construction of multistream networks and their organic fusion. First, we train a convolution neural networks (CNN) model to learn CNN‐based features with the raw skeleton coordinates and their temporal differences serving as input signals. The attention mechanism is injected into the CNN model to weigh more effective and important information. Then, we employ long short‐term memory (LSTM) to obtain long‐term temporal features from action sequences. Finally, we generate the hybrid features by fusing the CNN and LSTM networks, and we classify action types with the hybrid features. The extensive experiments are performed on several large‐scale publically available databases, and promising results demonstrate the efficacy and effectiveness of our proposed framework.

Abstract In this article, we propose a novel hybrid framework by combining smoothed particle hydrodynamics and adaptive narrow band fluid implicit particle method (NB‐FLIP) to faithfully model the multiphysical processes involving heat transfer and phase transition, and to precisely simulate the dynamics of condensed droplets moving along intricate objects. We first formulate a governing physical model built upon an improved phase transition model and an augmented on‐surface drop analysis method to achieve realistic condensation effects over intricate hydrophilic/hydrophobic interface. To achieve both high‐fidelity interactions and high‐resolution visual effects, we further develop an adaptive NB‐FLIP solver with octree‐dictated background grid in order to further enhance the performance of our framework. Experimental results have shown that our approach can be used to efficiently and realistically simulate the small‐scale interaction details between condensed drops and complex objects with arbitrary geometry.

Abstract At present, virtual reality (VR) ‐based medical simulators provide an efficient and cost‐effective alternative without exposing risk to the traditional training approaches. As an essential and indispensable task in fundamental surgical skills training, the research of suturing simulation still remains insufficient in the field of virtual surgery. In this paper, we present a real‐time suturing simulation framework which can handle the complex interactions between surgical instruments and soft tissue. The simulation consists of two stages: external interaction and internal coupling. External interaction involves the interplay between needle/suture and the soft tissue, which are both deformed by position‐based dynamics (PBD) with different constraints. At the internal coupling stage, once the force exceeds a threshold, the needle tip will puncture and penetrate into the soft tissue and generate a path. To guarantee the needle/suture accurately following the path inside the soft tissue, we propose a novel coupling method by matching and generating the constraints among needle, suture, and penetration path. We have applied this suturing simulation into a VR laparoscopic surgery simulator with haptic force. Our experimental results demonstrate that our approach can achieve real‐time performance with a high degree of visual realism and haptic fidelity.

This paper proposes to utilize supervised deep convolutional neural networks to take full advantage of the long-term spatial-temporal information in order to improve the video saliency detection performance. The conventional methods, which use the temporally neighbored frames solely, could easily encounter transient failure cases when the spatial-temporal saliency clues are less-trustworthy for a long period. To tackle the aforementioned limitation, we plan to identify those beyond-scope frames with trustworthy long-term saliency clues first and then align it with the current problem domain for an improved video saliency detection.

To solve the saliency detection problem in RGB-D images, the depth information plays a critical role in distinguishing salient objects or foregrounds from cluttered backgrounds. As the complementary component to color information, the depth quality directly dictates the subsequent saliency detection performance. However, due to artifacts and the limitation of depth acquisition devices, the quality of the obtained depth varies tremendously across different scenarios. Consequently, conventional selective fusion-based RGB-D saliency detection methods may result in a degraded detection performance in cases containing salient objects with low color contrast coupled with a low depth quality. To solve this problem, we make our initial attempt to estimate additional high-quality depth information, which is denoted by Depth+. Serving as a complement to the original depth, Depth+ will be fed into our newly designed selective fusion network to boost the detection performance. To achieve this aim, we first retrieve a small group of images that are similar to the given input, and then the inter-image, nonlocal correspondences are built accordingly. Thus, by using these inter-image correspondences, the overall depth can be coarsely estimated by utilizing our newly designed depth-transferring strategy. Next, we build fine-grained, object-level correspondences coupled with a saliency prior to further improve the depth quality of the previous estimation. Compared to the original depth, our newly estimated Depth+ is potentially more informative for detection improvement. Finally, we feed both the original depth and the newly estimated Depth+ into our selective deep fusion network, whose key novelty is to achieve an optimal complementary balance to make better decisions toward improving saliency boundaries.

Color constancy is one of the fundamental tasks in computer vision. Many supervised methods, including recently proposed Convolutional Neural Networks (CNN)-based methods, have been proved to work well on this problem, but they often require a sufficient number of labeled data. However, it is expensive and time-consuming to collect a large number of labeled training images with accurately measured illumination. In order to reduce the dependence on labeled images and leverage unlabeled ones without measured illumination, we propose a novel semi-supervised framework with limited training samples for illumination estimation. Our key insight is that the images with similar features from different cues will share similar lighting conditions. Consequently, three graphs based on three visual cues, low-level RGB color distribution, mid-level initial illuminant estimates and high-level scene content, are constructed to represent the relationship among different images. Then a multi-cue semi-supervised color constancy method (MSCC) is proposed after integrating these three graphs into a unified model. Extensive experiments on benchmark datasets demonstrate that our proposed MSCC method outperforms nearly all the existing supervised methods with limited labeled samples. Even with no unlabeled samples, MSCC still obtains better performance and stableness than most supervised methods.

Despite growing progresses in recent years, cross-scenario person re-identification remains challenging, mainly due to the pedestrians commonly surrounded by highly-complex environment contexts. In reality, the human perception mechanism could adaptively find proper contextualized spatial-temporal clues towards pedestrian recognition. However, conventional methods fall short in adaptively leveraging the long-term spatial-temporal information due to ever-increasing computational cost. Moreover, CNN-based deep learning methods are hard to conduct optimization due to the non-differentiable property of the built-in context search operation. To ameliorate, this paper proposes a novel Context-Interactive CNN (CI-CNN) to dynamically find both spatial and temporal contexts by embedding multi-task Reinforcement Learning (MTRL). The CI-CNN streamlines the multi-task reinforcement learning by using an actor-critic agent to capture the temporal-spatial context simultaneously, which comprises a context-policy network and a context-critic network. The former network learns policies to determine the optimal spatial context region and temporal sequence range. Based on the inferred temporal-spatial cues, the latter one focuses on the identification task and provides feedback for the policy network. Thus, CI-CNN can simultaneously zoom in/out the perception field in spatial and temporal domain for the context interaction with the environment. By fostering the collaborative interaction between the person and context, our method could achieve outstanding performance on various public benchmarks, which confirms the rationality of our hypothesis, and verifies the effectiveness of our CI-CNN framework.

Specular reflections (i.e., highlight) always exist in endoscopic images, and they can severely disturb surgeons' observation and judgment. In an augmented reality (AR)-based surgery navigation system, the highlight may also lead to the failure of feature extraction or registration. In this paper, we propose an adaptive robust principal component analysis (Adaptive-RPCA) method to remove the specular reflections in endoscopic image sequences. It can iteratively optimize the sparse part parameter during RPCA decomposition. In this new approach, we first adaptively detect the highlight image based on pixels. With the proposed distance metric algorithm, it then automatically measures the similarity distance between the sparse result image and the detected highlight image. Finally, the low-rank and sparse results are obtained by enforcing the similarity distance between the two types of images to fall within a certain range. Our method has been verified by multiple different types of endoscopic image sequences in minimally invasive surgery (MIS). The experiments and clinical blind tests demonstrate that the new Adaptive-RPCA method can obtain the optimal sparse decomposition parameters directly and can generate robust highlight removal results. Compared with the state-of-the-art approaches, the proposed method not only achieves the better highlight removal results but also can adaptively process image sequences.

Conventional video saliency detection methods frequently follow the common bottom-up thread to estimate video saliency within the short-term fashion. As a result, such methods can not avoid the obstinate accumulation of errors when the collected low-level clues are constantly ill-detected. Also, being noticed that a portion of video frames, which are not nearby the current video frame over the time axis, may potentially benefit the saliency detection in the current video frame. Thus, we propose to solve the aforementioned problem using our newly-designed key frame strategy (KFS), whose core rationale is to utilize both the spatial-temporal coherency of the salient foregrounds and the objectness prior (i.e., how likely it is for an object proposal to contain an object of any class) to reveal the valuable long-term information. We could utilize all this newly-revealed long-term information to guide our subsequent “self-paced” saliency diffusion, which enables each key frame itself to determine its diffusion range and diffusion strength to correct those ill-detected video frames. At the algorithmic level, we first divide a video sequence into short-term frame batches, and the object proposals are obtained in a frame-wise manner. Then, for each object proposal, we utilize a pre-trained deep saliency model to obtain high-dimensional features in order to represent the spatial contrast. Since the contrast computation within multiple neighbored video frames (i.e., the non-local manner) is relatively insensitive to the appearance variation, those object proposals with high-quality low-level saliency estimation frequently exhibit strong similarity over the temporal scale. Next, the long-term common consistency (e.g., appearance models/movement patterns) of the salient foregrounds could be explicitly revealed via similarity analysis accordingly. We further boost the detection accuracy via long-term information guided saliency diffusion in a self-paced manner. We have conducted extensive experiments to compare our method with 16 state-of-the-art methods over 4 largest public available benchmarks, and all results demonstrate the superiority of our method in terms of both accuracy and robustness.

In recent years deep neural networks have been widely applied to visual saliency detection tasks with remarkable detection performance improvements. As for the salient object detection in single image, the automatically computed convolutional features frequently demonstrate high discriminative power to distinguish salient foregrounds from its non-salient surroundings in most cases. Yet, the obstinate feature conflicts still persist, which naturally gives rise to the learning ambiguity, arriving at massive failure detections. To solve such problem, we propose to jointly re-learn common consistency of inter-image saliency and then use it to boost the detection performance. Its core rationale is to utilize the easy-to-detect cases to re-boost much harder ones. Compared with the conventional methods, which focus on their problem domain within the single image scope, our method attempts to utilize those beyond-scope information to facilitate the current salient object detection. To validate our new approach, we have conducted a comprehensive quantitative comparisons between our approach and 13 state-of-the-art methods over 5 publicly available benchmarks, and all the results suggest the advantage of our approach in terms of accuracy, reliability, and versatility.

Directly benefited from deep learning techniques, depth estimation from single image has gained great momentum in recent years. However, most of the existing approaches treat depth prediction as an isolated problem without taking into consideration high-level semantic context information, which results in inefficient utilization of training dataset and unavoidably requires a large number of captured depth data during the training phase. To ameliorate, this paper develops a novel scene-aware contextualized convolution neural network (CCNN), which characterizes the semantic context relationship at the class-level and refines depth at the pixel-level. Our newly-proposed CCNN is built upon the intrinsic exploitation of context-dependent depth association, including inner-object continuous depth and inter-object depth change priors nearby. Specifically, rather than conducting regression on depth in single CNN, we make the first attempt to integrate both class-level and pixel-level conditional random fields (CRFs) based probabilistic graphical model into the powerful CNN framework to simultaneously learn different-level features within the same CNN layer. With our CCNN, the former model will guide the latter one to learn the contextualized RGB-Depth mapping. Hence, CCNN has desirable properties in both class-level integrity and pixel-level discrimination, which makes it ideal to share such two-level convolutional features in parallel during the end-to-end training with the commonly-used back-propagation algorithm. We conduct extensive experiments and comprehensive evaluations on public benchmarks involving various indoor and outdoor scenes, and all the experiments confirm that, our method outperforms the state-of-the-art depth estimation methods, especially for the cases where only small-scale training data are readily available.

This paper presents Poisson vector graphics (PVG), an extension of the popular diffusion curves (DC), for generating smooth-shaded images. Armed with two new types of primitives, called Poisson curves and Poisson regions, PVG can easily produce photorealistic effects such as specular highlights, core shadows, translucency and halos. Within the PVG framework, the users specify color as the Dirichlet boundary condition of diffusion curves and control tone by offsetting the Laplacian of colors, where both controls are simply done by mouse click and slider dragging. PVG distinguishes itself from other diffusion based vector graphics for 3 unique features: 1) explicit separation of colors and tones, which follows the basic drawing principle and eases editing; 2) native support of seamless cloning in the sense that PCs and PRs can automatically fit into the target background; and 3) allowed intersecting primitives (except for DC-DC intersection) so that users can create layers. Through extensive experiments and a preliminary user study, we demonstrate that PVG is a simple yet powerful authoring tool that can produce photo-realistic vector graphics from scratch.

The 2D image based salient object detection (SOD) has been extensively explored, while the 360° omnidirectional image based SOD has received less research attention and there exist three major bottlenecks that are limiting its performance. Firstly, the currently available training data is insufficient for the training of 360° SOD deep model. Secondly, the visual distortions in 360° omnidirectional images usually result in large feature gap between 360° images and 2D images; consequently, the widely used stage-wise training-a widely-used solution to alleviate the training data shortage problem, becomes infeasible when conducing SOD in 360° omnidirectional images. Thirdly, the existing 360° SOD approach has followed a multi-task methodology that performs salient object localization and segmentation-like saliency refinement at the same time, being faced with extremely large problem domain, making the training data shortage dilemma even worse. To tackle all these issues, this paper divides the 360° SOD into a multi-staqe task, the key rationale of which is to decompose the original complex problem domain into sequential easy sub problems that only demand for small-scale training data. Meanwhile, we learn how to rank the "object-level semantical saliency", aiming to locate salient viewpoints and objects accurately. Specifically, to alleviate the training data shortage problem, we have released a novel dataset named 360-SSOD, containing 1,105 360° omnidirectional images with manually annotated object-level saliency ground truth, whose semantical distribution is more balanced than that of the existing dataset. Also, we have compared the proposed method with 13 SOTA methods, and all quantitative results have demonstrated the performance superiority.

This paper extends the recently proposed power-particle-based fluid simulation method with staggered discretization, GPU implementation, and adaptive sampling, largely enhancing the efficiency and usability of the method. In contrast to the original formulation which uses co-located pressures and velocities, in this paper, a staggered scheme is adapted to the Power Particles to benefit visual details and computing efficiency. Meanwhile, we propose a novel facet-based power diagrams construction algorithm suitable for parallelization and explore its GPU implementation, achieving an order of magnitude boost in performance over the existing code library. In addition, to utilize the potential of Power Particles to control individual cell volume, we apply adaptive particle sampling to improve the detail level with varying resolution. The proposed method can be entirely carried out on GPUs, and our extensive experiments validate our method both in terms of efficiency and visual quality.

Thyroid nodule classification in ultrasound images has gained great momentum based on deep convolutional neural networks in recent years. Nevertheless, it is still challenging to intelligently classify the fine-grained thyroid nodules, which is significant for the subsequent clinical treatments. The difficulties mainly stem from four aspects: few fine-grained training dataset, highly-variable appearances of intra-class nodules, overall-similar characteristics of inter-class nodules, and the low resolution and contrast degree of the ultrasonic images as well as the influence of intrinsic speckle noises. In this paper, we propose a multi-semantic attention networks (MSAN) for fine-grained thyroid nodule classification in ultrasound images. Specifically, we employ a main network branch for coarse granularity feature extraction, which only focuses on the benign and malignant characteristics, and simultaneously employ multi-semantic network branches to extract discriminative features from the fine-grained pathological categories. Meanwhile, we introduce an self-attention scheme together with global average pooling (GAP) in our network, which facilitates to learn from the dynamically-selected nodule regions ranging from local to global. Extensive experiments demonstrate that, our MSAN gives rise to significant improvement of classification accuracy and outperforms the state-of-the-art methods.

Corneal disease is one of the most common causes of blindness for human beings in the world. Deep anterior lamellar k-eratoplasty (DALK) is a widely-used corneal transplantation technique, which requires precise control of surgical tools. This paper proposes a deep learning framework of augmented reality (AR) based surgical navigation to guide the suturing process in DALK. It aims to track the cutting corneal contour robustly through semantic segmentation and occlusion reconstruction. We devise a novel optical flow inpainting network to restore the missing motion caused by occlusion. The occluded regions are obtained using weakly-supervised segmentation of surgical tools and reconstructed by the key-frame warping along the completed optical flow. We introduce two kinds of loss functions to adapt the inpainting network to the optical flow space. The performance of our techniques is evaluated using real surgery videos from Shandong Eye Hospital. All experimental results show that our approach can achieve accurate corneal contour tracking subject to complex disturbance of tools in real-time surgical scenarios.

Monocular depth estimation has gained great momentum and achieved growing success recently. Nonetheless, due to the intrinsic difficulty associated with large-scale RGB-D data capture for training purpose and the inefficient utilization of existing training datasets, it is still challenging to accommodate flexibly-changing scenarios. To ameliorate, we propose a fewshot learning method for monocular depth estimation augmented by local object-object relationship. Our method is based on the insight that the depth changing between neighboring objects is relatively stable across diverse but similar scenarios. At the technical front, we first learn the object relationship based on the relative distance between single objects. Towards this goal, we design a CNN architecture to simultaneously encode the object spatial context into object-object relationship features and encode the original image into global context features. Hence we can complementally leverage few-shot dataset with only a few samples for depth estimation while preserving the global depth changing range and respecting the local object-object depth details. As a result, our novel approach could estimate depth from various indoor RGB images, which greatly alleviates the training dataset dependency in monocular depth estimation. Finally, we conduct extensive experiments and comprehensive evaluations on the widely-used public benchmarks, and all the experiments confirm that, our method outperforms the state-of-the-art depth estimation methods, especially for the cases where only smallscale training samples are available.

Convolutional Neural Network (CNN) has brought tremendous improvements in estimating 3D human pose from a monocular RGB image. However, the task of 3D human pose estimation still remains extremely challenging, especially when the task is geared towards estimating the depth of human body parts. Different from 2D human pose estimation, which focuses on the fusion of spatial information and context information, depth estimation demands more context information. Inspired by this, we build a Context-Aware Network (CAN) which can fully explore the context information to discover the underlying relationships among different body parts. The key ingredient of our network is High-Level Depth Estimation Module (HLDEM) designed to extract context information effectively. Additionally, multi-scale supervision is introduced in our network to extract context information at different scales. Experimental results show that our network achieves competitive performance compared with state-of-the-art methods on Human3.6M dataset.

The rapid generation and flexible reuse of characters animation by commodity devices are of significant importance to rich digital content production in virtual reality. This paper aims to handle the challenges of current motion imitation for human body in several indoor scenes (e.g., fitness training). We develop a real-time system based on single Kinect device, which is able to capture stable human motions and retarget to virtual characters. A large variety of motions and characters are tested to validate the efficiency and effectiveness of our system.

Powdered materials, such as sand and flour, are quite common in nature, whose properties always range from granular particles to smog materials under the air friction while throwing. This paper presents a hybrid method that tightly couples APIC solver with density field to accomplish the transformation of continuous powdered materials varying among granular particles, smog, powders and their natural mixtures. In our method, a part of the granular particles will be transformed to dust smog while interacting with air and represented by density field, then, as velocity decreases the density-based dust will deposit to powder particles. We construct a unified framework to imitate the mutual transformation process for the powdered materials of different scales, which greatly enhance the details of particle-based materials modeling. We have conducted extensive experiments to verify the performance of our model, and get satisfactory results in terms of stability, efficiency and visual authenticity as expected.

Abstract Generating realistic spray details in liquid simulations remains computationally expensive. This paper proposes a data‐driven method to simulate high‐resolution sprays on low‐resolution grids by retrieving details with the most compatible details from a precomputed repository efficiently. We first employ a random forest‐based distance (RFD) to measure the similarity of liquid regions. In consideration of spatiotemporal relationships between one liquid region and its neighbors, we define a multinary label for RFD instead of the original binary one. Our improved RFD enables us to retrieve details that fit ground truth the best. To ensure temporal continuity of our result and to generate new details from existing ones, we formulate a series of forests with a training set from different time steps. Then, we synthesize results of each forest according to their distances. Finally, we put the synthesis result in correct positions to generate desired sprays motion. In our method, a state‐of‐the‐art cascade forest is employed for a higher accuracy. Several experiments with various grid resolutions validate our method both in visual effect and computational cost.

Abstract In this paper, we present a new strategy, a joint deep learning architecture, for two classic tasks in computer graphics: water surface reconstruction and water image synthesis. Modeling water surfaces from single images can be regarded as the inverse of image rendering, which converts surface geometries into photorealistic images. On the basis of this fact, we therefore consider these two problems as a cycle image‐to‐image translation and propose to tackle them together using a pair of neural networks, with the three‐dimensional surface geometries being represented as two‐dimensional surface normal maps. Furthermore, we also estimate the imaging parameters from the existing water images with a subnetwork to reuse the lighting conditions when synthesizing new images. Experiments demonstrate that our method achieves an accurate reconstruction of surfaces from monocular images efficiently and produces visually plausible new images under variable lighting conditions.

The size and shape of a nodule are the essential indicators of malignancy in lung cancer diagnosis. However, effectively capturing the nodule's structural information from CT scans in a computer-aided system is a challenging task. Unlike previous models that proposed computationally intensive deep ensemble models or three-dimensional CNN models, we propose a lightweight, multiple view sampling based multi-section CNN architecture. The model obtains a nodule's cross sections from multiple view angles and encodes the nodule's volumetric information into a compact representation by aggregating information from its different cross sections via a view pooling layer. The compact feature is subsequently used for the task of nodule classification. The method does not require the nodule's spatial annotation and works directly on the cross sections generated from volume enclosing the nodule. We evaluated the proposed method on lung image database consortium (LIDC) and image database resource initiative (IDRI) dataset. It achieved the state-of-the-art performance with a mean 93.18% classification accuracy. The architecture could also be used to select the representative cross sections determining the nodule's malignancy that facilitates in the interpretation of results. Because of being lightweight, the model could be ported to mobile devices, which brings the power of artificial intelligence (AI) driven application directly into the practitioner's hand.

Thyroid ultrasonography is a widely used clinical technique for nodule diagnosis in thyroid regions. However, it remains difficult to detect and recognize the nodules due to low contrast, high noise, and diverse appearance of nodules. In today's clinical practice, senior doctors could pinpoint nodules by analyzing global context features, local geometry structure, and intensity changes, which would require rich clinical experience accumulated from hundreds and thousands of nodule case studies. To alleviate doctors' tremendous labor in the diagnosis procedure, we advocate a machine learning approach to the detection and recognition tasks in this paper. In particular, we develop a multitask cascade convolution neural network (MC-CNN) framework to exploit the context information of thyroid nodules. It may be noted that our framework is built upon a large number of clinically confirmed thyroid ultrasound images with accurate and detailed ground truth labels. Other key advantages of our framework result from a multitask cascade architecture, two stages of carefully designed deep convolution networks in order to detect and recognize thyroid nodules in a pyramidal fashion, and capturing various intrinsic features in a global-to-local way. Within our framework, the potential regions of interest after initial detection are further fed to the spatial pyramid augmented CNNs to embed multiscale discriminative information for fine-grained thyroid recognition. Experimental results on 4309 clinical ultrasound images have indicated that our MC-CNN is accurate and effective for both thyroid nodules detection and recognition. For the correct diagnosis rate of malignant and benign thyroid nodules, its mean Average Precision (mAP) performance can achieve up to [Formula: see text] accuracy, which outperforms the common CNNs by [Formula: see text] on average. In addition, we conduct rigorous user studies to confirm that our MC-CNN outperforms experienced doctors, yet only consuming roughly [Formula: see text] ( 1/48) of doctors' examination time on average. Therefore, the accuracy and efficiency of our new method exhibit its great potential in clinical applications.

Mesh-based animation, usually represented as dynamic meshes with fixed connectivity, is becoming more and more prevalent in movies, games and other graphics applications nowadays, and there is a growing need to compactly store and rapidly transmit these meshes for practical use, especially for those with high-quality geometric details. In this paper, we explore a novel key-frame based dynamic mesh compression method, wherein we apply pose-similarity with spectral techniques to define piece-wise manifold harmonic bases to reduce spatial-temporal redundancy. We first partition the sequence into several clusters with similar poses, and then decompose the meshes in each cluster into primary poses and geometric details using the manifold harmonic bases derived from the extracted key-frame in that cluster. The primary poses can be characterized as linear combinations of manifold harmonic bases, and the geometric details can be recovered by deformation transfer technique. Thus, we only need a small number of key-frames and a few coefficients for compressing dynamic meshes, which saves a significant amount of storage comparing to traditional methods in which bases are stored explicitly. Furthermore, we apply a second-order linear prediction coding to the harmonic coefficients to further reduce the temporal redundancy. Our extensive experiments and evaluations on various datasets have manifested that our novel method could obtain a high compression ratio while preserving high-fidelity geometry details and guaranteeing limited human perceived distortion rate simultaneously.

This paper proposes a semi-supervised learning method based on weakly-labeled data to automatically classify ultrasound (US) thyroid nodules. Key to our new approach is the unification of multi-instance learning (MIL) with deep learning. Benefiting from that, our method can directly use off-the-shelf clinical data, which involves no labels to indicate nodule classes. To this end, we take the US images of a patient as a bag, and take the corresponding pathology report as the bag label. Specifically, we first propose a bag generating method, wherein the detected thyroid nodules are considered as instances corresponding to certain bag. After that, we design an effective EM algorithm to train a convolutional neural network (CNN) for nodule classification. We conduct extensive experiments and comprehensive evaluations on different datasets, and all the experiments confirm that, our method significantly outperforms state-of-the-art MIL algorithms, which exhibits great potential in clinical applications.

This paper proposes a novel method for decorating 3D surfaces using a new type of vector graphics, called Poisson Vector Graphics (PVG). Unlike other existing techniques that frequently require local/global parameterization, our approach advocates a parameterization-free paradigm, affording decoration of geometric models with any topological type while minimizing the overall computational expenses. Since PVG supports a set of simple discrete curves, it is straightforward for users to edit colors and synthesize geometry details. Meanwhile, the details could be organized by Poisson Region (PR), leading to much smoother decoration than those of Diffusion Curve (DC). Consequently, it is an ideal tool to create smooth relief. It may be noted that, DC is adequate to create sharp or discontinuous results. But PR is superior to DC, supporting level-of-details editing on meshes thanks to its smoothness. To render PVG on meshes efficiently, we develop a Poisson solver based on harmonic B-splines, which could be constructed using geodesic Voronoi diagram . Our Poisson solver is a local solver for rendering with more flexibility and versatility. We demonstrate the efficacy of our approach on synthetic and real-world 3D models.

After years of extensive studies, the salient motion detection problem has gained plausible performance improvement that was primarily propelled by the rapid development of self-adaptive top-down modeling techniques. Nevertheless, almost all the conventional solutions are still not robust enough to handle video sequences captured by hand-hold cameras. This is mainly due to the absence of the position alignment information that is indispensable for top-down background modeling. In contrast, the bottom-up video saliency detection methods, though achieving excellent salient motion detection in either stationary or nonstationary videos, still have rather poor detection performance in scenarios with massive dynamic background. In this letter, we explore a bottom-up saliency framework by introducing a novel spatial-temporal regional filter method to handle the dynamic background problem. Our key rationale is to assign large saliency value to those regions with stable spatial-temporal coherency while eliminating irregular, repeating dynamic background. As far as we know, this is the first work to address the dynamic background problem from the perspective of the bottom-up video saliency. We conduct massive quantitative evaluations over public available benchmarks to validate the effectiveness and robustness of our method.

This paper advocates a novel learning solution to the modeling of long-term spatial-temporal saliency consistency in order to boost the accuracy for video saliency detection. Conventional methods typically utilize the “slack” spatial-temporal model to locally ensure the smoothness of the computed video saliency, yet they could easily encounter the performance tradeoff dilemma (i.e., detection' accuracy and integrity). In contrast, our novel approach proposes the bilevel learning strategy to globally exploit the saliency consistency while overcoming the aforementioned difficulty. Our method first starts with the contrast computation of low-level saliency clues in a frame-wise manner. Then, based on such obtained saliency clues, we devise a novel bilevel Markov Random Field (bMRF) solution to conduct semantic labelling, which can explicitly indicates both the salient salient foregrounds and nonsalient nearby surroundings with high confidence while shrinking the low confidence remains. In such a way, the spatial-temporal consistency constraint is embedded intrinsically into the above explicit semantic labels, and we prevent the performance tradeoff problem from occurring. Next, based on those semantic labels made by our bMRF method, we further propose learning multiple nonlinear feature transformations to enlarge the feature margin between the salient foregrounds and the non-salient nearby surroundings, whose key rationale is to resort to long-term common consistencies to enforce the spatial-temporal smoothness. Thus, we can utilize these learned non-linear feature transformations to simultaneously suppress those short-term false-alarms and correct those hollow effects. To validate our new approach, we conduct extensive experiments on five publicly available benchmarks, and make comprehensive, quantitative evaluations between our method and 17 state-of-the-art techniques. All of the results demonstrate our method's advantages in terms of accuracy, reliability, robustness, and versatility.

Cell detection and counting are critical and essential tasks for many biological and clinical studies. Traditionally, these tasks are usually performed by visual inspection, which is time consuming and prone to induce subjective bias. These make automatic cell counting and detection essential for large- scale and objective studies. Unfortunately, the hard examples such as cell blur, clutter, bleed-through and imaging noise make these tasks extremely challenging. Over the last few years, automatic cell detection and counting have evolved from earlier methods that are often based on filters to the current state-of- the-art deep learning methods. In this paper, we propose a novel efficient method for robust counting and detection task based on fully convolution networks (FCN). Our method is able to handle most of detection and counting problems from different kinds of cell datasets, and can cover most senior microscopy images, such as bright field, pathology stained material and electron. Extensive experiments on the public and private datasets demonstrate the effectiveness and reliability of our approach.

Physically-based fluid animation and solid deformation driven by numerical simulation have manifested their significance for many graphics applications during the past two decades. For example, the fluid implicit particle (FLIP) method and shape matching technique based on position based dynamics (PBD) have demonstrated their unique graphics strength in fluid and solid animation, respectively. We propose a novel integrated approach supporting the seamless unification of FLIP and shape matching. We devise new algorithms to tackle existing difficulties when handling new phenomena such as high-fidelity fluid-solid interaction and solid melting. The key innovation of this paper is a unified Lagrangian framework that seamlessly blends FLIP and PBD based shape matching constraint towards the natural yet strong coupling between fluid and deformable solid. Within our integrated framework, it enables many complicated fluid-solid phenomena with ease. We conduct various kinds of experiments. All the results demonstrate the advantages of our unified hybrid approach towards visual fidelity, efficiency, stability, and versatility.

Nationality identification unlocks important demographic information, with many applications in biomedical and sociological research. Existing name-based nationality classifiers use name substrings as features and are trained on small, unrepresentative sets of labeled names, typically extracted from Wikipedia. As a result, these methods achieve limited performance and cannot support fine-grained classification.

Accurate motion capture and flexible retargeting of underwater creatures such as fish remain to be difficult due to the long-lasting challenges of marker attachment and feature description for soft bodies in the underwater environment. Despite limited new research progresses appeared in recent years, the fish motion retargeting with a desirable motion pattern in real-time remains elusive. Strongly motivated by our ambitious goal of achieving high-quality data-driven fish animation with a light-weight, mobile device, this paper develops a novel framework of motion capturing and retargeting for a fish. We capture the motion of actual fish by a monocular camera without the utility of any marker. The elliptical Fourier coefficients are then integrated into the contour-based feature extraction process to analyze the fish swimming patterns. This novel approach can obtain the motion information in a robust way, with smooth medial axis as the descriptor for a soft fish body. For motion retargeting, we propose a two-level scheme to properly transfer the captured motion into new models, such as 2D meshes (with texture) generated from pictures or 3D models designed by artists, regardless of different body geometry and fin proportions among various species. Both motion capture and retargeting processes are functioning in real time. Hence, the system can simultaneously create fish animation with variation, while obtaining video sequences of real fish by a monocular camera.

Abstract This paper advocates a novel method for modelling physically realistic flow from captured incompressible gas sequence via modal analysis in frequency‐constrained subspace. Our analytical tool is uniquely founded upon empirical mode decomposition (EMD) and modal reduction for fluids, which are seamlessly integrated towards a powerful, style‐controllable flow modelling approach. We first extend EMD, which is capable of processing 1D time series but has shown inadequacies for 3D graphics earlier, to fit gas flows in 3D. Next, frequency components from EMD are adopted as candidate vectors for bases of modal reduction. The prerequisite parameters of the Navier–Stokes equations are then optimized to inversely model the physically realistic flow in the frequency‐constrained subspace. The estimated parameters can be utilized for re‐simulation, or be altered toward fluid editing. Our novel inverse‐modelling technique produces real‐time gas sequences after precomputation, and is convenient to couple with other methods for visual enhancement and/or special visual effects. We integrate our new modelling tool with a state‐of‐the‐art fluid capturing approach, forming a complete pipeline from real‐world fluid to flow re‐simulation and editing for various graphics applications.

Abstract In this paper, we propose a novel integrated method for effective modeling and realistic enhancement of scale‐sensitive fluid simulation details. The core of our method is the organic of multi‐layer depth image regression analysis and fluid implicit particle fluid simulation of which the regression analysis induces the criterion where the fluid details should be produced. First, we capture the depth buffer of the fluid surface dynamically from the top of scene. Second, we employ depth peeling technique to decompose the target fluid volume into multiple depth layers and conduct time‐space analysis over surface layers. Third, we propose a logistic regression‐based model to rigorously pinpoint the complex interacting regions, wherein multiple detail‐relevant factors are taken into account based on the captured multiple depth layers. Finally, details are enhanced by animating extra diffuse materials and augmenting the air‐fluid mixing phenomenon. It is evident that, with depth peeling technology, we can afford rigorous analysis not only across surface layers at different fluid depth but along the depth direction as well. After integrating the analysis results from these two sources, we are capable of performing detail enhancement both on the fluid surface and inside the fluid to obtain a great visual effect, even when large occlusion exists. Directly benefiting from the flexibility of image‐space‐dominant processing, our unified framework can be entirely implemented on graphics processing units and thus achieves interactive performance. For various fluid phenomena with different diffuse materials (e.g., spray, foam, and bubble), comprehensive experiments and evaluations have demonstrated its superiority in high‐fidelity fluid detail enhancement and its interaction with surrounding environment.

Abstract Many complex natural phenomena with dramatic spatial and temporal variation are difficult to animate accurately with anticipated performance in many graphics tasks and applications, because oftentimes in prior art, a single type of physical process could not afford high fidelity and effective scene production. Volcano eruption and its subsequent interaction with earth is one such complicated phenomenon that must depend on multiphysical processes and their tight coupling. This paper documents a novel and effective particle‐based solution for volcano animation that embraces multiphysical processes and their tight unification. First, we introduce a governing physical model consisting of multiphysical processes enabling flexible state transition among solid, fluid, and gas. This computational physics model is dictated by temperature and accommodates dynamic viscosity that is changing according to the temperature. Second, we propose an augmented smoothed particle hydrodynamics as the underlying numerical model to simulate the behavior of lava and smoke with several required physical attributes. Third, multiphysical quantities are tightly coupled to support the interaction with surroundings including fluid–solid coupling, ground friction, and lava–smoke coupling. We also develop a temperature‐directed rendering technique with nearly no extra computational cost and demonstrate realistic graphics effects of volcano eruption and its interaction with earth with visual appeal.

This paper advocates a novel video saliency detection method based on the spatial-temporal saliency fusion and low-rank coherency guided saliency diffusion. In sharp contrast to the conventional methods, which conduct saliency detection locally in a frame-by-frame way and could easily give rise to incorrect low-level saliency map, in order to overcome the existing difficulties, this paper proposes to fuse the color saliency based on global motion clues in a batch-wise fashion. And we also propose low-rank coherency guided spatial-temporal saliency diffusion to guarantee the temporal smoothness of saliency maps. Meanwhile, a series of saliency boosting strategies are designed to further improve the saliency accuracy. First, the original long-term video sequence is equally segmented into many short-term frame batches, and the motion clues of the individual video batch are integrated and diffused temporally to facilitate the computation of color saliency. Then, based on the obtained saliency clues, inter-batch saliency priors are modeled to guide the low-level saliency fusion. After that, both the raw color information and the fused low-level saliency are regarded as the low-rank coherency clues, which are employed to guide the spatial-temporal saliency diffusion with the help of an additional permutation matrix serving as the alternative rank selection strategy. Thus, it could guarantee the robustness of the saliency map's temporal consistence, and further boost the accuracy of the computed saliency map. Moreover, we conduct extensive experiments on five public available benchmarks, and make comprehensive, quantitative evaluations between our method and 16 state-of-the-art techniques. All the results demonstrate the superiority of our method in accuracy, reliability, robustness, and versatility.

This paper systematically advocates a robust and efficient unsupervised multi-class co-segmentation approach by leveraging underlying subspace manifold propagation to exploit the cross-image coherency. It can combat certain image co-segmentation difficulties due to viewpoint change, partial occlusion, complex background, transient illumination, and cluttering texture patterns. Our key idea is to construct a powerful hyper-graph joint-cut framework, which incorporates mid-level image regions-based intra-image feature representation and L1-manifold graph-based inter-image coherency exploration. For local image region generation, we propose a bi-harmonic distance distribution difference metric to govern the super-pixel clustering in a bottom-up way. It not only affords drastic data reduction but also gives rise to discriminative and structure meaningful feature representation. As for the inter-image coherency, we leverage multi-type features involved L1-graph to detect the underlying local manifold from cross-image regions. As a result, the implicit supervising information could be encoded into the unsupervised hyper-graph joint-cut framework. We conduct extensive experiments and make comprehensive evaluations with other state-of-the-art methods over various benchmarks, including iCoseg, MSRC, and Oxford flower. All the results demonstrate the superiorities of our method in terms of accuracy, robustness, efficiency, and versatility.

Biharmonic B-splines, proposed by Feng and Warren, are an elegant generalization of univariate B-splines to planar and curved domains with fully irregular knot configuration. Despite the theoretic breakthrough, certain technical difficulties are imperative, including the necessity of Voronoi tessellation, the lack of analytical formulation of bases on general manifolds, expensive basis re-computation during knot refinement/removal, being applicable for simple domains only (e.g., such as euclidean planes, spherical and cylindrical domains, and tori). To ameliorate, this paper articulates a new biharmonic B-spline computing paradigm with a simple formulation. We prove that biharmonic B-splines have an equivalent representation, which is solely based on a linear combination of Green's functions of the bi-Laplacian operator. Consequently, without explicitly computing their bases, biharmonic B-splines can bypass the Voronoi partitioning and the discretization of bi-Laplacian, enable the computational utilities on any compact 2-manifold. The new representation also facilitates optimization-driven knot selection for constructing biharmonic B-splines on manifold triangle meshes. We develop algorithms for spline evaluation, data interpolation and hierarchical data decomposition. Our results demonstrate that biharmonic B-splines, as a new type of spline functions with theoretic and application appeal, afford progressive update of fully irregular knots, free of singularity, without the need of explicit parameterization, making it ideal for a host of graphics tasks on manifolds.

In graph-oriented machine learning research, L1 graph is an efficient way to represent the connections of input data samples. Its construction algorithm is based on a numerical optimization motivated by Compressive Sensing theory. As a result, It is a nonparametric method which is highly demanded. However, the information of data such as geometry structure and density distribution are ignored. In this paper, we propose a Structure Aware (SA) L1 graph to improve the data clustering performance by capturing the manifold structure of input data. We use a local dictionary for each datum while calculating its sparse coefficients. SA-L1 graph not only preserves the locality of data but also captures the geometry structure of data. The experimental results show that our new algorithm has better clustering performance than L1 graph.

Massive routinely-acquired raw volumetric datasets are hard to be deeply exploited by cyber worlds related downstream applications due to the challenges in accurate and efficient shape modeling. This paper systematically advocates an interactive 3D shape modeling framework for raw volumetric datasets by iteratively optimizing Hessian-constrained local implicit surfaces. The key idea is to incorporate contour based interactive segmentation into the generalized local implicit surface reconstruction. Our framework allows a user to flexibly define derivative constraints up to the second order via intuitively placing contours on the cross sections of volumetric images and fine-tuning the eigenvector frame of Hessian matrix. It enables detail-preserving local implicit representation while combating certain difficulties due to ambiguous image regions, low-quality irregular data, close sheets, and massive coefficients involved extra computing burden. Moreover, we conduct extensive experiments on some volumetric images with blurry object boundaries, and make comprehensive, quantitative performance evaluation between our method and the state-of-the-art radial basis function based techniques. All the results demonstrate our method's advantages in the accuracy, detail-preserving, efficiency, and versatility of shape modeling.

Abstract Because of intensive inter‐node communications, image compositing has always been a bottleneck in parallel visualization systems. In a heterogeneous networking environment, the variation of link bandwidth and latency adds more uncertainty to the system performance. In this paper, we present a pipelining image compositing algorithm in heterogeneous networking environments, which is able to rearrange the direction of data flow of a compositing pipeline under strict ordering constraint. We introduce a novel directional image compositing operator that specifies not only the color and α channels of the output but also the direction of data flow when performing compositing. Based on this new operator, we thoroughly study the properties of image compositing pipelines in heterogeneous environments. We develop an optimization algorithm that could find the optimal pipeline from an exponentially large searching space in polynomial time. We conducted a comprehensive evaluation on the ns‐3 network simulator. Experimental results demonstrate the efficiency of our method. Copyright © 2016 John Wiley & Sons, Ltd.

There is growing demand for accuracy in image processing and visualization, and the super-resolution (SR) technique for multi-observed RGB-D images has become popular, because it provides space-redundant information and produces a detailed reconstruction even with a large magnification factor. This technique has been thoroughly investigated in recent years. Nevertheless, technical challenges remain, such as finding sub-pixel correspondences with low-resolution (LR) observations, exploiting space-redundant information, formulating space homogeneity constraints, and leveraging cross-image similarities in structures. To address these challenges, this paper proposes a unified optimization framework to estimate both the super-resolved RGB image and the super-resolved depth image from the multi-observed LR RGB-D images using their correlations. Using depth-assisted cross-image correspondences, the RGB image SR problem is formulated as an effective regularization function by incorporating the normalized bilateral total variation regularizer, and it is efficiently solved by a first-order primal-dual algorithm. The depth image SR estimate can be obtained by minimizing a nonlocal regression-based energy, which integrates the structural cues of the super-resolved RGB image in a detail-preserving fashion. Essentially, our unified optimization framework uses the RGB image and depth image as a priori knowledge that the SR process uses for better accuracy. Our extensive experiments on public RGB-D benchmarks and real data and our quantitative comparison with several state-of-the-art methods demonstrate the superiority of our method in terms of accuracy, versatility, and reliability of details and sharp feature preservation.

In this paper, we present an efficient robust labeling method for coronary arteries from X-ray angiograms based on energy optimization. The fundamental goal of this research is to facilitate the analysis and diagnosis of interventional surgery in the most efficient way, and such effort could also improve the performance during doctor training, and surgery simulation and planning. Compared to the prior state-of-the-art, our method is much more robust to resist noises and is tolerant to even incomplete data because of the "built-in" nature of global optimization. We start with a fully parallelized algorithm based on Hessian matrix to extract the tubular structure from the X-ray angiograms as vessel candidates. Then, instead of using the candidates directly, we use the grow cut (Vezhnevets and V. Konouchine, Growcut: Interactive multi-label N-D image segmentation by cellular automata, in Proc. of Graphicon, 2005, pp. 150-156.) method, which is similar to graph cut (Boykov et al. , Fast approximate energy minimization via graph cuts, IEEE Trans. Pattern Anal. Mach. Intell. , vol. 23, no. 11, pp. 1222-1239, Nov. 2001.)but with better performance to extract the precise vessel structure from the images. Next, we use the fast marching method with second derivatives and cross neighbors to extract the accurate skeleton segments. After that, we propose an efficient method based on iterative closest point (Z. Zhang, Iterative point matching for registration of free-form curves and surfaces, Int J. Comput. Vis., vol. 13, no. 2, pp. 119-152, 1994.) to organize the skeleton segments by treating the continuity and similarity as extra constraints. Finally, we formulate the vessel labeling problem as an energy optimization problem and solve it using belief propagation. We also demonstrate several typical applications including flow velocity estimation, heart beat estimation, and vessel diameter estimation to show its practical uses in clinical diagnosis and treatment. Our experiments exhibit the correctness and robustness, as well as the high performance of our algorithm. We envision that our system would be of high utility for diagnosis and therapy to treat vessel-related diseases in a clinical setting in the near future.

Manifold learning, especially spectral embedding, is known as one of the most effective learning approaches on high dimensional data, but for real-world applications it raises a serious computational burden in constructing spectral embeddings for large datasets. To overcome this computational complexity, we propose a novel efficient embedding construction, Diverse Power Iteration Embedding (DPIE). DPIE shows almost the same effectiveness of spectral embeddings and yet is three order of magnitude faster than spectral embeddings computed from eigen-decomposition. Our DPIE is unique in that (1) it finds linearly independent embeddings and thus shows diverse aspects of dataset; (2) the proposed regularized DPIE is effective if we need many embeddings; (3) we show how to efficiently orthogonalize DPIE if one needs; and (4) Diverse Power Iteration Value (DPIV) provides the importance of each DPIE like an eigen value. Such various aspects of DPIE and DPIV ensure that our algorithm is easy to apply to various applications, and we also show the effectiveness and efficiency of DPIE on clustering, anomaly detection, and feature selection as our case studies.

Informative and discriminative feature descriptors are vital in qualitative and quantitative shape analysis for a large variety of graphics applications. Conventional feature descriptors primarily concentrate on discontinuity of certain differential attributes at different orders that naturally give rise to their discriminative power in depicting point, line, small patch features, etc. This paper seeks novel strategies to define generalized, user-specified features anywhere on shapes. Our new region-based feature descriptors are constructed primarily with the powerful spectral graph wavelets (SGWs) that are both multi-scale and multi-level in nature, incorporating both local (differential) and global (integral) information. To our best knowledge, this is the first attempt to organize SGWs in a hierarchical way and unite them with the bi-harmonic diffusion field towards quantitative region-based shape analysis. Furthermore, we develop a local-to-global shape feature detection framework to facilitate a host of graphics applications, including partial matching without point-wise correspondence, coarse-to-fine recognition, model recognition, etc. Through the extensive experiments and comprehensive comparisons with the state-of-the-art, our framework has exhibited many attractive advantages such as being geometry-aware, robust, discriminative, isometry-invariant, etc.

Despite the long R&D history of interactive minimally-invasive surgery and therapy simulations, the guide wire/catheter behavior modeling remains challenging in Percutaneous Coronary Intervention (PCI) surgery simulators. This is primarily due to the heterogeneous heart physiological structures and complex intravascular inter-dynamic procedures. To ameliorate, this paper advocates a novel, robust, and efficient guide wire/catheter modeling method based on heterogeneous and integrated chain-mails, that can afford medical practitioners and trainees the unique opportunity to experience the entire guide wire-dominant PCI procedures in virtual environments as our model aims to mimic what occurs in clinical settings. Our approach's originality is primarily founded upon this new method's unconditional stability, real time performance, flexibility, and high-fidelity realism for guide wire/catheter simulation. Considering the front end of the guide wire has different stiffness with its conjunctive slender body and the guide wire length is adaptive to the surrounding environment, we propose to model the spatially-varying six-degree of freedom behaviors by solely resorting to the generalized 3D chain-mails. Meanwhile, to effectively accommodate the motion constraints caused by the beating vessels and flowing blood, we integrate heterogeneous volumetric chain mails to streamline guide wire modeling and its interaction with surrounding substances. By dynamically coupling guide wire chain-mails with the surrounding media via virtual links, we are capable of efficiently simulating the collision-involved interdynamic behaviors of the guide wire. Finally, we showcase a PCI prototype simulator equipped with hap tic feedback for mimicing the guide wire intervention therapy, including pushing, pulling, and twisting operations, where the built-in high-fidelity, real-time efficiency, and stableness show great promise for its practical applications in clinical training and surgery rehearsal fields.

A great number of volumetric datasets have been routinely acquired everyday and their qualities are varying tremendously, without proper processing they could not be directly utilized. Specifically, volumetric segmentation plays a vital role in many downstream applications, including geometric modeling, scientific visualization, and medical diagnosis. So far, many volume segmentation methods have been proposed, Top et al. [2011] designed an interactive segmentation tool by interactively contouring on some sparse slices and Ijiri et al. [2013] developed a system to extract contours and evaluate the scalar field in spatial domain.

This paper advocates a novel integrated method to tightly couple simulation with analysis for the effective modeling and enhancement of scale-aware fluid details. It brings forth a suite of innovations in a unified framework, including depth-image-based space analysis for multi-scale detail detection, time-space analysis based on the logistic regression model that integrates both geometry and physics criteria, and depth-image-based sampling for quality-efficiency tradeoff. Our method contains an intertwined two-level processing architecture at its core. At the analysis level, we propose a rigorous time-space analysis model to pinpoint complex interacting regions, which can take into account multiple detail-relevant factors based on the depth-image sequence captured from FLIP-driven simulation sequence. At the simulation level, details are enhanced by animating extra diffuse materials, and augmenting the air-fluid mixing phenomenon. Directly benefitting from the flexibility of image-space-dominant processing, our unified framework can be entirely implemented on GPU, hence interactive performance could be guaranteed. Comprehensive experiments and evaluations on various diffuse phenomena (e.g., spray, foam, and bubble) have demonstrated its superiority in high-fidelity detail enhancement during fluid simulation and its interaction with surrounding environment for VR applications.

Abstract In this paper, we formulate a novel trivariate biharmonic B‐spline defined over bounded volumetric domain. The properties of bi‐Laplacian have been well investigated, but the straightforward generalization from bivariate case to trivariate one gives rise to unsatisfactory discretization, due to the dramatically uneven distribution of neighbouring knots in 3D. To ameliorate, our original idea is to extend the bivariate biharmonic B‐spline to the trivariate one with novel formulations based on quadratic programming, approximating the properties of localization and partition of unity. And we design a novel discrete biharmonic operator which is optimized more robustly for a specific set of functions for unevenly sampled knots compared with previous methods. Our experiments demonstrate that our 3D discrete biharmonic operators are robust for unevenly distributed knots and illustrate that our algorithm is superior to previous algorithms .

Abstract This paper systematically describes an interactive dissection approach for hybrid soft tissue models governed by extended position‐based dynamics. Our framework makes use of a hybrid geometric model comprising both surface and volumetric meshes. The fine surface triangular mesh with high‐precision geometric structure and texture at the detailed level is employed to represent the exterior structure of soft tissue models. Meanwhile, the interior structure of soft tissues is constructed by coarser tetrahedral mesh, which is also employed as physical model participating in dynamic simulation. The less details of interior structure can effectively reduce the computational cost during simulation. For physical deformation, we design and implement an extended position‐based dynamics approach that supports topology modification and material heterogeneities of soft tissue. Besides stretching and volume conservation constraints, it enforces the energy preserving constraints, which take the different spring stiffness of material into account and improve the visual performance of soft tissue deformation. Furthermore, we develop mechanical modeling of dissection behavior and analyze the system stability. The experimental results have shown that our approach affords real‐time and robust cutting without sacrificing realistic visual performance. Our novel dissection technique has already been integrated into a virtual reality‐based laparoscopic surgery simulator. Copyright © 2015 John Wiley & Sons, Ltd.

This paper advocates a novel multiscale, structure-sensitive saliency detection method, which can distinguish multilevel, reliable saliency from various natural pictures in a robust and versatile way. One key challenge for saliency detection is to guarantee the entire salient object being characterized differently from nonsalient background. To tackle this, our strategy is to design a structure-aware descriptor based on the intrinsic biharmonic distance metric. One benefit of introducing this descriptor is its ability to simultaneously integrate local and global structure information, which is extremely valuable for separating the salient object from nonsalient background in a multiscale sense. Upon devising such powerful shape descriptor, the remaining challenge is to capture the saliency to make sure that salient subparts actually stand out among all possible candidates. Toward this goal, we conduct multilevel low-rank and sparse analysis in the intrinsic feature space spanned by the shape descriptors defined on over-segmented super-pixels. Since the low-rank property emphasizes much more on stronger similarities among super-pixels, we naturally obtain a scale space along the rank dimension in this way. Multiscale saliency can be obtained by simply computing differences among the low-rank components across the rank scale. We conduct extensive experiments on some public benchmarks, and make comprehensive, quantitative evaluation between our method and existing state-of-the-art techniques. All the results demonstrate the superiority of our method in accuracy, reliability, robustness, and versatility.

Current spectral clustering algorithms suffer from the sensitivity to existing noise and parameter scaling and may not be aware of different density distributions across clusters. If these problems are left untreated, the consequent clustering results cannot accurately represent true data patterns, in particular, for complex real-world datasets with heterogeneous densities. This article aims to solve these problems by proposing a diffusion-based Aggregated Heat Kernel (AHK) to improve the clustering stability, and a Local Density Affinity Transformation (LDAT) to correct the bias originating from different cluster densities. AHK statistically models the heat diffusion traces along the entire time scale, so it ensures robustness during the clustering process, while LDAT probabilistically reveals the local density of each instance and suppresses the local density bias in the affinity matrix. Our proposed framework integrates these two techniques systematically. As a result, it not only provides an advanced noise-resisting and density-aware spectral mapping to the original dataset but also demonstrates the stability during the processing of tuning the scaling parameter (which usually controls the range of neighborhood). Furthermore, our framework works well with the majority of similarity kernels, which ensures its applicability to many types of data and problem domains. The systematic experiments on different applications show that our proposed algorithm outperforms state-of-the-art clustering algorithms for the data with heterogeneous density distributions and achieves robust clustering performance with respect to tuning the scaling parameter and handling various levels and types of noise.

Ratio-based algorithms are proven to be effective methods for removing batch effects that exist among micro array expression data from different data sources. They are outperforming than other methods in the enhancement of cross-batch prediction, especially for cancer data sets. However, their overall power is limited by: (1) Not every batch has control samples. The original method uses all negative samples to calculate the subtrahend. (2) Micro array experimental data may not have clear labels, especially in the prediction application, the labels of test data set are unknown. In this paper, we propose an Improved Ratio-Based (IRB) method to relieve these two constraints for cross-batch prediction applications. For each batch in a single study, we select one reference sample based on the idea of aligning probability density functions (pdfs) of each gene in different batches. Moreover, for data sets without label information, we transfer the problem of finding reference sample to the dense sub graph problem in graph theory. Our newly-proposed IRB method is straightforward and efficient, and can be extended for integrating large volume micro array data sets. The experiments show that our method is stable and has high performance in tumor/non-tumor prediction.

Spectral Embedding is one of the most effective dimension reduction algorithms in data mining. However, its computation complexity has to be mitigated in order to apply it for real-world large scale data analysis. Many researches have been focusing on developing approximate spectral embeddings which are more efficient, but meanwhile far less effective. This paper proposes Diverse Power Iteration Embeddings (DPIE), which not only retains the similar efficiency of power iteration methods but also produces a series of diverse and more effective embedding vectors. We test this novel method by applying it to various data mining applications (e.g. Clustering, anomaly detection and feature selection) and evaluating their performance improvements. The experimental results show our proposed DPIE is more effective than popular spectral approximation methods, and obtains the similar quality of classic spectral embedding derived from eigen-decompositions. Moreover it is extremely fast on big data applications. For example in terms of clustering result, DPIE achieves as good as 95% of classic spectral clustering on the complex datasets but 4000+ times faster in limited memory environment.

Unsupervised feature selection is an important issue for high dimensional dataset analysis. However popular methods are susceptible to noisy instances (observations) or noisy features. We propose a noise-resistant feature selection algorithm by capturing multi-perspective correlations. Our proposed approach, called Noise-Resistant Unsupervised Feature Selection (NRFS), is based on multi-perspective correlation that reflects the importance of feature with respect to noise-resistant representative instances and various global trends from spectral decomposition. In this way, the model concisely captures a wide variety of local patterns. Experimental results demonstrate the effectiveness of our algorithm.

This paper describes an interactive dissection approach for hybrid soft tissue models governed by position-based dynamics. Our framework makes use of a hybrid geometric model comprising both surface and volumetric meshes. The fine surface triangular mesh is used to represent the exterior structure of soft tissue models. Meanwhile, the interior structure of soft tissues is constructed by coarser tetrahedral meshes, which are also employed as physical models participating in dynamic simulation. The less details of interior structure can effectively reduce the computational cost of deformation and geometric subdivision during dissection. For physical deformation, we design and implement a position-based dynamics approach that supports topology modification and enforces the volume-preserving constraint. Experimental results have shown that, this hybrid dissection method affords real-time and robust cutting simulation without sacrificing realistic visual performance.

Abstract This paper presents a novel hybrid particle‐grid method that tightly couples Lagrangian particle approach with Eulerian grid approach to simulate multi‐scale diffuse materials varying from disperse droplets to dissipating spray and their natural mixture and transition, originated from a violent (high‐speed) liquid stream. Despite the fact that Lagrangian particles are widely employed for representing individual droplets and Eulerian grid‐based method is ideal for volumetric spray modeling, using either one alone has encountered tremendous difficulties when effectively simulating droplet/spray mixture phenomena with high fidelity. To ameliorate, we propose a new hybrid model to tackle such challenges with many novel technical elements. At the geometric level, we employ the particle and density field to represent droplet and spray respectively, modeling their creation from liquid as well as their seamless transition. At the physical level, we introduce a drag force model to couple droplets and spray, and specifically, we employ Eulerian method to model the interaction among droplets and marry it with the widely‐used Lagrangian model. Moreover, we implement our entire hybrid model on CUDA to guarantee the interactive performance for high‐effective physics‐based graphics applications. The comprehensive experiments have shown that our hybrid approach takes advantages of both particle and grid methods, with convincing graphics effects for disperse droplets and spray simulation.

ABSTRACT This paper systematically advocates an interactive volumetric image manipulation framework, which can enable the rapid deployment and instant utility of patient‐specific medical images in virtual surgery simulation while requiring little user involvement. We seamlessly integrate multiple technical elements to synchronously accommodate physics‐plausible simulation and high‐fidelity anatomical structures visualization. Given a volumetric image, in a user‐transparent way, we build a proxy to represent the geometrical structure and encode its physical state without the need of explicit 3‐D reconstruction. On the basis of the dynamic update of the proxy, we simulate large‐scale deformation, arbitrary cutting, and accompanying collision response driven by a non‐linear finite element method. By resorting to the upsampling of the sparse displacement field resulted from non‐linear finite element simulation, the cut/deformed volumetric image can evolve naturally and serves as a time‐varying 3‐D texture to expedite direct volume rendering. Moreover, our entire framework is built upon CUDA (Beihang University, Beijing, China) and thus can achieve interactive performance even on a commodity laptop. The implementation details, timing statistics, and physical behavior measurements have shown its practicality, efficiency, and robustness. Copyright © 2013 John Wiley & Sons, Ltd.

ABSTRACT This paper advocates a method for real‐time physical deformation and arbitrary cutting simulation of heterogeneous objects with multi‐material distribution, whose originality centers on the tight coupling of domain‐specific finite element method (FEM) and material distance‐aware meshless approach in a CUDA‐centric parallel simulation framework. We employ hierarchical hexahedron serving as basic building blocks for accurate material‐aware FEM simulation. Meanwhile, local meshless systems are designed to support cross‐FEM‐domain coupling and material‐sensitive propagation while respecting the regularity of finite elements. Directly benefiting from the structural regularity and uniformity of finite elements, our hybrid solution enables the local stiffness matrix pre‐computation and dynamic assembling, adaptive topological updating and precise cutting reconstruction. Moreover, our mathematically‐rigorous solver guarantees unconditional stableness. Experiments demonstrate the superiorities of our system. Copyright © 2014 John Wiley & Sons, Ltd.

Current popular anomaly detection algorithms are capable of detecting global anomalies but often fail to distinguish local anomalies from normal instances. Inspired by contemporary physics theory (i.e., heat diffusion and quantum mechanics), we propose two unsupervised anomaly detection algorithms. Building on the embedding manifold derived from heat diffusion, we devise Local Anomaly Descriptor (LAD), which faithfully reveals the intrinsic neighborhood density. It uses a scale-dependent umbrella operator to bridge global and local properties, which makes LAD more informative within an adaptive scope of neighborhood. To offer more stability of local density measurement on scaling parameter tuning, we formulate Fermi Density Descriptor (FDD), which measures the probability of a fermion particle being at a specific location. By choosing the stable energy distribution function, FDD steadily distinguishes anomalies from normal instances with any scaling parameter setting. To further enhance the efficacy of our proposed algorithms, we explore the utility of anisotropic Gaussian kernel (AGK), which offers better manifold-aware affinity information. We also quantify and examine the effect of different Laplacian normalizations for anomaly detection. Comprehensive experiments on both synthetic and benchmark datasets verify that our proposed algorithms outperform the existing anomaly detection algorithms.

In this paper, we present an efficient 3D vessels reconstruction algorithm based on multi-views of X-ray Angiography assisting interventional surgery. First, we extract the vascular-like structures from the image sequences using a geometrical analysis of multi-scale Hessian matrix eigen-system and use the fast marching method to extract the skeleton of the structure, from which we derive the vascular topological configurations. Second, we regard the 3D space as a Markov Random Field and formulate the reconstruction problem as an energy minimization problem with consistent, continuous and topological constraints to coarsely register and reconstruct the 3D vessels. Third, we refine the reconstructed vessels to register and reconstruct the 3D vessels accurately. We demonstrate our system in coronary arteries reconstruction for percutaneous coronary intervention surgery to help doctors learn about the configurations of the coronary arteries of specific patient during operation. We envision that our system will be used for clinic treatment to advance vessel reconstruction for diagnosis and therapy in the near future.

This paper proposes a consolidation method for scanned point clouds that are usually corrupted by noises, outliers, and thickness. At the beginning, we construct neighborhood of a point based on shared nearest neighbor relationship. Then, the points with few number of neighbors are regarded as outliers and removed. After that, we propose a feature-aware projection operator to thin the thick point clouds by considering spatial distances, normal diversifications, and the squash directions of thick point clouds. Experiment results of scanned point clouds show that our method can consolidate the thick point clouds while preserving sharp features and geometry details.

This paper proposes a parallel and direct semantic feature curve extraction method from 3D volume image for vectorization and illustration. Our approach is motivated by reconstructing 3D geometric information from multiple rendered images under multi-view in computer vision. The 2D rendered images are rich in the visual sense by color and opacity that convey the structure of volume data, so it is significant for the user to understand the structure of 3D volume data better if we can recover feature curves from those 2D images. Compared with conventional line extraction methods, which mainly focus on extracting feature curves from iso-surfaces in object space, we extract feature curves directly from volume images. Most of the computation can be computed in parallel on GPU with CUDA acceleration.

Most of Focus Context visualization methods differentiate the magnification unit only by simply assigning each voxel/cell with an importance value while ignoring the shape content embedded in the volume data. In this paper, we take the volumetric structure information as important cue to facilitate Focus Context visualization, which can homogeneously or non-homogeneously scale the volume data in a structure-sensitive way.

This paper articulates a novel method for the heterogeneous feature extraction and classification directly on volumetric images, which covers multi-scale point feature, multi-scale surface feature, multi-level curve feature, and blob feature. To tackle the challenge of complex volumetric inner structure and diverse feature forms, our technical solution hinges upon the integrated approach of locally-defined diffusion tensor (DT), DT-based anisotropic convolution kernel (DACK), DACK-based multi-scale analysis, and DT-governed curve feature growing. The extracted structural features can be further semantically classified. At the computational fronts, we design CUDA-based algorithm to conduct parallel computation for time consuming tasks. Various experiments and timing tests demonstrate the effectiveness, robustness, and high performance of our method.

Despite the recent success of extensive co-segmentation studies, they still suffer from limitations in accommodating multiple-foreground, large-scale, high-variability image set, as well as their underlying capability for parallel implementation. To improve, this paper proposes a bi-harmonic distance governed flexible method for the robust coherent segmentation of the overlapping/similar contents co-existing in image group, which is independent of supervised learning and any other user-specified prior. The central idea is the novel integration of bi-harmonic distance metric design and multi-level deformable graph generation for multi-level clustering, which gives rise to a host of unique advantages: accommodating multiple-foreground images, respecting both local structures and global semantics of images, being more robust and accurate, and being convenient for parallel acceleration. Critical pipeline of our method involves intrinsic content-coherent measuring, super-pixel assisted bottom-up clustering, and multi-level deformable graph clustering based cross-image optimization. We conduct extensive experiments on the iCoseg benchmark and Oxford flower datasets, and make comprehensive evaluations to demonstrate the superiority of our method via comparison with state-of-the-art methods collected in the MSRC database.

Real-time and realistic physics-based simulation of deformable objects is of great value to medical intervention, training, and planning in virtual environments. This paper advocates a virtual-reality (VR) approach to minimally-invasive surgery/therapy (e.g., percutaneous coronary intervention) in medical procedures. In particular, we devise a robust and accurate physics-based modeling and simulation algorithm for the guidewire interaction with blood vessels. We also showcase a VR-based prototype system for simulating percutaneous coronary intervention and mimicing the intervention therapy, which affords the utility of flexible, slender guidewires to advance diagnostic or therapeutic catheters into a patient's vascular anatomy, supporting various real-world interaction tasks. The slender body of guidewires are modeled using the famous Cosserat theory of elastic rods. We derive the equations of motion for guidewires with continuous energies and integrate them with the implicit Euler solver, that guarantees robustness and stability. Our approach's originality is primarily founded upon its power, flexibility, and versatility when interacting with the surrounding environment, including novel strategies in the hybrid of geometry and physics, material variability, dynamic sampling, constraint handling and energy-driven physical responses. Our experimental results have shown that this prototype system is both stable and efficient with real-time performance. In the long run, our algorithm and system are expected to contribute to interactive VR-based procedure training and treatment planning.

Abstract We present a novel methodology that utilizes four‐dimensional (4D) space deformation to simulate a magnification lens on versatile volume datasets and textured solid models. Compared with other magnification methods (e.g. geometric optics, mesh editing), 4D differential geometry theory and its practices are much more flexible and powerful for preserving shape features (i.e. minimizing angle distortion), and easier to adapt to versatile solid models. The primary advantage of 4D space lies at the following fact: we can now easily magnify the volume of regions of interest (ROIs) from the additional dimension, while keeping the rest region unchanged. To achieve this primary goal, we first embed a 3D volumetric input into 4D space and magnify ROIs in the fourth dimension. Then we flatten the 4D shape back into 3D space to accommodate other typical applications in the real 3D world. In order to enforce distortion minimization, in both steps we devise the high‐dimensional geometry techniques based on rigorous 4D geometry theory for 3D/4D mapping back and forth to amend the distortion. Our system can preserve not only focus region, but also context region and global shape. We demonstrate the effectiveness, robustness and efficacy of our framework with a variety of models ranging from tetrahedral meshes to volume datasets.

ABSTRACT This paper presents a hybrid animation approach to the flexible and rapid crack simulation of brittle material. At the physical level, the local stress tensors induced by collision are analyzed by using the smoothed particle hydrodynamics (SPH) formulation. Specifically, in order to determine the internal stress when rigid bodies collide with each other or neighboring environments, we treat all of them as completely rigid body that has infinite stiffness and then evaluate virtual displacement for colliding particles. At the geometric level, in order to faithfully maintain the fracture interface during the crack simulation, we utilize an efficient shape representation of solid based on the tetrahedral decomposition of the original solid geometry. This novel hybrid approach resorts to local particle models, whose goal is to avoid heavy computational burden during crack interface updating and topological changing, and meanwhile, it facilitates the user‐initiated interactive control during the crack generation and propagation. Our animation experiments demonstrate the effectiveness of our novel particle‐based method to simulate the crack of brittle material. Copyright © 2013 John Wiley & Sons, Ltd.

The extraction and classification of multitype (point, curve, patch) features on manifolds are extremely challenging, due to the lack of rigorous definition for diverse feature forms. This paper seeks a novel solution of multitype features in a mathematically rigorous way and proposes an efficient method for feature classification on manifolds. We tackle this challenge by exploring a quasi-harmonic field (QHF) generated by elliptic PDEs, which is the stable state of heat diffusion governed by anisotropic diffusion tensor. Diffusion tensor locally encodes shape geometry and controls velocity and direction of the diffusion process. The global QHF weaves points into smooth regions separated by ridges and has superior performance in combating noise/holes. Our method's originality is highlighted by the integration of locally defined diffusion tensor and globally defined elliptic PDEs in an anisotropic manner. At the computational front, the heat diffusion PDE becomes a linear system with Dirichlet condition at heat sources (called seeds). Our new algorithms afford automatic seed selection, enhanced by a fast update procedure in a high-dimensional space. By employing diffusion probability, our method can handle both manufactured parts and organic objects. Various experiments demonstrate the flexibility and high performance of our method.

As signal processing tools, diffusion wavelets and biorthogonal diffusion wavelets have been propelled by recent research in mathematics. They employ diffusion as a smoothing and scaling process to empower multiscale analysis. However, their applications in graphics and visualization are overshadowed by nonadmissible wavelets and their expensive computation. In this paper, our motivation is to broaden the application scope to space-frequency processing of shape geometry and scalar fields. We propose the admissible diffusion wavelets (ADW) on meshed surfaces and point clouds. The ADW are constructed in a bottom-up manner that starts from a local operator in a high frequency, and dilates by its dyadic powers to low frequencies. By relieving the orthogonality and enforcing normalization, the wavelets are locally supported and admissible, hence facilitating data analysis and geometry processing. We define the novel rapid reconstruction, which recovers the signal from multiple bands of high frequencies and a low-frequency base in full resolution. It enables operations localized in both space and frequency by manipulating wavelet coefficients through space-frequency filters. This paper aims to build a common theoretic foundation for a host of applications, including saliency visualization, multiscale feature extraction, spectral geometry processing, etc.

This paper develops a novel volumetric parameterization and spline construction framework, which is an effective modeling tool for converting surface meshes to volumetric splines. Our new splines are defined upon a novel parametric domain called generalized polycubes (GPCs). A GPC comprises a set of regular cube domains topologically glued together. Compared with conventional polycubes (CPCs), the GPC is much more powerful and flexible and has improved numerical accuracy and computational efficiency when serving as a parametric domain. We design an automatic algorithm to construct the GPC domain while also permitting the user to improve shape abstraction via interactive intervention. We then parameterize the input model on the GPC domain. Finally, we devise a new volumetric spline scheme based on this seamless volumetric parameterization. With a hierarchical fitting scheme, the proposed splines can fit data accurately using reduced number of superfluous control points. Our volumetric modeling scheme has great potential in shape modeling, engineering analysis, and reverse engineering applications.

Current popular anomaly detection algorithms are capable of detecting global anomalies but oftentimes fail to distinguish local anomalies from normal instances. This paper aims to improve unsupervised anomaly detection via the exploration of physics-based diffusion space. Building upon the embedding manifold derived from diffusion maps, we devise Local Anomaly Descriptor (LAD) whose originality results from faithfully preserving intrinsic and informative density-relevant neighborhood information. This robust and effective algorithm is designed with a weighted umbrella Laplacian operator to bridge global and local properties. To further enhance the efficacy of our proposed algorithm, we explore the utility of anisotropic Gaussian kernel (AGK) which can offer better manifold-aware affinity information. Comprehensive experiments on both synthetic and UCI real datasets verify that our LAD outperforms existing anomaly detection algorithms.

The primary originality of this paper lies at the fact that we have made the first attempt to apply quantum mechanics theory to anomaly (outlier) detection in high-dimensional datasets for data mining. We propose Fermi Density Descriptor (FDD) which represents the probability of measuring a fermion at a specific location for anomaly detection. We also quantify and examine different Laplacian normalization effects and choose the best one for anomaly detection. Both theoretical proof and quantitative experiments demonstrate that our proposed FDD is substantially more discriminative and robust than the commonly-used algorithms.

Abstract In this paper, we present a novel method to couple Smoothed Particle Hydrodynamics (SPH) and nonlinear FEM to animate the interaction of fluids and deformable solids in real time. To accurately model the coupling, we generate proxy particles over the boundary of deformable solids to facilitate the interaction with fluid particles, and develop an efficient method to distribute the coupling forces of proxy particles to FEM nodal points. Specifically, we employ the Total Lagrangian Explicit Dynamics (TLED) finite element algorithm for nonlinear FEM because of many of its attractive properties such as supporting massive parallelism, avoiding dynamic update of stiffness matrix computation, and efficient solver. Based on a predictor‐corrector scheme for both velocity and position, different normal and tangential conditions can be realized even for shell‐like thin solids. Our coupling method is entirely implemented on modern GPUs using CUDA. We demonstrate the advantage of our two‐way coupling method in computer animation via various virtual scenarios.

Image deblurring has been a very challenging problem in recent decades. In this article, we propose a high-quality image deblurring method with a novel image prior based on a new imaging system. The imaging system has a newly designed sensor pattern achieved by adding panchromatic (pan) pixels to the conventional Bayer pattern. Since these pan pixels are sensitive to all wavelengths of visible light, they collect a significantly higher proportion of the light striking the sensor. A new demosaicing algorithm is also proposed to restore full-resolution images from pixels on the sensor. The shutter speed of pan pixels is controllable to users. Therefore, we can produce multiple images with different exposures. When long exposure is needed under dim light, we read pan pixels twice in one shot: one with short exposure and the other with long exposure. The long-exposure image is often blurred, while the short-exposure image can be sharp and noisy. The short-exposure image plays an important role in deblurring, since it is sharp and there is no alignment problem for the one-shot image pair. For the algorithmic aspect, our method runs in a two-step maximum-a-posteriori (MAP) fashion under a joint minimization of the blur kernel and the deblurred image. The algorithm exploits a combined image prior with a statistical part and a spatial part, which is powerful in ringing controls. Extensive experiments under various conditions and settings are conducted to demonstrate the performance of our method.

This paper develops a novel surface fitting scheme for automatically reconstructing a genus-0 object into a continuous parametric spline surface. A key contribution for making such a fitting method both practical and accurate is our spherical generalization of the Delaunay configuration B-spline (DCB-spline), a new non-tensor-product spline. In this framework, we efficiently compute Delaunay configurations on sphere by the union of two planar Delaunay configurations. Also, we develop a hierarchical and adaptive method that progressively improves the fitting quality by new knot-insertion strategies guided by surface geometry and fitting error. Within our framework, a genus-0 model can be converted to a single spherical spline representation whose root mean square error is tightly bounded within a user-specified tolerance. The reconstructed continuous representation has many attractive properties such as global smoothness and no auxiliary knots. We conduct several experiments to demonstrate the efficacy of our new approach for reverse engineering and shape modeling.

This paper presents a complete and robust solution for dense registration of partial nonrigid shapes. Its novel contributions are founded upon the newly proposed heat kernel coordinates (HKCs) that can accurately position points on the shape, and the priority-vicinity search that ensures geometric compatibility during the registration. HKCs index points by computing heat kernels from multiple sources, and their magnitudes serve as priorities of queuing points in registration. We start with shape features as the sources of heat kernels via feature detection and matching. Following the priority order of HKCs, the dense registration is progressively propagated from feature sources to all points. Our method has a superior indexing ability that can produce dense correspondences with fewer flips. The diffusion nature of HKCs, which can be interpreted as a random walk on a manifold, makes our method robust to noise and small holes avoiding surface surgery and repair. Our method searches correspondence only in a small vicinity of registered points, which significantly improves the time performance. Through comprehensive experiments, our new method has demonstrated its technical soundness and robustness by generating highly compatible dense correspondences.

This paper presents a novel modeling framework to build 3D models of Chinese architectures from elevation drawing. Our algorithm integrates the capability of automatic drawing recognition with powerful procedural modeling to extract production rules from elevation drawing. First, different from the previous symbol-based floor plan recognition, based on the novel concept of repetitive pattern trees, small horizontal repetitive regions of the elevation drawing are clustered in a bottom-up manner to form architectural components with maximum repetition, which collectively serve as building blocks for 3D model generation. Second, to discover the global architectural structure and its components' interdependencies, the components are structured into a shape tree in a top-down subdivision manner and recognized hierarchically at each level of the shape tree based on Markov Random Fields (MRFs). Third, shape grammar rules can be derived to construct 3D semantic model and its possible variations with the help of a 3D component repository. The salient contribution lies in the novel integration of procedural modeling with elevation drawing, with a unique application to Chinese architectures.

This paper presents a volumetric modeling framework to construct a novel spline scheme called restricted trivariate polycube splines (RTP-splines). The RTP-spline aims to generalize both trivariate T-splines and tensor-product B-splines; it uses solid polycube structure as underlying parametric domains and strictly bounds blending functions within such domains. We construct volumetric RTP-splines in a top-down fashion in four steps: 1) Extending the polycube domain to its bounding volume via space filling; 2) building the B-spline volume over the extended domain with restricted boundaries; 3) inserting duplicate knots by adding anchor points and performing local refinement; and 4) removing exterior cells and anchors. Besides local refinement inherited from general T-splines, the RTP-splines have a few attractive properties as follows: 1) They naturally model solid objects with complicated topologies/bifurcations using a one-piece continuous representation without domain trimming/patching/merging. 2) They have guaranteed semistandardness so that the functions and derivatives evaluation is very efficient. 3) Their restricted support regions of blending functions prevent control points from influencing other nearby domain regions that stay opposite to the immediate boundaries. These features are highly desirable for certain applications such as isogeometric analysis. We conduct extensive experiments on converting complicated solid models into RTP-splines, and demonstrate the proposed spline to be a powerful and promising tool for volumetric modeling and other scientific/engineering applications where data sets with multiattributes are prevalent.

Current spectral clustering algorithms suffer from both sensitivity to scaling parameter selection in similarity matrix construction, and data perturbation. This paper aims to improve robustness in clustering algorithms and combat these two limitations based on heat kernel theory. Heat kernel can statistically depict traces of random walk, so it has an intrinsic connection with diffusion distance, with which we can ensure robustness during any clustering process. By integrating heat distributed along time scale, we propose a novel method called Aggregated Heat Kernel (AHK) to measure the distance between each point pair in their eigen space. Using AHK and Laplace-Beltrami Normalization (LBN) we are able to apply an advanced noise-resisting robust spectral mapping to original dataset. Moreover it offers stability on scaling parameter tuning. Experimental results show that, compared to other popular spectral clustering methods, our algorithm can achieve robust clustering results on both synthetic and UCI real datasets.

This paper proposes a new image deconvolution method using multi-stage convex relaxation, and presents a metric for perceptual evaluation of deconvolution results. Recent work in image deconvolution addresses the deconvolution problem via minimization with non-convex regularization. Since all regularization terms in the objective function are non-convex, this problem can be well modeled and solved by multi-stage convex relaxation. This method, adopted from machine learning, iteratively refines the convex relaxation formulation using concave duality. The newly proposed deconvolution method has outstanding performance in noise removal and artifact control. A new metric, transduced contrast-to-distortion ratio (TCDR), is proposed based on a human vision system (HVS) model that simulates human responses to visual contrasts. It is sensitive to ringing and boundary artifacts, and very efficient to compute. We conduct comprehensive perceptual evaluation of image deconvolution using visual signal-to-noise ratio (VSNR) and TCDR. Experimental results of both synthetic and real data demonstrate that our method indeed improves the visual quality of deconvolution results with low distortions and artifacts.

In this paper, we develop a method for learning illumination from a single image, which can benefit illumination-invariant algorithms in computer vision and image-based rendering in graphics. Illumination learning has been widely studied, yet still has some shortcomings such as the restriction of Lambertian surfaces and the prerequisite of known shape or texture. Our method can adaptively learn illumination from images of vehicles with unknown shape and texture. We formulate the illumination model with both diffusion and specularity components using a frequency-space representation, and adopt an iterative strategy to estimate lighting, shape, and texture under a joint energy function. Using our method, we can perform de-lighting and re-lighting on input images, and render other 3D models with learned illumination. Experimental results show that our method can work in a wide range of real-world environments with both indoor and outdoor illumination conditions.

The natural image prior has been proven to be a powerful tool for image deblurring in recent years, though its performance against noise in various applications has not been thoroughly studied. In this paper, we present a multigrid natural image prior for image deconvolution that enhances its robustness against noise, and afford three applications of image deconvolution using this prior: deblurring, super-resolution, and denoising. The prior is based on a remarkable property of natural images that derivatives with different resolutions are subject to the same heavy-tailed distribution with a spatial factor. It can serve in both blind and non-blind deconvolutions. The performances of the proposed prior in different applications are demonstrated by corresponding experimental results.

This paper develops a new trivariate hierarchical spline scheme for volumetric data representation. Unlike conventional spline formulations and techniques, our new framework is built upon a novel parametric domain called Generalized PolyCube (GPC), comprising a set of regular cubes being glued together. Compared with the conventional PolyCube (PC) that could serve as a "one-piece'' 3-manifold domain, GPC has more powerful and flexible representation ability. We develop an effective framework that parameterizes a solid model onto a topologically equivalent GPC domain, and design a hierarchical fitting scheme based on trivariate T-splines. The entire data-spline-conversion modeling framework provides high-accuracy data fitting and greatly reduce the number of superfluous control points. It is a powerful toolkit with broader application appeal in shape modeling, engineering analysis, and reverse engineering.

Constructing smooth freeform surfaces of arbitrary topology with higher order continuity is one of the most fundamental problems in shape and solid modeling. This paper articulates a novel method to construct C∞ smooth surfaces with negative Euler numbers based on hyperbolic geometry and discrete curvature flow. According to Riemann uniformization theorem, every surface with negative Euler number has a unique conformal Riemannian metric, which induces Gaussian curvature of --1 everywhere. Hence, the surface admits hyperbolic geometry. Such uniformization metric can be computed using the discrete curvature flow method: hyperbolic Ricci flow. Consequently, the basis function for each control point can be naturally defined over a hyperbolic disk, and through the use of partition-of-unity, we build a freeform surface directly over hyperbolic domains while having C∞ property. The use of radial, exponential basis functions gives rise to a true meshless method for modeling freeform surfaces with greatest flexibilities, without worrying about control point connectivity. Our algorithm is general for arbitrary surfaces with negative Euler characteristic. Furthermore, it is C∞ continuous everywhere across the entire hyperbolic domain without singularities. Our experimental results demonstrate the efficiency and efficacy of the proposed new approach for shape and solid modeling.

Harmonic volumetric mapping aims to establish a smooth bijective correspondence between two solid shapes with the same topology. In this paper, we develop an automatic meshless method for creating such a mapping between two given objects. With the shell surface mapping as the boundary condition, we first solve a linear system constructed by a boundary method called themethodoffundamentalsolution, and then represent the mapping using a set of points with different weights in the vicinity of the shell of the given model. Our algorithm is a true meshless method (without the need of any specific meshing structure within the solid interior) and the behavior of the interior region is directly determined by the boundary, which can improve the computational efficiency and robustness significantly. Therefore, our algorithm can be applied to massive volume data sets with various geometric primitives and topological types. We demonstrate the utility and efficacy of our algorithm in information transfer, shape registration, deformation sequence analysis, tetrahedral remeshing, and solid texture synthesis.

Surface mapping is fundamental to shape computing and various downstream applications. This paper develops a pants decomposition framework for computing maps between surfaces with arbitrary topologies. The framework first conducts pants decomposition on both surfaces to segment them into consistent sets of pants patches (a pants patch is intuitively defined as a genus-0 surface with three boundaries), then composes global mapping between two surfaces by using harmonic maps of corresponding patches. This framework has several key advantages over existing techniques. First, it is automatic. It can automatically construct mappings for surfaces with complicated topology, guaranteeing the one-to-one continuity. Second, it is general and powerful. It flexibly handles mapping computation between surfaces with different topologies. Third, it is flexible. Despite topology and geometry, it can also integrate semantics requirements from users. Through a simple and intuitive human-computer interaction mechanism, the user can flexibly control the mapping behavior by enforcing point/curve constraints. Compared with traditional user-guided, piecewise surface mapping techniques, our new method is less labor intensive, more intuitive, and requires no user's expertise in computing complicated surface maps between arbitrary shapes. We conduct various experiments to demonstrate its modeling potential and effectiveness.

The six selected papers in this special section are improved and extended versions of three papers from the ACM Solid and Physical Modeling Symposium 2008 (SPM '08) and three papers from the International Conference on Shape Modeling and Applications 2008 (SMI '08).

Automatic non-rigid registration of 3D time-varying data is fundamental in many vision and graphics applications such as facial expression analysis, synthesis, and recognition. Despite many research advances in recent years, it still remains to be technically challenging, especially for 3D dynamic, densely-sampled facial data with a large number of degrees of freedom (necessarily used to represent rich and subtle facial expressions). In this paper, we present a new method for automatic non-rigid registration of 3D dynamic facial data using least-squares conformal maps, and based on this registration method, we also develop a new framework of facial expression synthesis and transfer. Nowadays more and more 3D dynamic, densely-sampled data become prevalent with the advancement of novel 3D scanning techniques. To analyze and utilize such huge 3D data, an efficient non-rigid registration algorithm is needed to establish one-to-one inter frame correspondences. Towards this goal, a non-rigid registration algorithm of 3D dynamic facial data is developed by using least-squares conformal maps with additional feature correspondences detected by employing active appearance models (AAM). The proposed method with additional, interior feature constraints guarantees that the non-rigid data will be accurately registered. The least-squares conformal maps between two 3D surfaces are globally optimized with the least angle distortion and the resulting 2D maps are stable and one-to-one. Furthermore, by using this non-rigid registration method, we develop a new system of facial expression synthesis and transfer. Finally, we perform a series of experiments to evaluate our non-rigid registration method and demonstrate its efficacy and efficiency in the applications of facial expression synthesis and transfer.

We consider feature extraction (dimensionality reduction) for compositional data, where the data vectors are constrained to be positive and constant-sum. In real-world problems, the data components (variables) usually have complicated "correlations" while their total number is huge. Such scenario demands feature extraction. That is, we shall de-correlate the components and reduce their dimensionality. Traditional techniques such as the Principle Component Analysis (PCA) are not suitable for these problems due to unique statistical properties and the need to satisfy the constraints in compositional data. This paper presents a novel approach to feature extraction for compositional data. Our method first identifies a family of dimensionality reduction projections that preserve all relevant constraints, and then finds the optimal projection that maximizes the estimated Dirichlet precision on projected data. It reduces the compositional data to a given lower dimensionality while the components in the lower-dimensional space are de-correlated as much as possible. We develop theoretical foundation of our approach, and validate its effectiveness on some synthetic and real-world datasets.

Surface matching is fundamental to shape computing and various downstream applications. This paper develops a powerful pants decomposition framework for computing maps between surfaces with arbitrary topologies. We first conduct pants decomposition on both surfaces to segment them into consistent sets of pants patches (here a pants patch is intuitively defined as a genus-zero surface with three boundaries). Then we compose global mapping between two surfaces by harmonic maps of corresponding patches. This framework has several key advantages over other state-of-the-art techniques. First, the surface decomposition is automatic and general. It can automatically construct mappings for surfaces with same but complicated topology, and the result is guaranteed to be one-to-one continuous. Second, the mapping framework is very flexible and powerful. Not only topology and geometry, but also the semantics can be easily integrated into this framework with a little user involvement. Specifically, it provides an easy and intuitive human-computer interaction mechanism so that mapping between surfaces with different topologies, or with additional point/curve constraints, can be properly obtained within our framework. Compared with previous user-guided, piecewise surface mapping techniques, our new method is more intuitive, less labor-intensive, and requires no user's expertise in computing complicated surface map between arbitrary shapes. We conduct various experiments to demonstrate its modeling potential and effectiveness. © 2008 ACM.

This paper presents a novel computational framework based on dynamic spherical volumetric simplex splines for simulation of genuszero real-world objects. In this framework, we first develop an accurate and efficient algorithm to reconstruct the high-fidelity digital model of a real-world object with spherical volumetric simplex splines which can represent with accuracy geometric, material, and other properties of the object simultaneously. With the tight coupling of Lagrangian mechanics, the dynamic volumetric simplex splines representing the object can accurately simulate its physical behavior because it can unify the geometric and material properties in the simulation. The visualization can be directly computed from the object's geometric or physical representation based on the dynamic spherical volumetric simplex splines during simulation without interpolation or resampling. We have applied the framework for biomechanic simulation of brain deformations, such as brain shifting during the surgery and brain injury under blunt impact. We have compared our simulation results with the ground truth obtained through intra-operative magnetic resonance imaging and the real biomechanic experiments. The evaluations demonstrate the excellent performance of our new technique presented in this paper.

Polycube T-spline has been formulated elegantly that can unify T-splines and manifold splines to define a new class of shape representations for surfaces of arbitrary topology by using polycube map as its parametric domain. In essense, The data fitting quality using polycube T-splines hinges upon the construction of underlying polycube maps. Yet, existing methods for polycube map construction exhibit some disadvantages. For example, existing approaches for polycube map construction either require projection of points from a 3D surface to its polycube approximation, which is therefore very difficult to handle the cases when two shapes differ significantly; or compute the map by conformally deforming the surfaces and polycubes to the common canonical domain and then construct the map using function composition, which is challenging to control the location of singularities and makes it hard for the data-fitting and hole-filling processes later on.

The International Conference on Computer Animation and Social Agents (CASA) organized by the Computer Graphics Society (CGS) is one of the premier academic conferences in the field of computer animation, specializing in character/object modeling, animation, and behavior simulation of social agents. This year, the conference was held in Seoul, Korea marking its 21st occurrence. The CASA international program committee has selected 32 full papers from 100 submissions around the world based on comments and scores from at least four reviewers assigned for each submitted paper. It is our great pleasure to have those distinguished work to be presented in this special issue of the Computer Animation and Virtual Worlds Journal (CAVW). The 32 papers presented in this journal are nicely categorized into: (1) character animation, (2) facial animation, (3) crowd modeling and simulation, (4) synthesis and 2D methods, (5) general modeling and deformation techniques, (6) physical simulation, and (7) applications. The Program Co-Chairs would like to thank the international program committee for spending their valuable time and effort in the reviewing process and the selected authors for their contribution.

Abstract This paper develops a new salient keypoints‐based shape description which extracts the salient surface keypoints with detected scales. Salient geometric features can then be defined collectively on all the detected scale normalized local patches to form a shape descriptor for surface matching purpose. The saliency‐driven keypoints are computed as local extrema of the difference of Gaussian function defined over a curved surface in geodesic scale space. This method can properly function on either manifold or non‐manifold surface without resorting to any surface mapping or parameterization procedures. Therefore, it has a wide utility in many applications such as shape matching, classification, and recognition. Our experiments on 3D shapes demonstrate that the salient keypoints and local feature descriptors are robust and stable to noisy input and insensitive to resolution change. We have applied our technique to the tasks of 3D shape matching, and the experimental results showed good performance and the effectiveness of this new method. Copyright © 2008 John Wiley & Sons, Ltd.

This paper presents a novel and efficient surface matching and visualization framework through the geodesic distance-weighted shape vector image diffusion. Based on conformal geometry, our approach can uniquely map a 3D surface to a canonical rectangular domain and encode the shape characteristics (e.g., mean curvatures and conformal factors) of the surface in the 2D domain to construct a geodesic distance-weighted shape vector image, where the distances between sampling pixels are not uniform but the actual geodesic distances on the manifold. Through the novel geodesic distance-weighted shape vector image diffusion presented in this paper, we can create a multiscale diffusion space, in which the cross-scale extrema can be detected as the robust geometric features for the matching and registration of surfaces. Therefore, statistical analysis and visualization of surface properties across subjects become readily available. The experiments on scanned surface models show that our method is very robust for feature extraction and surface matching even under noise and resolution change. We have also applied the framework on the real 3D human neocortical surfaces, and demonstrated the excellent performance of our approach in statistical analysis and integrated visualization of the multimodality volumetric data over the shape vector image.

Computing smooth and optimal one-to-one maps between surfaces of same topology is a fundamental problem in computer graphics and such a method provides us a ubiquitous tool for geometric modeling and data visualization. Its vast variety of applications includes shape registration/matching, shape blending, material/data transfer, data fusion, information reuse, etc. The mapping quality is typically measured in terms of angular distortions among different shapes. This paper proposes and develops a novel quasi-conformal surface mapping framework to globally minimize the stretching energy inevitably introduced between two different shapes. The existing state-of-the-art inter-surface mapping techniques only afford local optimization either on surface patches via boundary cutting or on the simplified base domain, lacking rigorous mathematical foundation and analysis. We design and articulate an automatic variational algorithm that can reach the global distortion minimum for surface mapping between shapes of arbitrary topology, and our algorithm is sorely founded upon the intrinsic geometry structure of surfaces. To our best knowledge, this is the first attempt towards numerically computing globally optimal maps. Consequently, our mapping framework offers a powerful computational tool for graphics and visualization tasks such as data and texture transfer, shape morphing, and shape matching.

Ricci flow is a powerful curvature flow method in geometric analysis. This work is the first application of surface Ricci flow in computer vision. We show that previous methods based on conformal geometries, such as harmonic maps and least-square conformal maps, which can only handle 3D shapes with simple topology are subsumed by our Ricci flow based method which can handle surfaces with arbitrary topology. Because the Ricci flow method is intrinsic and depends on the surface metric only, it is invariant to rigid motion, scaling, and isometric and conformal deformations. The solution to Ricci flow is unique and its computation is robust to noise. Our Ricci flow based method can convert all 3D problems into 2D domains and offers a general framework for 3D surface analysis. Large non-rigid deformations can be registered with feature constraints, hence we introduce a method that constrains Ricci flow computation using feature points and feature curves. Finally, we demonstrate the applicability of this intrinsic shape representation through standard shape analysis problems, such as 3D shape matching and registration.

Proportional data (normalized histograms) have been frequently occurring in various areas, and they could be mathematically abstracted as points residing in a geometric simplex. A proper distance metric on this simplex is of importance in many applications including classification and information retrieval. In this paper, we develop a novel framework to learn an optimal metric on the simplex. Major features of our approach include: 1) its flexibility to handle correlations among bins/dimensions; 2) widespread applicability without being limited to ad hoc backgrounds; and 3) a "real" global solution in contrast to existing traditional local approaches. The technical essence of our approach is to fit a parametric distribution to the observed empirical data in the simplex. The distribution is parameterized by affinities between simplex vertices, which is learned via maximizing likelihood of observed data. Then, these affinities induce a metric on the simplex, defined as the earth mover's distance equipped with ground distances derived from simplex vertex affinities.

This paper develops a novel computational technique to define and construct powerful manifold splines with only one singular point by employing the rigorous mathematical theory of Ricci flow. The central idea and new computational paradigm of manifold splines are to systematically extend the algorithmic pipeline of spline surface construction from any planar domain to arbitrary topology. As a result, manifold splines can unify planar spline representations as their special cases. Despite their earlier success, the existing manifold spline framework is plagued by the topology-dependent, large number of singular points (i.e., |2g -- 2| for any genus-g surface), where the analysis of surface behaviors such as continuity remains extremely difficult. The unique theoretical contribution of this paper is that we devise new mathematical tools so that manifold splines can now be constructed with only one singular point, reaching their theoretic lower bound of singularity for real-world applications. Our new algorithm is founded upon the concept of discrete Ricci flow and associated techniques. First, Ricci flow is employed to compute a special metric of any manifold domain (serving as a parametric domain for manifold splines), such that the metric becomes flat everywhere except at one point. Then, the metric naturally induces an affine atlas covering the entire manifold except this singular point. Finally, manifold splines are defined over this affine atlas. The Ricci flow method is theoretically sound, and practically simple and efficient. We conduct various shape experiments and our new theoretical and algorithmic results alleviate the modeling difficulty of manifold splines, and hence, promising to promote the widespread use of manifold splines in surface and solid modeling, geometric design, and reverse engineering.

Harmonic volumetric mapping for two solid objects establishes a one-to-one smooth correspondence between them. It finds its applications in shape registration and analysis, shape retrieval, information reuse, and material/texture transplant. In sharp contrast to harmonic surface mapping techniques, little research has been conducted for designing volumetric mapping algorithms due to its technical challenges. In this paper, we develop an automatic and effective algorithm for computing harmonic volumetric mapping between two models of the same topology. Given a boundary mapping between two models, the volumetric (interior) mapping is derived by solving a linear system constructed from a boundary method called the fundamental solution method. The mapping is represented as a set of points with different weights in the vicinity of the solid boundary. In a nutshell, our algorithm is a true meshless method (with no need of specific connectivity) and the behavior of the interior region is directly determined by the boundary. These two properties help improve the computational efficiency and robustness. Therefore, our algorithm can be applied to massive volume data sets with various geometric primitives and topological types. We demonstrate the utility and efficacy of our algorithm in shape registration, information reuse, deformation sequence analysis, tetrahedral remeshing and solid texture synthesis.

This paper proposes a new concept of polycube splines and develops novel modeling techniques for using the polycube splines in solid modeling and shape computing. Polycube splines are essentially a novel variant of manifold splines which are built upon the polycube map, serving as its parametric domain. Our rationale for defining spline surfaces over polycubes is that polycubes have rectangular structures everywhere over their domains except a very small number of corner points. The boundary of polycubes can be naturally decomposed into a set of regular structures, which facilitate tensor-product surface definition, GPU-centric geometric computing, and image-based geometric processing. We develop algorithms to construct polycube maps, and show that the introduced polycube map naturally induces the affine structure with a finite number of extraordinary points. Besides its intrinsic rectangular structure, the polycube map may approximate any original scanned data-set with a very low geometric distortion, so our method for building polycube splines is both natural and necessary, as its parametric domain can mimic the geometry of modeled objects in a topologically correct and geometrically meaningful manner. We design a new data structure that facilitates the intuitive and rapid construction of polycube splines in this paper. We demonstrate the polycube splines with applications in surface reconstruction and shape computing.

Volumetric Subdivision (VS) is a powerful paradigm that enables volumetric sculpting and realistic volume deformations that give rise to the concept of "virtual clay". In VS, volumes are commonly represented as a space-filling set of deformed polyhedra, which can be further decomposed into a mesh of tetrahedra for rendering. Images can then be generated via tetrahedral projection or raycasting. A current shortcoming in VS-based operations is the need for a very high level of subdivision to represent fine detail in the mesh and to obtain a high-fidelity visualization. However, we have discovered that the subdivision process itself can be closely simulated with radial basis functions (RBFs), making it possible to replace the finer subdivision levels by a coarser aggregation of RBF kernels. This reduction to a simplified assembly of RBFs subsequently enables interactive rendering of volumetric subdivision shapes within a GPU-based volume splatting framework.

We present a complete framework for computing a subdivision surface to approximate unorganized point sample data, which is a separable nonlinear least squares problem. We study the convergence and stability of three geometrically-motivated optimization schemes and reveal their intrinsic relations with standard methods for constrained nonlinear optimization. A commonly-used method in graphics, called point distance minimization, is shown to use a variant of the gradient descent step and thus has only linear convergence. The second method, called tangent distance minimization, which is well-known in computer vision, is shown to use the Gauss-Newton step, and thus demonstrates near quadratic convergence for zero residual problems but may not converge otherwise. Finally, we show that an optimization scheme called squared distance minimization, recently proposed by Pottmann et al., can be derived from the Newton method. Hence, with proper regularization, tangent distance minimization and squared distance minimization are more efficient than point distance minimization. We also investigate the effects of two step size control methods -- Levenberg-Marquardt regularization and the Armijo rule -- on the convergence stability and efficiency of the above optimization schemes.

Parametric PDE techniques, which use partial differential equations (PDEs) defined over a 2D or 3D parametric domain to model graphical objects and processes, can unify geometric attributes and functional constraints of the models. PDEs can also model implicit shapes defined by level sets of scalar intensity fields. In this paper, we present an approach that integrates parametric and implicit trivariate PDEs to define geometric solid models containing both geometric information and intensity distribution subject to flexible boundary conditions. The integrated formulation of second-order or fourth-order elliptic PDEs permits designers to manipulate PDE objects of complex geometry and/or arbitrary topology through direct sculpting and free-form modeling. We developed a PDE-based geometric modeling system for shape design and manipulation of PDE objects. The integration of implicit PDEs with parametric geometry offers more general and arbitrary shape blending and free-form modeling for objects with intensity attributes than pure geometric models.

Conveying shape using feature lines is an important visualization tool in visual computing. The existing feature lines (e.g., ridges, valleys, silhouettes, suggestive contours, etc.) are solely determined by local geometry properties (e.g., normals and curvatures) as well as the view position. This paper is strongly inspired by the observation in human vision and perception that a sudden change in the luminance plays a critical role to faithfully represent and recover the 3D information. In particular, we adopt the edge detection techniques in image processing for 3D shape visualization and present Photic Extremum Lines (PELs) which emphasize significant variations of illumination over 3D surfaces. Comparing with the existing feature lines, PELs are more flexible and offer users more freedom to achieve desirable visualization effects. In addition, the user can easily control the shape visualization by changing the light position, the number of light sources, and choosing various light models. We compare PELs with the existing approaches and demonstrate that PEL is a flexible and effective tool to illustrate 3D surface and volume for visual computing.

This paper develops an efficient new method for 3D partial shape retrieval. First, a Monte Carlo sampling strategy is employed to extract local shape signatures from each 3D model. After vector quantization, these features are represented by using a bag-of-words model. The main contributions of this paper are threefold as follows: 1) a partial shape dissimilarity measure is proposed to rank shapes according to their distances to the input query, without using any timeconsuming alignment procedure; 2) by applying the probabilistic text analysis technique, a highly compact representation "Shape Topics" and accompanying algorithms are developed for efficient 3D partial shape retrieval, the mapping from "Shape Topics" to "object categories" is established using multi-class SVMs; and 3) a method for evaluating the performance of partial shape retrieval is proposed and tested. To our best knowledge, very few existing methods are able to perform well online partial shape retrieval for large 3D shape repositories. Our experimental results are expected to validate the efficacy and effectiveness of our novel approach.

We develop a new surface matching framework to handle surface comparisons based on the mathematical analysis of curves on surfaces, and propose a unique signature for any closed curve on a surface. The signature describes not only the shape of the curve, but also the intrinsic relationship between the curve and its embedding surface; and furthermore, the signature metric is stable across surfaces sharing similar Riemannian geometry metrics. Based on this theoretical advance, we analyze and align features defined as closed curves on surfaces using their signatures. These curves segment a surface into different regions which are mapped onto canonical domains for the matching purpose. The experimental results are very promising, demonstrating that the curve signatures and the comparison framework are robust and discriminative for the effective shape comparison. Besides its utility in our current framework, we believe the curve signature will also serve as a powerful shape segmentation/mapping tool and can be used to aid in many existing techniques towards effective shape analysis

This paper presents a novel 3D shape descriptor "the generalized shape distributions" for effective shape matching and analysis, by taking advantage of both local and global shape signatures. We start this process by generating spin images on meshes. These local shape descriptors are then quantized via k-means clustering. The key contribution of this paper is to represent a global 3D shape as the spatial configuration of a set of specific local shapes. We achieve this goal by computing the distributions of the Euclidean distance of pairs of local shape clusters. Because of the spatial, sparse distribution of local shapes defined over a 3D model, an indexing data structure is adopted to reduce the space complexity of the proposed shape descriptor. The technical merits of our new approach are at least two-fold: (1) it is robust to non-trivial shape occlusions and deformations, since there are statistically a large number of chances that some local shape signatures and their spatial layouts are unchanged and users can easily identify those unchanged parts; (2) it is more discriminative than a simple collection of local shape signatures, since the spatial layouts of a global shape are explicitly computed. Our preliminary experiments have shown the effectiveness of this new approach for shape comparison and analysis

Radial Basis Functions (RBFs) have become a popular rendering primitive, both in surface and in volume rendering. This paper focuses on volume visualization, giving rise to 3D kernels. RBFs are especially convenient for the representation of scattered and irregularly distributed point samples, where the RBF kernel is used as a blending function for the space in between samples. Common representations employ radially symmetric RBFs, and various techniques have been introduced to render these, also with efficient implementations on programmable graphics hardware (GPUs). In this paper, we extend the existing work to more generalized, ellipsoidal RBF kernels, for the rendering of scattered volume data. We devise a post-shaded kernel-centric rendering approach, specifically designed to run efficiently on GPUs, and we demonstrate our renderer using datasets from subdivision volumes and computational science.

This paper presents a new approach to the physically-based thin-shell simulation of point-sampled geometry via explicit, global conformal point-surface parameterization and meshless dynamics. The point-based global parameterization is founded upon the rigorous mathematics of Riemann surface theory and Hodge theory. The parameterization is globally conformal everywhere except for a minimum number of zero points. Within our parameterization framework, any well-sampled point surface is functionally equivalent to a manifold, enabling popular and powerful surface-based modeling and physically-based simulation tools to be readily adapted for point geometry processing and animation. In addition, we propose a meshless surface computational paradigm in which the partial differential equations (for dynamic physical simulation) can be applied and solved directly over point samples via Moving Least Squares (MLS) shape functions defined on the global parametric domain without explicit connectivity information. The global conformal parameterization provides a common domain to facilitate accurate meshless simulation and efficient discontinuity modeling for complex branching cracks. Through our experiments on thin-shell elastic deformation and fracture simulation, we demonstrate that our integrative method is very natural, and that it has great potential to further broaden the application scope of point-sampled geometry in graphics and relevant fields.

This paper presents a new surface content completion framework that can restore both shape and appearance from scanned, incomplete point set inputs. First, the geometric holes can be robustly identified from noisy and defective data sets without the need of any normal or orientation information, using the method of active deformable models. The geometry and texture information of the holes can then be determined either automatically from the models' context, or semi-automatically with minimal users' intervention. The central idea for this repair process is to establish a quantitative similarity measurement among local surface patches based on their local parameterizations and curvature computation. The geometry and texture information of each hole can be completed by warping the candidate region and gluing it to the hole. The displacement for the alignment process is computed by solving a Poisson equation in 2D. Our experiments show that the unified framework, founded upon the techniques of deformable models, local parameterization, and PDE modeling, can provide a robust and elegant solution for content completion of defective, complex point surfaces.

Constructing splines whose parametric domain is an arbitrary manifold and effectively computing such splines in real-world applications are of fundamental importance in solid and shape modeling, geometric design, graphics, etc. This paper presents a general theoretical and computational framework, in which spline surfaces defined over planar domains can be systematically extended to manifold domains with arbitrary topology with or without boundaries. We study the affine structure of domain manifolds in depth and prove that the existence of manifold splines is equivalent to the existence of a manifold's affine atlas. Based on our theoretical breakthrough, we also develop a set of practical algorithms to generalize triangular B-spline surfaces from planar domains to manifold domains. We choose triangular B-splines mainly because of its generality and many of its attractive properties. As a result, our new spline surface defined over any manifold is a piecewise polynomial surface with high parametric continuity without the need for any patching and/or trimming operations. Through our experiments, we hope to demonstrate that our novel manifold splines are both powerful and efficient in modeling arbitrarily complicated geometry and representing continuously-varying physical quantities defined over shapes of arbitrary topology.

Traditional approaches for modeling a closed manifold surface with either regular tensor-product or triangular splines (defined over an open planar domain) require decomposing the acquired geometric data into a group of charts, mapping each chart to a planar parametric domain, fitting an open surface patch of certain degree to each chart, and finally, trimming the patches (if necessary) and stitching all of them together to form a closed manifold. In this paper, we develop a novel modeling method which does not need any cutting or patching operations for genus zero surfaces. Our new approach is founded upon the concept of spherical splines proposed by Pfeifle and Seidel. Our work is strongly inspired by the fact that, for genus zero surfaces, it is both intuitive and necessary to employ spheres as their natural domains. Using this framework, we can convert genus zero mesh to a single rational spherical spline whose maximal error deviated from the original data is less than a user-specified tolerance. With the rational spherical splines, we can model sharp features and edit both the global shape and the local details with ease. Furthermore, we can accurately compute the differential quantities without resorting to any numerical approximations. We conduct several experiments in order to demonstrate the efficacy of our approach for reverse engineering, shape modeling, and interactive graphics.

This paper presents a flexible, scalable framework for interactive hands-on shape design in a haptic, stereoscopic virtual environment. The framework is founded upon the concept of PDE-based geometric surface flow. Given an input polygonal mesh, a user can interactively define implicit functions around regions of interest of the mesh model, and the locally or globally affected regions of the model will automatically deform according to the underlying partial differential equations and reconstruct the implicitly defined shape. During the model deformation process, the model can always maintain its regularity and can properly modify its topology when collisions between different parts of the model occur. With augmented haptics functionality and stereoscopic display, our system provides a more intuitive interface, which allows users to directly manipulate 3D polygonal objects with hands.

Topological concepts and techniques have been broadly applied in computer graphics and geometric modeling. However, the homotopy type of a mapping between two surfaces has not been addressed before. In this paper, we present a novel solution to the problem of computing continuous maps with different homotopy types between two arbitrary triangle meshes with the same topology. Inspired by the rich theory of topology as well as the existing body of work on surface mapping, our newly-developed mapping techniques are both fundamental and unique, offering many attractive advantages. First, our method allows the user to change the homotopy type or global structure of the mapping with minimal intervention. Moreover, to locally affect shape correspondence, we articulate a new technique that robustly satisfies hard feature constraints, without the use of heuristics to ensure validity. In addition to acting as a useful tool for computer graphics applications, our method can be used as a rigorous and practical mechanism for the visualization of abstract topological concepts such as homotopy type of surface mappings, homology basis, fundamental domain, and universal covering space. At the core of our algorithm is a procedure for computing the canonical homology basis and using it as a common cut graph for any surface with the same topology. We demonstrate our results by applying our algorithm to shape morphing in this paper.

Abstract This paper presents an innovative method for naturally and smoothly morphing point‐sampled surfaces via dynamic meshless simulation on point‐sampled surfaces. While most existing literature on shape morphing emphasizes the issue of finding a good correspondence map between two object representations, this research primarily investigates the challenging problem of how to find a smooth, physically‐meaningful transition path between two homeomorphic point‐set surfaces. We analyze the deformation of surface involved in the morphing process using concepts in differential geometry and continuum mechanics. The morphing paths can be determined by optimizing an energy functional, which characterizes the intrinsic deformation of the surface away from its rest shape. As demonstrated in the examples, our method automatically produces a series of natural and physically‐plausible in‐between shapes, which greatly alleviates the shrinking, stretching, and self‐intersection problems that often occur when linear interpolation is employed for the morphing of two objects. We envision that our new technique will continue to broaden the application scope of point‐set surfaces and their dynamic animation. Copyright © 2005 John Wiley & Sons, Ltd.

Abstract In this paper, we articulate a meshless computational paradigm for the effective modeling, accurate physical simulation, and real‐time animation of point‐sampled solid objects. Both the interior and the boundary geometry of our volumetric object representation only consist of points, further extending the powerful and popular method of point‐sampled surfaces to the volumetric setting. We build the point‐based physical model upon continuum mechanics, which affords to effectively model the dynamic elastic behavior of point‐based volumetric objects. When only surface samples are provided, our prototype system first generates both interior volumetric points and a volumetric distance field with octree structure. The physics of these volumetric points in a solid interior are simulated using the Meshless Moving Least Squares (MLS) shape functions. In sharp contrast to the traditional finite element method (FEM), the meshless property of our new technique expedites the accurate representation and precise simulation of the underlying discrete model, without the need of domain meshing. In order to achieve real‐time simulations, we utilize the warped modal analysis method that is locally linear in nature but globally warped to account for rotational deformation. The structural simplicity and real‐time performance of our meshless simulation framework are ideal for interactive animation and game/movie production. Copyright © 2005 John Wiley & Sons, Ltd.

We articulate a new modeling paradigm for both local and global editing on complicated point set surfaces of arbitrary topology. In essence, the proposed technique leads to a novel point-set methodology that can unify the topological advantage of the level-set methods and the simplicity of point-sampled surfaces. Any user-specified region of a point set surface in our system can be embedded into a grid-based level-set framework. The super-imposed grid structure enables both powerful local surface editing and global scalar-field free-form deformation anywhere across the point-sampled geometry. Furthermore, the underlying level-set representation, coupled with the concept of digital topology, greatly facilitates the topological modification of the sculpted point-set geometry whenever necessary during shape deformation. We have developed a variety of editing toolkits that can allow users to directly manipulate the point-set surface through interactive sketching, smoothing, embossing, and global free-form deformations with ease. We demonstrate the usefulness and efficacy of our prototype system for the point-sampled geometry via many examples.

The last few decades have seen enormous progress in both geometric subdivision, and physics-based simulation techniques. Mesh-based dynamic systems often require both subdivision and physical simulation for realistic and accurate results. However, the simulation parameters have been independent of the subdivision parameters, and vice-versa. This paper attempts to bridge this gap. We propose a hybrid approach that combines the physics-based simulation techniques and geometric subdivision algorithms, and demonstrate a mass-spring based system with physics-based butterfly subdivision. The initial subdivision coefficients are extracted using the physical properties of the base (L/sub 0/) mesh. Latter subdivision steps generate both the geometric and physical properties of the subdivided (L/sub k/) mesh. This approach conserves mass, center of gravity, linear momentum and external force, and minimizes the distance between the L/sub k/ and L/sub k+1/ meshes, at any time step. Our approach is general, efficient, and serve as a foundation for many applications in many fields.

This paper presents a modeling technique for reconstructing a triangular B-spline surface from a set of scanned 3D points. Unlike existing surface reconstruction methods based on tensor-product B-splines which primarily generate a network of patches and then enforce certain continuity (usually, G/sup 1/ or C/sup 1/) between adjacent patches, our algorithm can avoid the complicated procedures of surface trimming and patching. In our framework, the user simply specifies the degree n of the triangular B-spline surface and fitting error tolerance /spl epsi/. The surface reconstruction procedure generates a single triangular B-spline patch that has C/sup n-1/ continuity over smooth regions and C/sup 0/ on sharp features. More importantly, all the knots and control points are determined by minimizing a linear combination of interpolation and fairness functionals. Examples are presented which demonstrate the effectiveness of the technique for real data sets.

We study the reconstruction of smooth surfaces from point clouds. We use a new squared distance error term in optimization to fit a subdivision surface to a set of unorganized points, which defines a closed target surface of arbitrary topology. The resulting method is based on the framework of squared distance minimization (SDM) proposed by Pottmann et al. Specifically, with an initial subdivision surface having a coarse control mesh as input, we adjust the control points by optimizing an objective function through iterative minimization of a quadratic approximant of the squared distance function of the target shape. Our experiments show that the new method (SDM) converges much faster than the commonly used optimization method using the point distance error function, which is known to have only linear convergence. This observation is further supported by our recent result that SDM can be derived from the Newton method with necessary modifications to make the Hessian positive definite and the fact that the Newton method has quadratic convergence.

We present a novel surface reconstruction algorithm that can recover high-quality surfaces from noisy and defective data sets without any normal or orientation information. A set of new techniques is introduced to afford extra noise tolerability, robust orientation alignment, reliable outlier removal, and satisfactory feature recovery. In our algorithm, sample points are first organized by an octree. The points are then clustered into a set of monolithically singly-oriented groups. The inside/outside orientation of each group is determined through a robust voting algorithm. We locally fit an implicit quadric surface in each octree cell. The locally fitted implicit surfaces are then blended to produce a signed distance field using the modified Shepard's method. We develop sophisticated iterative fitting algorithms to afford improved noise tolerance both in topology recognition and geometry accuracy. Furthermore, this iterative fitting algorithm, coupled with a local model selection scheme, provides a reliable sharp feature recovery mechanism even in the presence of bad input.

Abstract This paper presents HapticFlow, a haptics‐based direct mesh editing system founded upon the concept of PDE‐based geometric surface flow. The proposed flow‐based approach for direct geometric manipulation offers a unified design paradigm that can seamlessly integrate implicit, distance‐field based shape modeling with dynamic, physics‐based shape design. HapticFlow provides an intuitive haptic interface and allows users to directly manipulate 3D polygonal objects with ease. To demonstrate the effectiveness of our new approach, we developed a variety of haptics‐based mesh editing operations such as embossing, engraving, sketching as well as force‐based shape manipulation operations. Copyright © 2004 John Wiley & Sons, Ltd.

This paper systematically presents a novel, interactive solid modeling framework, Haptics-based Dynamic Implicit Solid Modeling, which is founded upon volumetric implicit functions and powerful physics-based modeling. In particular, we augment our modeling framework with a haptic mechanism in order to take advantage of additional realism associated with a 3D haptic interface. Our dynamic implicit solids are semi-algebraic sets of volumetric implicit functions and are governed by the principles of dynamics, hence responding to sculpting forces in a natural and predictable manner. In order to directly manipulate existing volumetric data sets as well as point clouds, we develop a hierarchical fitting algorithm to reconstruct and represent discrete data sets using our continuous implicit functions, which permit users to further design and edit those existing 3D models in real-time using a large variety of haptic and geometric toolkits, and visualize their interactive deformation at arbitrary resolution. The additional geometric and physical constraints afford more sophisticated control of the dynamic implicit solids. The versatility of our dynamic implicit modeling enables the user to easily modify both the geometry and the topology of modeled objects, while the inherent physical properties can offer an intuitive haptic interface for direct manipulation with force feedback.

Mapping of features such as texture and geometric detail is an important tool that enhances realism of surface geometry in graphics and animation. This essentially involves a transformation of 2D coordinates with associated attributes to a target surface in 3D. Moreover, users are often in need of more powerful techniques that can also enable cut-and-paste functionality to directly map a feature from one surface in 3D to another with high-fidelity. Such an operation should minimize feature deformation on the target surface. In practice, it is also desirable to hide geometric complexities and transformations from ordinary users, requiring minimal input and providing an intuitive interface for painting. This paper develops a generalized feature mapping technique using a physically based particle system to map both geometry and associative attributes between two curved surfaces with the physically correct minimum distortion of these attributes over the target surface.

This paper presents a novel paradigm for point set surface editing, which takes advantages of the potential of implicit surfaces, the strength of physics based modeling techniques, and the simplicity of point sampled surfaces. Our point set surface is evaluated as the zero set of the weighted sum of the collection of the scalar trivariate B-spline functions defined over the local domain of each point sample. The implicit representation of the point set surfaces allows the user to easily modify the topology of the sculpted objects. The deformation of the surfaces is conducted by dynamically modifying the local reference domains, as well as their scalar control coefficients. We have developed a variety of sculpting toolkits that can dynamically manipulate the implicit point set surface and easily perform CSG Boolean operation on arbitrarily shaped objects. Our research work complements existing point rendering and modeling pipelines for efficient interactive sculpting and deformation.

Subdivision has gained popularity in computer graphics and shape modeling during the past two decades, yet volumetric subdivision has received much less attention. In this paper, we develop a new subdivision scheme, which can interpolate all of the initial control points in 3D and generate a continuous volume in the limit. We devise a set of solid subdivision rules to facilitate a simple subdivision procedure. The conversion between the subdivided mesh and a simplicial complex is straightforward and effective, which can be directly utilized in solid meshing, finite element simulation, and other numerical processes. In principle, our solid subdivision process is a combination of simple linear interpolations in 3D. Affine operations of neighboring control points produce new control points in the next level, yet inherit the original control points and achieve the interpolatory effect. A parameter is offered to control the tension between control points. The interpolatory property of our solid subdivision offers many benefits, which are desirable in many design applications and physics simulations, including intuitive manipulation on control points and ease of constraint enforcement in numerical procedures. We outline a proof that can guarantee the convergence and C/sup 1/ continuity of our volumetric subdivision and limit volumes in regular cases. In addition to solid subdivision, we derive special rules to generate C/sup 1/ surfaces as B-reps and to model shapes of non-manifold topology. Several examples demonstrate the ability of our subdivision to handle complex manifolds easily. Numerical experiments and future research suggestions for extraordinary cases are also presented.

This paper addresses the problem of surface reconstruction of highly noisy point clouds. The surfaces to be reconstructed are assumed to be 2-manifolds of piecewise C/sup 1/ continuity, with isolated small irregular regions of high curvature, sophisticated local topology or abrupt burst of noise. At each sample point, a quadric field is locally fitted via a modified moving least squares method. These locally fitted quadric fields are then blended together to produce a pseudo-signed distance field using Shepard's method. We introduce a prioritized front growing scheme in the process of local quadrics fitting. Flatter surface areas tend to grow faster. The already fitted regions will subsequently guide the fitting of those irregular regions in their neighborhood.

We introduce a method for the animation of fire propagation and the burning consumption of objects represented as volumetric data sets. Our method uses a volumetric fire propagation model based on an enhanced distance field. It can simulate the spreading of multiple fire fronts over a specified isosurface without actually having to create that isosurface. The distance field is generated from a specific shell volume that rapidly creates narrow spatial bands around the virtual surface of any given isovalue. The complete distance field is then obtained by propagation from the initial bands. At each step multiple fire fronts can evolve simultaneously on the volumetric object. The flames of the fire are constructed from streams of particles whose movement is regulated by a velocity field generated with the hardware-accelerated Lattice Boltzmann Model (LBM). The LBM provides a physically-based simulation of the air flow around the burning object. The object voxels and the splats associated with the flame particles are rendered in the same pipeline so that the volume data with its external and internal structures can be displayed along with the fire.

The recent non-uniform subdivision approach extends traditional uniform subdivision schemes with variable rules, offering additional shape parameters (such as knot spacings) for feature control. Despite its flexibility, shape modification based on non-uniform subdivision usually requires designers to interactively adjust a large number of degrees of freedom (DOFs) to achieve the desired shapes, which can often be laborious. This paper extends the principle of variational subdivision and integrates the non-uniform subdivision schemes with powerful physics-based shape sculpting techniques, providing a universal method for arbitrary subdivision schemes with adjustable rules. The subdivision control points and knot spacings evolve in response to the shape deformation resulting from the numerical integration of Lagrangian dynamics equation or the optimization of shape energy functional. Thus, our system allows users to manipulate the desired shape in a direct and intuitive fashion. In addition, we propose a novel and efficient discrete functional evaluation method for polygonal meshes or point clouds of arbitrary topology based on implicit functions, in which no parameterization is needed. Finally, we develop a simple prototype sculpting system demonstrating many advantages of our novel physics-based, non-uniform subdivision modeling system.

We present a novel, interactive shape modeling technique: dynamic implicit solid modeling, which unifies volumetric implicit functions and powerful physics-based modeling. Although implicit functions are extremely powerful in graphics, geometric design, and shape modeling, the full potential of implicit functions is yet to be fully realized due to the lack of flexible and interactive design techniques. In order to broaden the accessibility of implicit functions in geometric modeling, we marry the implicit solids, which are semi-algebraic sets of volumetric implicit functions, with the principle of physics-based models and formulate dynamic implicit solids. By using "density springs" to connect the scalar values of implicit functions, we offer a viable solution to introduce the elasticity into implicit representations. As a result, our dynamic implicit solids respond to sculpting forces in a natural and predictive manner. The geometric and physical behaviors are tightly coupled in our modeling system. The flexibility of our modeling technique allows users to easily modify the geometry and topology of sculpted objects, while the inherent physical properties can provide a natural interface for direct, force-based free-form deformation. The additional constraints provide users more control on the dynamic implicit solids. We have developed a sculpting system equipped with a large variety of physics-based toolkits and an intuitive haptic interface to facilitate the direct, natural editing of implicit functions in real-time. Our experiments demonstrate many attractive advantages of our dynamic approach for implicit modeling such as intuitive control, direct manipulation, real-time haptic feedback, and capability to model complicated geometry and arbitrary topology.

PDE techniques, which use partial differential equations (PDEs) to model the shapes of various real-world objects, can unify their geometric attributes and functional constraints in geometric computing and graphics. This paper presents a unified dynamic approach that allows modelers to define the solid geometry of sculptured objects using the second-order or fourth-order elliptic PDEs subject to flexible boundary conditions. Founded upon the previous work on PDE solids by Bloor and Wilson (1989, 1990, 1993), as well as our recent research on the interactive sculpting of physics-based PDE surfaces, our new formulation and its associated dynamic principle permit designers to directly deform PDE solids whose behaviors are natural and intuitive subject to imposed constraints. Users can easily model and interact with solids of complicated geometry and/or arbitrary topology from locally-defined PDE primitives through trimming operations. We employ the finite-difference discretization and the multi-grid subdivision to solve the PDEs numerically. Our PDE-based modeling software offers users various sculpting toolkits for solid design, allowing them to interactively modify the physical and geometric properties of arbitrary points, curve spans, regions of interest (either in the isoparametric or nonisoparametric form) on boundary surfaces, as well as any interior parts of modeled objects.

Implicit functions characterized by the zero-set of polynomial-based algebraic equations and other commonly-used analytic equations are extremely powerful in graphics, geometric design, and visualization. But the potential of implicit functions is yet to be fully realized due to the lack of flexible and interactive design techniques. The paper presents a haptic sculpting system founded upon scalar trivariate B-spline functions. All the solids sculpted in our environment are semi-algebraic sets of volumetric implicit functions. We develop a large variety of sculpting toolkits equipped with an intuitive haptic interface to facilitate the direct manipulation of implicit functions in real-time. To facilitate multiresolution editing and different levels of details, we employ three techniques: hierarchical B-splines, CSG-based functional composition, and knot insertion. Our experiments demonstrate that our algorithms and haptics-based techniques can greatly overcome the modeling difficulties associated with implicit functions. The novel modeling techniques and their haptics-based design principle are extensible to the design of arbitrary implicit functions.

Article Share on Virtual clay: a real-time sculpting system with haptic toolkits Authors: Kevin T. McDonnell State Univ. of New York, Stony Brook State Univ. of New York, Stony BrookView Profile , Hong Qin State Univ. of New York, Stony Brook State Univ. of New York, Stony BrookView Profile , Robert A. Wlodarczyk State Univ. of New York, Stony Brook State Univ. of New York, Stony BrookView Profile Authors Info & Claims I3D '01: Proceedings of the 2001 symposium on Interactive 3D graphicsMarch 2001 Pages 179–190https://doi.org/10.1145/364338.364395Online:01 March 2001Publication History 92citation1,570DownloadsMetricsTotal Citations92Total Downloads1,570Last 12 Months38Last 6 weeks7 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access

This paper presents a novel modeling technique and develops an interactive algorithm that facilitates the automatic determination of non-uniform knot vectors as well as other control variables for NURBS curves and surfaces through the unified methodology of energy minimization, variational principle, and numerical techniques. Many geometric algorithms have been developed for NURBS during the past three decades. Recently, the optimization principle has been widely studied, which affords designers to interactively manipulate NURBS via energy functionals, simulated forces, qualitative and quantitative constraints, etc. The existing techniques primarily concentrate on NURBS control points. In this paper we further augment our NURBS modeling capabilities by incorporating NURBS' non-uniform knot sequence into our shape parameter set. The automatic determination of NURBS knots will facilitate the realization of the full geometric potential of NURBS. We also have developed a modeling framework which supports a large variety of functionals ranging from simple quadratic energy forms to non-linear curvature-based (or area-based) objective functionals.

In this paper, we present hierarchical D-NURBS as a new shape modeling representation which generalizes powerful, physics-based D-NURBS for interactive geometric design. Our hierarchical D-NURBS can be viewed as a collection of standard D-NURBS finite elements, organized hierarchically in a tree structure and subject to continuity constraints across the shared boundaries of adjacent D-NURBS elements at different levels. The layered and composite construction of hierarchical D-NURBS affords users the effective creation of local features and their flexible, global/local control at different level of details. Within the framework of hierarchical D-NURBS, users can interactively sculpt NURBS geometry more intuitively and conveniently through both global and local toolkits. Based on the data structure of hierarchical D-NURBS, we have developed a prototype software equipped with various physics based toolkits in the form of geometric constraints and simulated forces. Our modeling system allows users to undertake the design tasks of point manipulation, normal editing, curvature control, curve fitting, and area sculpting in interactive graphics and CAD/CAM.

This paper presents a sculptured solid modeling system founded upon dynamic Catmull-Clark subdivision-based solids of arbitrary topology. Our primary contribution is that we integrate the geometry of sculptured free-form solids with the powerful physics-based modeling framework by augmenting pure geometric entities with material properties such as mass, damping, and stiffness distributions and with physical behaviors such as elasticity, plasticity, and natural deformation under external forces. Our novel dynamic model of free-form solids frees users from having to deal with low-level central point operations and permits them to interact with subdivision-based virtual clay in a more natural and intuitive fashion via "forces".

PDE surfaces, whose behavior is governed by partial differential equations (PDEs), have demonstrated many modeling advantages in surface blending, free-form surface modeling, and surface aesthetic or functional specifications. Although PDE surfaces can potentially unify geometric attributes and functional constraints for surface design, current PDE based techniques exhibit certain difficulties such as the restrained topological structure of modeled objects and the lack of interactive editing functionalities. We propose an integrated approach and develop a set of algorithms that augment conventional PDE surfaces with material properties and dynamic behavior. The authors incorporate PDE surfaces into the powerful physics based framework, aiming to realize the full potential of the PDE methodology. We have implemented a prototype software environment that can offer users a wide array of PDE surfaces with flexible topology (through trimming and joining operations) as well as generalized boundary constraints. Using our system, designers can dynamically manipulate PDE surfaces at arbitrary location with applied forces. Our sculpting toolkits allow users to interactively modify arbitrary point, curve span, and/or region of interest throughout the entire PDE surface in an intuitive and predictable way. To achieve real time sculpting, we employ several simple, yet efficient numerical techniques such as finite difference discretization, multi-grid subdivision, and FEM approximation. Our experiments demonstrate many advantages of physics based PDE formulation such as intuitive control, real time feedback, and usability to both professional and non-expert users.

Recent years have witnessed a dramatic growth in the use of subdivision schemes for graphical modeling and animation, especially for the representation of smooth, often complex, shapes of arbitrary topology. Nevertheless, conventional interactive approaches to subdivision objects can be extremely laborious and inefficient. Users must carefully specify the initial mesh and/or painstakingly manipulate the control vertices at different levels of the subdivision hierarchy to satisfy a diverse set of functional requirements and aesthetic criteria in the modeled object. This modeling drawback results from the lack of direct manipulation tools for the limit geometric shape. To improve the efficiency of interactive design, we have developed a unified finite element method (FEM) based dynamic methodology for arbitrary subdivision schemes by marrying principles of computational physics and finite element analysis with powerful subdivision geometry. Our dynamic framework permits users to directly manipulate the limit surface obtained from any subdivision procedure via simulated "force" tools. Our experiments demonstrate that the new unified FEM-based framework promises a greater potential for subdivision techniques in geometric modeling, finite element analysis, engineering design, computer graphics and other visual computing applications.

This paper presents an integrated approach and a unified algorithm that combine the benefits of PDE surfaces and powerful physics‐based modeling techniques within one single modeling framework, in order to realize the full potential of PDE surfaces. We have developed a novel system that allows direct manipulation and interactive sculpting of PDE surfaces at arbitrary location, hence supporting various interactive techniques beyond the conventional boundary control. Our prototype software affords users to interactively modify point, normal, curvature, and arbitrary region of PDE surfaces in a predictable way. We employ several simple, yet effective numerical techniques including the finite‐difference discretization of the PDE surface, the multigrid‐like subdivision on the PDE surface, the mass‐spring approximation of the elastic PDE surface, etc. to achieve real‐time performance. In addition, our dynamic PDE surfaces can also be approximated using standard bivariate B‐spline finite elements, which can subsequently be sculpted and deformed directly in real‐time subject to intrinsic PDE constraints. Our experiments demonstrate many attractive advantages of our dynamic PDE formulation such as intuitive control, real‐time feedback, and usability to the general public.

The authors develop integrated techniques that unify physics based modeling with geometric subdivision methodology and present a scheme for dynamic manipulation of the smooth limit surface generated by the (modified) butterfly scheme using physics based "force" tools. This procedure based surface model obtained through butterfly subdivision does not have a closed form analytic formulation (unlike other well known spline based models), and hence poses challenging problems to incorporate mass and damping distributions, internal deformation energy, forces, and other physical quantities required to develop a physics based model. Our primary contributions to computer graphics and geometric modeling include: (1) a new hierarchical formulation for locally parameterizing the butterfly subdivision surface over its initial control polyhedron, (2) formulation of dynamic butterfly subdivision surface as a set of novel finite elements, and (3) approximation of this new type of finite elements by a collection of existing finite elements subject to implicit geometric constraints. Our new physics based model can be sculpted directly by applying synthesized forces and its equilibrium is characterized by the minimum of a deformation energy subject to the imposed constraints. We demonstrate that this novel dynamic framework not only provides a direct and natural means of manipulating geometric shapes, but also facilitates hierarchical shape and nonrigid motion estimation from large range and volumetric data sets using very few degrees of freedom (control vertices that define the initial polyhedron).

Conventional free-form surface design usually require tedious control-point manipulation and/or painstaking constraint specification via unnatural mouse-based interfaces. This paper presents a novel haptic approach for the direct manipulation of physics-based B-spline surfaces. Our method permits users to interactively sculpt virtual yet real material with a standard haptic device, and feel the physically realistic presence of virtual B-spline objects with force feedback throughout the design process. We aim to develop various haptic sculpting tools to expedite the direct manipulation of B-spline surfaces with haptic feedback and constraints. One significant contribution of this paper is that point, normal, and curvature constraints can be specified interactively and modified naturally using forces. We propose and formulate a dual representation for Bspline surfaces in both physical and mathematical space. This massspring model is mathematically constrained by the B-spline surface throughout the sculpting session. The equations of motion controlling the physical behavior of the B-spline surface are solved using a tractable numerical solver in real-time. The integration of haptics with traditional geometric modeling will increase the bandwidth of human-computer interaction, and thus shorten the time-consuming design cycle. We envision that this integrated approach promises a much greater potential in computer-integrated design and manufacturing, haptic interface, interactive graphics, medical applications, and virtual environments.

A new dynamic subdivision surface model is proposed for shape recovery from 3D data sets. The model inherits the attractive properties of the Catmull-Clark subdivision scheme and is set in a physics-based modeling paradigm. Unlike other existing methods, our model does not require a parameterized input mesh to recover shapes of arbitrary topology, allows direct manipulation of the limit surface via application of forces and provides a fast, robust, and hierarchical approach to recover complex shapes from 3D data with very few degrees of freedom (control vertices). We provide an analytic formulation and introduce the physical quantities required to develop the dynamic subdivision surface model which can be deformed by applying forces synthesized from the data. Our experiments demonstrate that this new dynamic model has a promising future in shape recovery from volume and range data sets.

Recursive subdivision schemes have been extensively used in computer graphics, computer-aided geometric design, and scientific visualization for modeling smooth surfaces of arbitrary topology. Recursive subdivision generates a visually pleasing smooth surface in the limit from an initial user-specified polygonal mesh through the repeated application of a fixed set of subdivision rules. We present a new dynamic surface model based on the Catmull-Clark subdivision scheme, a popular technique for modeling complicated objects of arbitrary genus. Our new dynamic surface model inherits the attractive properties of the Catmull-Clark subdivision scheme, as well as those of the physics-based models. This new model provides a direct and intuitive means of manipulating geometric shapes, and an efficient hierarchical approach for recovering complex shapes from large range and volume data sets using very few degrees of freedom (control vertices). We provide an analytic formulation and introduce the "physical" quantities required to develop the dynamic subdivision surface model which can be interactively deformed by applying synthesized forces. The governing dynamic differential equation is derived using Lagrangian mechanics and the finite element method. Our experiments demonstrate that this new dynamic model has a promising future in computer graphics, geometric shape design, and scientific visualization.

Recursive subdivision schemes have been extensively used in computer graphics and scientific visualization for modeling smooth surfaces of arbitrary topology. Recursive subdivision generates a visually pleasing smooth surface in the limit from an initial user-specified polygonal mesh through the repeated application of a fixed set of subdivision rules. In this paper, we present a new dynamic surface model based on the Catmull-Clark (1978) subdivision scheme, which is a very popular method to model complicated objects of arbitrary genus because of many of its nice properties. Our new dynamic surface model inherits the attractive properties of the Catmull-Clark subdivision scheme as well as that of the physics-based modeling paradigm. This new model provides a direct and intuitive means of manipulating geometric shapes, a fast, robust and hierarchical approach for recovering complex geometric shapes from range and volume data using very few degrees of freedom (control vertices). We provide an analytic formulation and introduce the physical quantities required to develop the dynamic subdivision surface model which can be interactively deformed by applying synthesized forces in real time. The governing dynamic differential equation is derived using Lagrangian mechanics and a finite element discretization. Our experiments demonstrate that this new dynamic model has a promising future in computer graphics, geometric shape design and scientific visualization.

Presents dynamic non-uniform rational B-splines (D-NURBS), a physics-based generalization of NURBS. NURBS have become a de facto standard in commercial modeling systems. Traditionally, however, NURBS have been viewed as purely geometric primitives, which require the designer to interactively adjust many degrees of freedom-control points and associated weights-to achieve the desired shapes. The conventional shape modification process can often be clumsy and laborious. D-NURBS are physics-based models that incorporate physical quantities into the NURBS geometric substrate. Their dynamic behavior, resulting from the numerical integration of a set of nonlinear differential equations, produces physically meaningful, and hence intuitive shape variation. Consequently, a modeler can interactively sculpt complex shapes to required specifications not only in the traditional indirect fashion, by adjusting control points and setting weights, but also through direct physical manipulation, by applying simulated forces and local and global shape constraints. We use Lagrangian mechanics to formulate the equations of motion for D-NURBS curves, tensor-product D-NURBS surfaces, swung D-NURBS surfaces and triangular D-NURBS surfaces. We apply finite element analysis to reduce these equations to efficient numerical algorithms computable at interactive rates on common graphics workstations. We implement a prototype modeling environment based on D-NURBS and demonstrate that D-NURBS can be effective tools in a wide range of computer-aided geometric design (CAGD) applications.

This article develops a dynamic generalization of the nonuniform rational B-spline (NURBS) model. NURBS have become a defacto standard in commercial modeling systems because of their power to represent free-form shapes as well as common analytic shapes. To date, however, they have been viewed as purely geometric primitives that require the user to manually adjust multiple control points and associated weights in order to design shapes. Dynamic NURBS, or D-NURBS, are physics-based models that incorporate mass distributions, internal deformation energies, and other physical quantities into the popular NURBS geometric substrate. Using D-NURBS, a modeler can interactively sculpt curves and surfaces and design complex shapes to required specifications not only in the traditional indirect fashion, by adjusting control points and weights, but also through direct physical manipulation, by applying simulated forces and local and global shape constraints. D-NURBS move and deform in a physically intuitive manner in response to the user's direct manipulations. Their dynamic behavior results from the numerical integration of a set of nonlinear differential equations that automatically evolve the control points and weights in response to the applied forces and constraints. To derive these equations, we employ Lagrangian mechanics and a finite-element-like discretization. Our approach supports the trimming of D-NURBS surfaces using D-NURBS curves. We demonstrate D-NURBS models and constraints in applications including the rounding of solids, optimal surface fitting to unstructured data, surface design from cross sections, and free-form deformation. We also introduce a new technique for 2D shape metamorphosis using constrained D-NURBS surfaces.