In cooperative video games, traditional AI companions are deployed to assist players, who control them using hotkeys or command wheels to issue predefined commands such as ''attack'', ''defend'', or ''retreat''. Despite their simplicity, these methods, which lack target specificity, limit players' ability to give complex tactical instructions and hinder immersive gameplay experiences. To address this, we propose the FPS AI Companion who Understands Language (F.A.C.U.L.), the first real-time AI system that enables players to communicate and collaborate with AI companions using natural language. By integrating natural language processing with a confidence-based framework, F.A.C.U.L. efficiently decomposes complex commands and interprets player intent. It also employs a dynamic entity retrieval method for environmental awareness, aligning human intentions with decision-making. Unlike traditional rule-based systems, our method supports real-time language interactions, enabling players to issue complex commands such as ''clear the second floor,'' ''take cover behind that tree,'' or ''retreat to the river''. The system provides real-time behavioral responses and vocal feedback, ensuring seamless tactical collaboration. Using the popular FPS game Arena Breakout: Infinite as a case study, we present comparisons demonstrating the efficacy of our approach and discuss the advantages and limitations of AI companions based on real-world user feedback.

Abstract To address the storage overhead of lightmaps and the limitations of existing compression techniques, we propose a novel UV‐space compression framework based on per‐triangle processing. By mapping triangles to a standardized domain, we cluster and repack color‐coherent regions into a compact atlas, generating a cascade texture refined via differentiable rendering. Experimental results show an average storage reduction of 83% with approximately 10 dB higher PSNR than existing methods. Our approach is the first dedicated lightmap compression framework compatible with standard block‐based formats, offering an effective solution for memory‐efficient 3D asset delivery.

ABSTRACT Generating diverse and realistic movements has long been a central challenge in computer graphics. Generative Adversarial Networks (GANs) remain a compelling solution due to their ability to perform well even with limited training data. However, traditional GANs generate samples directly, which can lead to the omission of certain data patterns. To address this limitation, we introduce SinMDGan , a hybrid deep learning framework for single‐motion synthesis that leverages a Diffusion‐GAN model. Our approach integrates the strengths of GANs, which capture global motion characteristics, with diffusion techniques, which refine local details, ensuring both authenticity and diversity in generated movements. Unlike conventional cascaded GANs, our framework employs a single generator‐discriminator pair, utilizing different diffusion time steps to synthesize novel and diverse motions from a single short sequence. Experimental evaluations demonstrate the effectiveness of our model in achieving stable data distribution coverage and enhancing output diversity. Additionally, we showcase various applications, including motion composition and long‐sequence generation, highlighting the versatility of our approach.

Modeling and simulating realistic insect flight pose unique challenges due to the complex interaction between multi-degree-of-freedom wing kinematics and highly precise aerodynamic forces. To solve this challenge, this article presents a bidirectional kinematics-aerodynamics coupled simulation framework for miniature insect flight. Our approach first models the kinematics of flying insects by parameterizing natural wingbeat cycles based on available real-world datasets. Subsequently, we compute aerodynamic forces utilizing an improved semi-empirical model, which extends from quasi-steady formulation by incorporating critical unsteady force components. To achieve closed-loop control for both kinematics and aerodynamics, we employ deep reinforcement learning to train a virtual insect to adaptively adjust flapping strategies in response to dynamic flight states. Finally, an integrated controller enables the simulated insect to autonomously regulate the wing motion and perform complex tasks such as visual obstacle avoidance. Extensive experiments and comparisons demonstrate that our framework can effectively generate physically plausible and autonomous insect flight across a variety of scenarios.

Using parts of existing models to rebuild new models, commonly termed as example-based modeling, is a classical methodology in the realm of computer graphics. Previous works mostly focus on shape composition, making them very hard to use for realistic composition of 3D objects captured from real-world scenes. This leads to combining multiple NeRFs into a single 3D scene to achieve seamless appearance blending. However, the current SeamlessNeRF method struggles to achieve interactive editing and harmonious stitching for real-world scenes due to its gradient-based strategy and grid-based representation. To this end, we present an example-based modeling method that combines multiple Gaussian fields in a point-based representation using sample-guided synthesis. Specifically, as for composition, we create a GUI to segment and transform multiple fields in real time, easily obtaining a semantically meaningful composition of models represented by 3D Gaussian Splatting (3DGS). For texture blending, due to the discrete and irregular nature of 3DGS, straightforwardly applying gradient propagation as SeamlssNeRF is not supported. Thus, a novel sampling-based cloning method is proposed to harmonize the blending while preserving the original rich texture and content. Our workflow consists of three steps: 1) real-time segmentation and transformation of 3DGS using a well-tailored GUI, 2) KNN analysis to identify boundary points in the intersecting area between the source and target models, and 3) two-phase optimization of the target model using sampling-based cloning and gradient constraints. Extensive experimental results validate that our approach significantly outperforms previous works in realistic synthesis, demonstrating its practicality.

Numerous researches on real-time motion generation primarily focus on kinematic aspects, often resulting in physically implausible outcomes. In this paper, we present POMP ("Physics-cOnstrainable Motion Generative Model through Phase Manifolds"), a kinematics-based framework that synthesizes physically realistic motions by leveraging phase manifolds to align motion priors with physics constraints. POMP operates as a frame-by-frame autoregressive model with three core components: a diffusion-based kinematic module, a simulation-based dynamic module, and a phase encoding module. At each timestep, the kinematic module first generates an initial pose, which is subsequently revised by the dynamic module through a simulation step to incorporate physical constraints. While individual simulation steps induce negligible kinematic distortion, accumulated discrepancies can drive the result beyond the motion prior learned by the kinematic module, leading to failure in subsequent motion generation. To address this, the phase encoding module applies semantic alignment in the phase manifold, projecting the simulated result back to the motion prior. Moreover, we present a pipeline in Unity for generating terrain maps and capturing full-body motion impulses from existing motion capture dataset. The collected terrain topology and motion impulse data facilitate the training of POMP, enabling it to robustly respond to underlying contact forces and applied dynamics. Extensive evaluations demonstrate the efficacy of POMP across various tasks.

Sewing patterns, the essential blueprints for fabric cutting and tailoring, act as a crucial bridge between design concepts and producible garments. However, existing uni-modal sewing pattern generation models struggle to effectively encode complex design concepts with a multimodal nature and correlate them with vectorized sewing patterns that possess precise geometric structures and intricate sewing relations. In this work, we propose a novel sewing pattern generation approach Design2GarmentCode based on Large Multimodal Models (LMMs), to generate parametric pattern-making programs from multi-modal design concepts. LMM offers an intuitive interface for interpreting diverse design inputs, while pattern-making programs could serve as well-structured and semantically meaningful representations of sewing patterns, and act as a robust bridge connecting the cross-domain pattern-making knowledge embedded in LMMs with vectorized sewing patterns. Experimental results demonstrate that our method can flexibly handle various complex design expressions such as images, textual descriptions, designer sketches, or their combinations, and convert them into size-precise sewing patterns with correct stitches. Compared to previous methods, our approach significantly enhances training efficiency, generation quality, and authoring flexibility. Project page: https://style3d.github.io/design2garmentcode.

Automated LEGO® design is challenging due to the extensive variety of LEGO® brick types and the necessity of constructing semantically meaningful models from individually meaningless components. Current automatic LEGO® generation methods face two key challenges: i) They typically rely on explicit modeling of brick connectivity to ensure structural validity. However, this requires extensive manual annotation, which is labor-intensive as the variety of LEGO® primitives increases. This limits training data diversity, restricting the variety of LEGO® bricks that can be effectively utilized. ii) To facilitate learning within neural networks, current methods often employ either volume or text-based descriptions to represent LEGO® models. However, volumetric representations are computationally expensive and hamper large-scale generative training, while text-based approaches rely on large language models and dedicated text-to-brick mapping rules, introducing a semantic gap between language tokens and 3D brick structures.

Abstract Fast and highly realistic multi‐view hair transfer plays a crucial role in evaluating the effectiveness of virtual hair try‐on systems. However, GAN‐based generation and editing methods face persistent challenges in feature disentanglement. Achieving pixel‐level, attribute‐specific modifications—such as changing hairstyle or hair color without affecting other facial features—remains a long‐standing problem. To address this limitation, we propose a novel multi‐view hair transfer framework that leverages a hair‐only intermediate facial representation and a 3D‐guided masking mechanism. Our approach disentangles tri‐plane facial features into spatial geometric components and global style descriptors, enabling independent and precise control over hairstyle and hair color. By introducing a dedicated intermediate representation focused solely on hair and incorporating a two‐stage feature fusion strategy guided by the generated 3D mask, our framework achieves fine‐grained local editing across multiple viewpoints while preserving facial integrity and improving background consistency. Extensive experiments demonstrate that our method produces visually compelling and natural results in side‐to‐front view hair transfer tasks, offering a robust and flexible solution for high‐fidelity hair reconstruction and manipulation.

Image inpainting is the task of filling in missing or masked regions of an image with semantically meaningful content. Recent methods have shown significant improvement in dealing with large missing regions. However, these methods usually require large training datasets to achieve satisfactory results, and there has been limited research into training such models on a small number of samples. To address this, we present a novel data-efficient generative residual image inpainting method that produces high-quality inpainting results. The core idea is to use an iterative residual reasoning method that incorporates convolutional neural networks (CNNs) for feature extraction and transformers for global reasoning within generative adversarial networks, along with image-level and patch-level discriminators. We also propose a novel forged-patch adversarial training strategy to create faithful textures and detailed appearances. Extensive evaluation shows that our method outperforms previous methods on the data-efficient image inpainting task, both quantitatively and qualitatively.

ABSTRACT Though current traffic simulation methods can produce impressive results, reversing behavior is always ignored, potentially reducing the diversity and plausibility of simulation data. Furthermore, while common traffic behaviors like following‐the‐leader and lane changing can be easily simulated, efficiently generating irregular cases in a human‐in‐the‐loop manner with specific motions based on user desires is less discussed. To address the gap, we present a novel interactive traffic editing and simulation framework that enables users to regulate vehicles via simple inputs to introduce reversing and generate desired trajectory data with both car‐following and irregular driving behaviors. With key states specified, lane‐level navigation, including forward/backward directions, is planned through heuristic search. The customized navigation brings the vehicles' new trajectories with both car‐following and reversing, and their surrounding neighbors are also adjusted accordingly. To provide smooth and plausible motions after editing, vehicles are updated via the optimization‐based simulation method, which takes vehicle kinematics, self‐motivation, path keeping, collision avoidance, and special interaction rules into account. We demonstrate that our framework can generate uncommon traffic cases and validate it through extensive experiments.

ABSTRACT High‐fidelity 3D face morphing aims to achieve seamless transitions between realistic 3D facial representations of different identities. Although 3D Gaussian Splatting (3DGS) excels in high‐quality rendering, its application to morphing is hindered by the lack of Gaussian primitive correspondence and variations in primitive quantities. To address this, we propose GSFaceMorpher, which is a novel framework for high‐fidelity 3D face morphing based on 3DGS. Our method constructs an auxiliary model that bridges the source and target face models by aligning the geometry through Radial Basis Function (RBF) warping and optimizing the appearance in the image space. This auxiliary model enables smooth parameter interpolation, whereas a diffusion‐based refinement step enhances critical facial details through attention replacement from the reference faces. Experiments demonstrate that our method produces visually coherent and high‐fidelity morphing sequences, significantly outperforming NeRF‐based baselines in terms of both quantitative metrics and user preferences. Our work establishes a new benchmark for high‐fidelity 3D face morphing with applications in visual effects, animation, and immersive experiences.

3D-aware face generators are typically trained on 2D real-life face image datasets that primarily consist of near-frontal face data. Due to data limitations, these generators cannot generateone-quarter headshot3D portraits with head, neck, and shoulder geometry, which is crucial for applications like talking heads. Two reasons account for this issue: First, existing facial recognition methods struggle with extracting facial data captured from large camera angles or back views. Second, it is challenging to learn a distribution of 3D portraits covering the one-quarter headshot region from single-view data due to significant geometric deformation caused by diverse body poses. To this end, we first create the dataset360°-Portrait-HQ(360°PHQfor short) which consists of high-quality single-view real portraits annotated with a variety of camera parameters (the yaw angles span the entire 360° range) and body poses. We then propose3DPortraitGAN, the first 3D-aware one-quarter headshot portrait generator that learns a canonical 3D avatar distribution from the360°PHQ dataset with body pose self-learning. Our model can generate view-consistent portrait images from all camera angles with a canonical one-quarter headshot 3D representation. Our experiments show that the proposed framework can accurately predict portrait body poses and generate view-consistent, realistic portrait images with complete geometry from all camera angles.

Geometry-aware online motion retargeting is crucial for real-time character animation in gaming and virtual reality. However, existing methods often rely on complex optimization procedures or deep neural networks, which constrain their applicability in real-time scenarios. Moreover, they offer limited control over fine-grained motion details involved in character interactions, resulting in less realistic outcomes. To overcome these limitations, we propose a novel optimization framework for ultrafast, lightweight motion retargeting with joint-level control (i.e., controls over joint position, bone orientation, etc,). Our approach introduces a semantic-aware objective grounded in a spherical geometry representation, coupled with a bone-length-preserving algorithm that iteratively solves this objective. This formulation preserves spatial relationships among spheres, thereby maintaining motion semantics, mitigating interpenetration, and ensuring contact. It is lightweight and computationally efficient, making it particularly suitable for time-critical real-time deployment scenarios. Additionally, we incorporate a heuristic optimization strategy that enables rapid convergence and precise joint-level control. We evaluate our method against state-of-the-art approaches on the Mixamo dataset, and experimental results demonstrate that it achieves comparable performance while delivering an order-of-magnitude speedup.

Realistic digital garment modeling remains a labor-intensive task due to the intricate process of translating 2D sewing patterns into high-fidelity, simulation-ready 3D garments. We introduce GarmageNet , a unified generative framework that automates the creation of 2D sewing patterns, the construction of sewing relationships, and the synthesis of 3D garment initializations compatible with physics-based simulation. Central to our approach is Garmage , a novel garment representation that encodes each panel as a structured geometry image, effectively bridging the semantic and geometric gap between 2D structural patterns and 3D garment geometries. Followed by GarmageNet , a latent diffusion transformer to synthesize panel-wise geometry images and GarmageJigsaw , a neural module for predicting point-to-point sewing connections along panel contours. To support training and evaluation, we build GarmageSet , a large-scale dataset comprising 14,801 professionally designed garments with detailed structural and style annotations. Our method demonstrates versatility and efficacy across multiple application scenarios, including scalable garment generation from multi-modal design concepts (text prompts, sketches, photographs), automatic modeling from raw flat sewing patterns, pattern recovery from unstructured point clouds, and progressive garment editing using conventional instructions, laying the foundation for fully automated, production-ready pipelines in digital fashion. Refer to our project page for open-sourced code and dataset.

Approximate Convex Decomposition (ACD) aims to approximate complex 3D shapes with convex components, which is widely applied to create compact collision representations for real-time applications, including VR/AR, interactive games, and robotic simulations. Efficiency and optimality are critical for ACD algorithms in approximating large-scale, complex 3D shapes, enabling high-quality decompositions with minimal components. Unfortunately, existing methods either employ sub-optimal greedy strategies or rely on computationally intensive multi-step searches. In this work, we propose RL-ACD, a data-driven, reinforcement learning-based approach for efficient and near-optimal convex shape decomposition. We formulate ACD as a Markov Decision Process (MDP), where cutting planes are iteratively applied based on the current stage's mesh fragments rather than the entire fine-grained mesh, leading to a novel, efficient geometric encoding. To train near-optimal policies for ACD, we propose a novel dual-state Bellman loss and analyze its convergence using a Q-learning algorithm. Comprehensive evaluations across diverse datasets validate the efficiency and accuracy of RL-ACD for convex decomposition tasks. Our method outperforms the multi-step tree search by 15× in terms of computational speed, while reducing the number of resulting components by 16% compared to the current state-of-the-art greedy algorithms, significantly narrowing the sub-optimality gap and enhancing downstream task performance.

Diffusion models have significantly advanced image manipulation techniques, and their ability to generate photorealistic images is beginning to transform retail workflows, particularly in presale visualization. Beyond artistic style transfer, the capability to perform fine-grained visual feature transfer is becoming increasingly important. Embroidery is a textile art form characterized by intricate interplay of diverse stitch patterns and material properties, which poses unique challenges for existing style transfer methods. To explore the customization for such fine-grained features, we propose a novel contrastive learning framework that disentangles fine-grained style and content features with a single reference image, building on the classic concept of image analogy. We first construct an image pair to define the target style, and then adopt a similarity metric based on the decoupled representations of pretrained diffusion models for style-content separation. Subsequently, we propose a two-stage contrastive LoRA modulation technique to capture fine-grained style features. In the first stage, we iteratively update the whole LoRA and the selected style blocks to initially separate style from content. In the second stage, we design a contrastive learning strategy to further decouple style and content through self-knowledge distillation. Finally, we build an inference pipeline to handle image or text inputs with only the style blocks. To evaluate our method on fine-grained style transfer, we build a benchmark for embroidery customization. Our approach surpasses prior methods on this task and further demonstrates strong generalization to three additional domains: artistic style transfer, sketch colorization, and appearance transfer. Our project is available at: https://style3d.github.io/embroidery_customization.

Supersampling has proven highly effective in enhancing visual fidelity by reducing aliasing, increasing resolution, and generating interpolated frames. It has become a standard component of modern real-time rendering pipelines. However, on mobile platforms, deep learning-based supersampling methods remain impractical due to stringent hardware constraints, while non-neural supersampling techniques often fall short in delivering perceptually high-quality results. In particular, producing visually pleasing reconstructions and temporally coherent interpolations is still a significant challenge in mobile settings. In this work, we present a novel, lightweight supersampling framework tailored for mobile devices. Our approach substantially improves both image reconstruction quality and temporal consistency while maintaining real-time performance. For super-resolution, we propose an intra-pixel object coverage estimation method for reconstructing high-quality anti-aliased pixels in edge regions, a gradient-guided strategy for non-edge areas, and a temporal sample accumulation approach to improve overall image quality. For frame interpolation, we develop an efficient motion estimation module coupled with a lightweight fusion scheme that integrates both estimated optical flow and rendered motion vectors, enabling temporally coherent interpolation of object dynamics and lighting variations. Extensive experiments demonstrate that our method consistently outperforms existing baselines in both perceptual image quality and temporal smoothness, while maintaining real-time performance on mobile GPUs. A demo application and supplementary materials are available on the project page.

Current 3D asset creation pipelines typically consist of three stages: creating multi-view concept art, producing 3D meshes based on the artwork, and painting textures for the meshes—an often labor-intensive process. Automated texture generation offers significant acceleration, but prior methods, which fine-tune 2D diffusion models with multi-view input images, often fail to preserve pixel-level details. These methods primarily emphasize semantic and subject consistency, which do not meet the requirements of artwork-guided texture workflows. To address this, we present AlignTex , a novel framework for generating high-quality textures from 3D meshes and multi-view artwork, ensuring both appearance detail and geometric consistency. AlignTex operates in two stages: aligned image generation and texture refinement. The core of our approach, AlignNet , resolves complex misalignments by extracting information from both the artwork and the mesh, generating images compatible with orthographic projection while maintaining geometric and visual fidelity. After projecting aligned images into the texture space, further refinement addresses seams and self-occlusion using an inpainting model and a geometry-aware texture dilation method. Experimental results demonstrate that AlignTex outperforms baseline methods in generation quality and efficiency, offering a practical solution to enhance 3D asset creation in gaming and film production.

As eusocial creatures, bees display unique macro collective behavior and local body dynamics that hold potential applications in various fields, such as computer animation, robotics, and social behavior. Unlike birds and fish, bees fly in a low-aligned zigzag pattern. Additionally, bees rely on visual signals for foraging and predator avoidance, exhibiting distinctive local body oscillations, such as body lifting, thrusting, and swaying. These inherent features pose significant challenges to realistic bee simulations in practical animation applications. In this article, we present a bio-inspired model for bee simulations capable of replicating both macro collective behavior and local body dynamics of bees. Our approach utilizes a visually-driven system to simulate a bee's local body dynamics, incorporating obstacle perception and body rolling control for effective collision avoidance. Moreover, we develop an oscillation rule that captures the dynamics of the bee's local bodies, drawing on insights from biological research. Our model extends beyond simulating individual bees' dynamics; it can also represent bee swarms by integrating a fluid-based field with the bees' innate noise and zigzag motions. To fine-tune our model, we utilize pre-collected honeybee flight data. Through extensive simulations and comparative experiments, we demonstrate that our model can efficiently generate realistic low-aligned and inherently noisy bee swarms.

Neural implicit fields, such as the neural signed distance field (SDF) of a shape, have emerged as a powerful representation for many applications, e.g., encoding a 3D shape and performing collision detection. Typically, implicit fields are encoded by Multi-layer Perceptrons (MLP) with positional encoding (PE) to capture high-frequency geometric details. However, a notable side effect of such PE-equipped MLPs is the noisy artifacts present in the learned implicit fields. While increasing the sampling rate could in general mitigate these artifacts, in this paper we aim to explain this adverse phenomenon through the lens of Fourier analysis. We devise a tool to determine the appropriate sampling rate for learning an accurate neural implicit field without undesirable side effects. Specifically, we propose a simple yet effective method to estimate the intrinsic frequency of a given network with randomized weights based on the Fourier analysis of the network's responses. It is observed that a PE-equipped MLP has an intrinsic frequency much higher than the highest frequency component in the PE layer. Sampling against this intrinsic frequency following the Nyquist-Sannon sampling theorem allows us to determine an appropriate training sampling rate. We empirically show in the setting of SDF fitting that this recommended sampling rate is sufficient to secure accurate fitting results, while further increasing the sampling rate would not further noticeably reduce the fitting error. Training PE-equipped MLPs simply with our sampling strategy leads to performances superior to the existing methods.

Existing facial editing methods have achieved remarkable results, yet they often fall short in supporting multimodal conditional local facial editing. One of the significant evidences is that their output image quality degrades dramatically after several iterations of incremental editing, as they do not support local editing. In this paper, we present a novel multimodal generative and fusion framework for globally-consistent local facial editing (FACEMUG) that can handle a wide range of input modalities and enable fine-grained and semantic manipulation while remaining unedited parts unchanged. Different modalities, including sketches, semantic maps, color maps, exemplar images, text, and attribute labels, are adept at conveying diverse conditioning details, and their combined synergy can provide more explicit guidance for the editing process. We thus integrate all modalities into a unified generative latent space to enable multimodal local facial edits. Specifically, a novel multimodal feature fusion mechanism is proposed by utilizing multimodal aggregation and style fusion blocks to fuse facial priors and multimodalities in both latent and feature spaces. We further introduce a novel self-supervised latent warping algorithm to rectify misaligned facial features, efficiently transferring the pose of the edited image to the given latent codes. We evaluate our FACEMUG through extensive experiments and comparisons to state-of-the-art (SOTA) methods. The results demonstrate the superiority of FACEMUG in terms of editing quality, flexibility, and semantic control, making it a promising solution for a wide range of local facial editing tasks.

Achieving immersive virtual reality (VR) experiences typically requires extensive computational resources to ensure high-definition visuals, high frame rates, and low latency in stereoscopic rendering. This challenge is particularly pronounced for lower-tier and standalone VR devices with limited processing power. To accelerate rendering, existing supersampling and image reprojection techniques have shown significant potential, yet to date, no previous work has explored their combination to minimize stereo rendering overhead. In this paper, we introduce a lightweight supersampling framework that integrates image projection with spatio-temporal supersampling to accelerate stereo rendering. Our approach effectively leverages the temporal and spatial redundancies inherent in stereo videos, enabling rapid image generation for unshaded viewpoints and providing resolution-enhanced and anti-aliased images for binocular viewpoints. We first blend a rendered low-resolution (LR) frame with accumulated temporal samples to construct an high-resolution (HR) frame. This HR frame is then reprojected to the other viewpoint to directly synthesize a new image. To address disocclusions in reprojected images, we utilize accumulated history data and low-pass filtering for filling, ensuring high-quality results with minimal delay. Extensive evaluations on both the PC and the standalone device confirm that our framework requires short runtime to generate high-fidelity images, making it an effective solution for stereo rendering across various VR platforms.

A variety of Neural Radiance Fields (NeRF) methods have recently achieved remarkable success in high render speed. However, current accelerating methods are specialized and incompatible with various implicit methods, preventing real-time composition over various types of NeRF works. Because NeRF relies on sampling along rays, it is possible to provide general guidance for acceleration. To that end, we propose a general implicit pipeline for composing NeRF objects quickly. Our method enables the casting of dynamic shadows within or between objects using analytical light sources while allowing multiple NeRF objects to be seamlessly placed and rendered together with any arbitrary rigid transformations. Mainly, our work introduces a new surface representation known as Neural Depth Fields (NeDF) that quickly determines the spatial relationship between objects by allowing direct intersection computation between rays and implicit surfaces. It leverages an intersection neural network to query NeRF for acceleration instead of depending on an explicit spatial structure.Our proposed method is the first to enable both the progressive and interactive composition of NeRF objects. Additionally, it also serves as a previewing plugin for a range of existing NeRF works.

Pedestrian trajectory prediction is the key technology in many applications for providing insights into human behavior and anticipating human future motions. Most existing empirical models are explicitly formulated by observed human behaviors using explicable mathematical terms with deterministic nature, while recent work has focused on developing hybrid models combined with learning-based techniques for powerful expressiveness while maintaining explainability. However, the deterministic nature of the learned steering behaviors from the empirical models limits the models' practical performance. To address this issue, this work proposes the social conditional variational autoencoder (SocialCVAE) for predicting pedestrian trajectories, which employs a CVAE to explore behavioral uncertainty in human motion decisions. SocialCVAE learns socially reasonable motion randomness by utilizing a socially explainable interaction energy map as the CVAE's condition, which illustrates the future occupancy of each pedestrian's local neighborhood area. The energy map is generated using an energy-based interaction model, which anticipates the energy cost (i.e., repulsion intensity) of pedestrians' interactions with neighbors. Experimental results on two public benchmarks including 25 scenes demonstrate that SocialCVAE significantly improves prediction accuracy compared with the state-of-the-art methods, with up to 16.85% improvement in Average Displacement Error (ADE) and 69.18% improvement in Final Displacement Error (FDE). Code is available at: https://github.com/ViviXiang/SocialCVAE.

Implicit neural representation has paved the way for new approaches to dynamic scene reconstruction. Nonetheless, cutting-edge dynamic neural rendering methods rely heavily on these implicit representations, which frequently struggle to capture the intricate details of objects in the scene. Furthermore, implicit methods have difficulty achieving real-time rendering in general dynamic scenes, limiting their use in a variety of tasks. To address the issues, we propose a deformable 3D Gaussians splatting method that reconstructs scenes using 3D Gaussians and learns them in canonical space with a deformation field to model monocular dynamic scenes. We also introduce an annealing smoothing training mechanism with no extra overhead, which can mitigate the impact of inaccurate poses on the smoothness of time interpolation tasks in real-world scenes. Through a differential Gaussian rasterizer, the deformable 3D Gaussians not only achieve higher rendering quality but also real-time rendering speed. Experiments show that our method outperforms existing methods significantly in terms of both rendering quality and speed, making it well-suited for tasks such as novel-view synthesis, time interpolation, and real-time rendering. Our code is available at https://github.com/ingra14m/Deformable-3D-Gaussians.

Face alignment is a critical and difficult task for many facial analysis applications. Existing VFA methods frequently ignore the consistency of facial geometries and textures across video sequences, limiting their ability to handle accurate and stable face alignment. This paper describes a robust and highly accurate 3D Morphable Model (3DMM)-based VFA approach that employs a novel texture generation method and a self-refined face alignment procedure. Our method iteratively fine-tunes facial geometries, textures, and poses by using a differentiable rendering technique and a self-refined optimization method. Experiment results show that our method outperforms existing state-of-the-art methods in terms of both accuracy and temporal stability. Visual results and source code are available at: https://pawindergit.github.io/SR-VFA/

Dichotomous Image Segmentation (DIS) tasks require highly precise annotations, and traditional dataset creation methods are labor intensive, costly, and require extensive domain expertise. Although using synthetic data for DIS is a promising solution to these challenges, current generative models and techniques struggle with the issues of scene deviations, noise-induced errors, and limited training sample variability. To address these issues, we introduce a novel approach, Mask Factory, which provides a scalable solution for generating diverse and precise datasets, markedly reducing preparation time and costs. We first introduce a general mask editing method that combines rigid and non-rigid editing techniques to generate high-quality synthetic masks. Specially, rigid editing leverages geometric priors from diffusion models to achieve precise viewpoint transformations under zero-shot conditions, while non-rigid editing employs adversarial training and self-attention mechanisms for complex, topologically consistent modifications. Then, we generate pairs of high-resolution image and accurate segmentation mask using a multi-conditional control generation method. Finally, our experiments on the widely-used DIS5K dataset benchmark demonstrate superior performance in quality and efficiency compared to existing methods. The code is available at https://qian-hao-tian.github.io/MaskFactory/.

Implicit representation has opened up new possibilities for inverse rendering. However, existing implicit neural inverse rendering methods struggle to handle strongly illuminated scenes with significant shadows and slight reflections. The existence of shadows and reflections can lead to an inaccurate understanding of the scene, making precise factorization difficult. To this end, we present RobIR, an implicit inverse rendering approach that uses ACES tone mapping and regularized visibility estimation to reconstruct accurate BRDF of the object. By accurately modeling the indirect radiance field, normal, visibility, and direct light simultaneously, we are able to accurately decouple environment lighting and the object's PBR materials without imposing strict constraints on the scene. Even in high-illumination scenes with shadows and specular reflections, our method can recover high-quality albedo and roughness with no shadow interference. RobIR outperforms existing methods in both quantitative and qualitative evaluations.

The recent advancements in 3D Gaussian splatting (3D-GS) have not only facilitated real-time rendering through modern GPU rasterization pipelines but have also attained state-of-the-art rendering quality. Nevertheless, despite its exceptional rendering quality and performance on standard datasets, 3D-GS frequently encounters difficulties in accurately modeling specular and anisotropic components. This issue stems from the limited ability of spherical harmonics (SH) to represent high-frequency information. To overcome this challenge, we introduce Spec-Gaussian, an approach that utilizes an anisotropic spherical Gaussian (ASG) appearance field instead of SH for modeling the view-dependent appearance of each 3D Gaussian. Additionally, we have developed a coarse-to-fine training strategy to improve learning efficiency and eliminate floaters caused by overfitting in real-world scenes. Our experimental results demonstrate that our method surpasses existing approaches in terms of rendering quality. Thanks to ASG, we have significantly improved the ability of 3D-GS to model scenes with specular and anisotropic components without increasing the number of 3D Gaussians. This improvement extends the applicability of 3D GS to handle intricate scenarios with specular and anisotropic surfaces.

Optical motion capture (MoCap) is the "gold standard" for accurately capturing full-body motions. To make use of raw MoCap point data, the system labels the points with corresponding body part locations and solves the full-body motions. However, MoCap data often contains mislabeling, occlusion and positional errors, requiring extensive manual correction. To alleviate this burden, we introduce RoMo, a learning-based framework for robustly labeling and solving raw optical motion capture data. In the labeling stage, RoMo employs a divide-and-conquer strategy to break down the complex full-body labeling challenge into manageable subtasks: alignment, full-body segmentation and part-specific labeling. To utilize the temporal continuity of markers, RoMo generates marker tracklets using a K-partite graph-based clustering algorithm, where markers serve as nodes, and edges are formed based on positional and feature similarities. For motion solving, to prevent error accumulation along the kinematic chain, we introduce a hybrid inverse kinematic solver that utilizes joint positions as intermediate representations and adjusts the template skeleton to match estimated joint positions. We demonstrate that RoMo achieves high labeling and solving accuracy across multiple metrics and various datasets. Extensive comparisons show that our method outperforms state-of-the-art research methods. On a real dataset, RoMo improves the F1 score of hand labeling from 0.94 to 0.98, and reduces joint position error of body motion solving by 25%. Furthermore, RoMo can be applied in scenarios where commercial systems are inadequate. The code and data for RoMo are available at https://github.com/non-void/RoMo.

Motion style transfer changes the style of a motion while retaining its content and is useful in computer animations and games. Contact is an essential component of motion style transfer that should be controlled explicitly in order to express the style vividly while enhancing motion naturalness and quality. However, it is unknown how to decouple and control contact to achieve fine-grained control in motion style transfer. In this paper, we present a novel style transfer method for fine-grained control over contacts while achieving both motion naturalness and spatial-temporal variations of style. Based on our empirical evidence, we propose controlling contact indirectly through the hip velocity, which can be further decomposed into the trajectory and contact timing, respectively. To this end, we propose a new model that explicitly models the correlations between motions and trajectory/contact timing/style, allowing us to decouple and control each separately. Our approach is built around a motion manifold, where hip controls can be easily integrated into a Transformer-based decoder. It is versatile in that it can generate motions directly as well as be used as post-processing for existing methods to improve quality and contact controllability. In addition, we propose a new metric that measures a correlation pattern of motions based on our empirical evidence, aligning well with human perception in terms of motion naturalness. Based on extensive evaluation, our method outperforms existing methods in terms of style expressivity and motion quality.

Abstract We present swRender, a new parallel rendering pipeline based on the new Sunway many-core architecture (SW26010P) for the Monte Carlo path-tracing algorithm. Previous parallel rendering schemes are unsuitable for our task due to issues such as vast differences in hardware architectures and bottlenecks in I/O communication efficiency. To that end, we create a new two-level parallel tile rendering framework to fully utilize the Sunway computing resources, a practical tile-grouping load-balancing method to maintain the framework’s stability, and a novel many-core acceleration optimization to improve the rendering performance at the pixel level. Our method achieves (1) an average speedup of 16x in multiple benchmarks when compared to the baseline path-tracing model on the Sunway architecture, and (2) an average speedup of 2x when compared to state-of-the-art CPU, co-processor, and GPU-based parallel rendering approaches. Moreover, we scale swRender to run on 15 million cores and obtain high scalable parallel efficiency of 92%.

Abstract The rapid development of the online apparel shopping industry demands innovative solutions for high‐quality digital apparel sample displays with virtual avatars. However, developing such displays is prohibitively expensive and prone to the well‐known “uncanny valley” effect, where a nearly human‐looking artifact arouses eeriness and repulsiveness, thus affecting the user experience. To effectively mitigate the “uncanny valley” effect and improve the overall authenticity of digital apparel sample displays, we present a novel photo‐realistic portrait generation framework. Our key idea is to employ transfer learning to learn an identity‐consistent mapping from the latent space of rendered portraits to that of real portraits. During the inference stage, the input portrait of an avatar can be directly transferred to a realistic portrait by changing its appearance style while maintaining the facial identity. To this end, we collect a new dataset, Daz‐Rendered‐Faces‐HQ (DRFHQ), specifically designed for rendering‐style portraits. We leverage this dataset to fine‐tune the StyleGAN2‐FFHQ generator, using our carefully crafted framework, which helps to preserve the geometric and color features relevant to facial identity. We evaluate our framework using portraits with diverse gender, age, and race variations. Qualitative and quantitative evaluations, along with ablation studies, highlight our method's advantages over state‐of‐the‐art approaches.

Abstract Facial action units (AUs) encode the activations of facial muscle groups, playing a crucial role in expression analysis and facial animation. However, current deep learning AU detection methods primarily focus on single‐image analysis, which limits the exploitation of rich temporal context for robust outcomes. Moreover, the scale of available datasets remains limited, leading models trained on these datasets to tend to suffer from overfitting issues. This paper proposes a novel AU detection method integrating spatial and temporal data with inter‐subject feature reassignment for accurate and robust AU predictions. Our method first extracts regional features from facial images. Then, to effectively capture both the temporal context and identity‐independent features, we introduce a temporal feature combination and feature reassignment (TC&FR) module, which transforms single‐image features into a cohesive temporal sequence and fuses features across multiple subjects. This transformation encourages the model to utilize identity‐independent features and temporal context, thus ensuring robust prediction outcomes. Experimental results demonstrate the enhancements brought by the proposed modules and the state‐of‐the‐art (SOTA) results achieved by our method.

Existing neural rendering-based text-to-3D-portrait generation methods typically make use of human geometry prior and diffusion models to obtain guidance. However, relying solely on geometry information introduces issues such as the Janus problem, over-saturation, and over-smoothing. We present Portrait3D , a novel neural rendering-based framework with a novel joint geometry-appearance prior to achieve text-to-3D-portrait generation that overcomes the aforementioned issues. To accomplish this, we train a 3D portrait generator, 3DPortraitGAN, as a robust prior. This generator is capable of producing 360° canonical 3D portraits, serving as a starting point for the subsequent diffusion-based generation process. To mitigate the "grid-like" artifact caused by the high-frequency information in the feature-map-based 3D representation commonly used by most 3D-aware GANs, we integrate a novel pyramid tri-grid 3D representation into 3DPortraitGAN. To generate 3D portraits from text, we first project a randomly generated image aligned with the given prompt into the pre-trained 3DPortraitGAN's latent space. The resulting latent code is then used to synthesize a pyramid tri-grid. Beginning with the obtained pyramid tri-grid , we use score distillation sampling to distill the diffusion model's knowledge into the pyramid tri-grid. Following that, we utilize the diffusion model to refine the rendered images of the 3D portrait and then use these refined images as training data to further optimize the pyramid tri-grid , effectively eliminating issues with unrealistic color and unnatural artifacts. Our experimental results show that Portrait3D can produce realistic, high-quality, and canonical 3D portraits that align with the prompt.

Style-guided texture generation aims to generate a texture that is harmonious with both the style of the reference image and the geometry of the input mesh, given a reference style image and a 3D mesh with its text description. Although diffusion-based 3D texture generation methods, such as distillation sampling, have numerous promising applications in stylized games and films, it requires addressing two challenges: 1) decouple style and content completely from the reference image for 3D models, and 2) align the generated texture with the color tone, style of the reference image, and the given text prompt. To this end, we introduce StyleTex, an innovative diffusion-model-based framework for creating stylized textures for 3D models. Our key insight is to decouple style information from the reference image while disregarding content in diffusion-based distillation sampling. Specifically, given a reference image, we first decompose its style feature from the image CLIP embedding by subtracting the embedding's orthogonal projection in the direction of the content feature, which is represented by a text CLIP embedding. Our novel approach to disentangling the reference image's style and content information allows us to generate distinct style and content features. We then inject the style feature into the cross-attention mechanism to incorporate it into the generation process, while utilizing the content feature as a negative prompt to further dissociate content information. Finally, we incorporate these strategies into StyleTex to obtain stylized textures. We utilize Interval Score Matching to address over-smoothness and over-saturation, in combination with a geometry-aware ControlNet that ensures consistent geometry throughout the generative process. The resulting textures generated by StyleTex retain the style of the reference image, while also aligning with the text prompts and intrinsic details of the given 3D mesh. Quantitative and qualitative experiments show that our method outperforms existing baseline methods by a significant margin.

Recent advancements in 2D diffusion models allow appearance generation on untextured raw meshes. These methods create RGB textures by distilling a 2D diffusion model, which often contains unwanted baked-in shading effects and results in unrealistic rendering effects in the downstream applications. Generating Physically Based Rendering (PBR) materials instead of just RGB textures would be a promising solution. However, directly distilling the PBR material parameters from 2D diffusion models still suffers from incorrect material decomposition, such as baked-in shading effects in albedo. We introduce DreamMat , an innovative approach to resolve the aforementioned problem, to generate high-quality PBR materials from text descriptions. We find out that the main reason for the incorrect material distillation is that large-scale 2D diffusion models are only trained to generate final shading colors, resulting in insufficient constraints on material decomposition during distillation. To tackle this problem, we first finetune a new light-aware 2D diffusion model to condition on a given lighting environment and generate the shading results on this specific lighting condition. Then, by applying the same environment lights in the material distillation, DreamMat can generate high-quality PBR materials that are not only consistent with the given geometry but also free from any baked-in shading effects in albedo. Extensive experiments demonstrate that the materials produced through our methods exhibit greater visual appeal to users and achieve significantly superior rendering quality compared to baseline methods, which are preferable for downstream tasks such as game and film production.

We present MobileSky, the first automatic method for real-time high-quality sky replacement for mobile AR applications. The primary challenge of this task is how to extract sky regions in camera feed both quickly and accurately. While the problem of sky replacement is not new, previous methods mainly concern extraction quality rather than efficiency, limiting their application to our task. We aim to provide higher quality, both spatially and temporally consistent sky mask maps for all camera frames in real time. To this end, we develop a novel framework that combines a new deep semantic network called FSNet with novel post-processing refinement steps. By leveraging IMU data, we also propose new sky-aware constraints such as temporal consistency, position consistency, and color consistency to help refine the weakly classified part of the segmentation output. Experiments show that our method achieves an average of around 30 FPS on off-the-shelf smartphones and outperforms the state-of-the-art sky replacement methods in terms of execution speed and quality. In the meantime, our mask maps appear to be visually more stable across frames. Our fast sky replacement method enables several applications, such as AR advertising, art making, generating fantasy celestial objects, visually learning about weather phenomena, and advanced video-based visual effects. To facilitate future research, we also create a new video dataset containing annotated sky regions with IMU data.

Although neural supersampling has achieved great success in various applications for improving image quality, it is still difficult to apply it to a wide range of real-time rendering applications due to the high computational power demand. Most existing methods are computationally expensive and require high-performance hardware, preventing their use on platforms with limited hardware, such as smartphones. To this end, we propose a new supersampling framework for real-time rendering applications to reconstruct a high-quality image out of a low-resolution one, which is sufficiently lightweight to run on smartphones within a real-time budget. Our model takes as input the renderer-generated low resolution content and produces high resolution and anti-aliased results. To maximize sampling efficiency, we propose using an alternate sub-pixel sample pattern during the rasterization process. This allows us to create a relatively small reconstruction model while maintaining high image quality. By accumulating new samples into a high-resolution history buffer, an efficient history check and re-usage scheme is introduced to improve temporal stability. To our knowledge, this is the first research in pushing real-time neural supersampling on mobile devices. Due to the absence of training data, we present a new dataset containing 57 training and test sequences from three game scenes. Furthermore, based on the rendered motion vectors and a visual perception study, we introduce a new metric called inter-frame structural similarity (IF-SSIM) to quantitatively measure the temporal stability of rendered videos. Extensive evaluations demonstrate that our supersampling model outperforms existing or alternative solutions in both performance and temporal stability.

Existing face generators exhibit exceptional performance on faces in small to medium poses (with respect to frontal faces) but struggle to produce realistic results for large poses. The distorted rendering results on large poses in 3D-aware generators further show that the generated 3D face shapes are far from the distribution of 3D faces in reality. We find that the above issues are caused by the training dataset’s pose imbalance. To this end, we present LPFF, a large-pose Flickr face dataset comprised of 19,590 high-quality real large-pose portrait images. We utilize our dataset to train a 2D face generator that can process large-pose face images, as well as a 3D-aware generator that can generate realistic human face geometry. To better validate our pose-conditional 3D-aware generators, we develop a new FID measure to evaluate the 3D-level performance. Through this novel FID measure and other experiments, we show that LPFF can help 2D face generators extend their latent space and better manipulate the large-pose data, and help 3D-aware face generators achieve better view consistency and more realistic 3D reconstruction results.

Optimizing the memory footprint of 3D models can have a major impact on the user experiences during real-time rendering and streaming visualization, where the major memory overhead lies in the high-resolution texture data. In this work, we propose a robust and automatic pipeline to content-aware, lossy compression for texture atlas. The design of our solution lies in two observations: 1) mapping multiple surface patches to the same texture region is seamlessly compatible with the standard rendering pipeline, requiring no decompression before any usage; 2) a texture image has background regions and salient structural features, which can be handled separately to achieve a high compression rate. Accordingly, our method contains joint operations of image segmentation, re-meshing, UV unwrapping, and texture baking. To evaluate the efficacy of our approach, we batch-processed a dataset containing 100 models collected online. On average, our method achieves a texture atlas compression ratio of 81.41% with an averaged PSNR and MS-SSIM scores of 40.90 and 0.98, a marginal error in visual appearance.

We present a novel locality-based learning method for cleaning and solving optical motion capture data. Given noisy marker data, we propose a new heterogeneous graph neural network which treats markers and joints as different types of nodes, and uses graph convolution operations to extract the local features of markers and joints and transform them to clean motions. To deal with anomaly markers (e.g. occluded or with big tracking errors), the key insight is that a marker’s motion shows strong correlations with the motions of its immediate neighboring markers but less so with other markers, a.k.a. locality, which enables us to efficiently fill missing markers (e.g. due to occlusion). Additionally, we also identify marker outliers due to tracking errors by investigating their acceleration profiles. Finally, we propose a training regime based on representation learning and data augmentation, by training the model on data with masking. The masking schemes aim to mimic the occluded and noisy markers often observed in the real data. Finally, we show that our method achieves high accuracy on multiple metrics across various datasets. Extensive comparison shows our method outperforms state-of-the-art methods in terms of prediction accuracy of occluded marker position error by approximately 20%, which leads to a further error reduction on the reconstructed joint rotations and positions by 30%. The code and data for this paper are available at https://github.com/non-void/LocalMoCap.

Abstract Realistic crowd simulation has been pursued for decades, but it still necessitates tedious human labour and a lot of trial and error. The majority of currently used crowd modelling is either empirical (model‐based) or data‐driven (model‐free). Model‐based methods cannot fit observed data precisely, whereas model‐free methods are limited by the availability/quality of data and are uninterpretable. In this paper, we aim at taking advantage of both model‐based and data‐driven approaches. In order to accomplish this, we propose a new simulation framework built on a physics‐based model that is designed to be data‐friendly. Both the general prior knowledge about crowds encoded by the physics‐based model and the specific real‐world crowd data at hand jointly influence the system dynamics. With a multi‐granularity physics‐based model, the framework combines microscopic and macroscopic motion control. Each simulation step is formulated as an energy optimization problem, where the minimizer is the desired crowd behaviour. In contrast to traditional optimization‐based methods which seek the theoretical minimizer, we designed an acceleration‐aware data‐driven scheme to compute the minimizer from real‐world data in order to achieve higher realism by parameterizing both velocity and acceleration. Experiments demonstrate that our method can produce crowd animations that are more realistically behaved in a variety of scales and scenarios when compared to the earlier methods.

Abstract Character hit reaction is an inherent component in game development. Natural hit reactions in games are typically achieved through the use of artist‐created hit animations and motion capture. To improve the realism of impact reactions, game developers combine physics simulation with distinct hit animations based on character statuses. However, there is currently no method that can automatically produce hit reactions based on hit information in game development. To this end, we propose a physics‐driven inverse kinematic method for generating character reaction animations. We postulate that a character's hit reactions are the result of an assault impulse spreading throughout the body and forcing the body to move. Five IK (inverse kinematics) solvers are used to control character poses. Each IK solver is used to control the movement of a different part of the body. The IK solvers, which are used to determine the positions of various bodily parts, are driven by unconstrained physics simulation. Furthermore, physics simulation with constraints is used to fine‐tune the character's movements. Experiment results show that our method outperforms Unreal Engine‐based hit animation and physics simulation.

Abstract Due to the complex and subtle behaviors of humans, realistic crowd simulation is difficult. To that end, we propose a novel crowd simulation method that can generate realistic crowd animations with behaviors similar to real crowds and model complex pedestrian behaviors at multiple levels using social long short‐term memory (LSTM) neural networks. At the high level, our multi‐level simulation model provides global group navigation while at the low level, it can simulate local individual interactions with collision avoidance. We introduce a data‐driven method using an improved social LSTM for learning local motion decisions from real pedestrian trajectories in order to capture the subtle movements of the crowd. To achieve scalability, we formulate the low‐level and high‐level motion control in a force‐based scheme. Extensive simulation results demonstrate that our method can produce realistic crowd animations in a variety of scenarios. Evaluations in various metrics show that our method produces better crowd behaviors than previous methods.

Current video semantic segmentation tasks involve two main challenges: how to take full advantage of multi-frame context information, and how to improve computational efficiency. To tackle the two challenges simultaneously, we present a novel Multi-Granularity Context Network (MGCNet) by aggregating context information at multiple granularities in a more effective and efficient way. Our method first converts image features into semantic prototypes, and then conducts a non-local operation to aggregate the per-frame and short-term contexts jointly. An additional long-term context module is introduced to capture the video-level semantic information during training. By aggregating both local and global semantic information, a strong feature representation is obtained. The proposed pixel-to-prototype non-local operation requires less computational cost than traditional non-local ones, and is video-friendly since it reuses the semantic prototypes of previous frames. Moreover, we propose an uncertainty-aware and structural knowledge distillation strategy to boost the performance of our method. Experiments on Cityscapes and CamVid datasets with multiple backbones demonstrate that the proposed MGCNet outperforms other state-of-the-art methods with high speed and low latency.

Virtual traffic benefits a variety of applications, including video games, traffic engineering, autonomous driving, and virtual reality. To date, traffic visualization via different simulation models can reconstruct detailed traffic flows. However, each specific behavior of vehicles is always described by establishing an independent control model. Moreover, mutual interactions between vehicles and other road users are rarely modeled in existing simulators. An all-in-one simulator that considers the complex behaviors of all potential road users in a realistic urban environment is urgently needed. In this work, we propose a novel, extensible, and microscopic method to build heterogeneous traffic simulation using the force-based concept. This force-based approach can accurately replicate the sophisticated behaviors of various road users and their interactions in a simple and unified manner. We calibrate the model parameters using real-world traffic trajectory data. The effectiveness of this approach is demonstrated through many simulation experiments, as well as comparisons to real-world traffic data and popular microscopic simulators for traffic animation.

Removing hair from portrait images is challenging due to the complex occlusions between hair and face, as well as the lack of paired portrait data with/without hair. To this end, we present a dataset and a baseline method for removing hair from portrait images using generative adversarial networks (GANs). Our core idea is to train a fully connected network HairMapper to find the direction of hair removal in the latent space of StyleGAN for the training stage. We develop a new separation boundary and diffuse method to generate paired training data for males, and a novel “female-male-bald” pipeline for paired data of females. Experiments show that our method can naturally deal with portrait images with variations on gender, age, etc. We validate the superior performance of our method by comparing it to state-of-the-art methods through extensive experiments and user studies. We also demonstrate its applications in hair design and 3D face reconstruction.

Abstract We present a novel traffic trajectory editing method which uses spatio‐temporal keyframes to control vehicles during the simulation to generate desired traffic trajectories. By taking self‐motivation, path following and collision avoidance into account, the proposed force‐based traffic simulation framework updates vehicle's motions in both the Frenet coordinates and the Cartesian coordinates. With the way‐points from users, lane‐level navigation can be generated by reference path planning. With a given keyframe, the coarse‐to‐fine optimization is proposed to efficiently generate the plausible trajectory which can satisfy the spatio‐temporal constraints. At first, a directed state‐time graph constructed along the reference path is used to search for a coarse‐grained trajectory by mapping the keyframe as the goal. Then, using the information extracted from the coarse trajectory as initialization, adjoint‐based optimization is applied to generate a finer trajectory with smooth motions based on our force‐based simulation. We validate our method with extensive experiments.

Abstract We present the first synthetic eyebrow matting datasets and a domain adaptation eyebrow matting network for learning domain‐robust feature representation using synthetic eyebrow matting data and unlabeled in‐the‐wild images with adversarial learning. Different from existing matting methods that may suffer from the lack of ground‐truth matting datasets, which are typically labor‐intensive to annotate or even worse, unable to obtain, we train the matting network in a semi‐supervised manner using synthetic matting datasets instead of ground‐truth matting data while achieving high‐quality results. Specifically, we first generate a large‐scale synthetic eyebrow matting dataset by rendering avatars and collect a real‐world eyebrow image dataset while maximizing the data diversity as much as possible. Then, we use the synthetic eyebrow dataset to train a multi‐task network, which consists of a regression task to estimate the eyebrow alpha mattes and an adversarial task to adapt the learned features from synthetic data to real data. As a result, our method can successfully train an eyebrow matting network using synthetic data without the need to label any real data. Our method can accurately extract eyebrow alpha mattes from in‐the‐wild images without any additional prior and achieves state‐of‐the‐art eyebrow matting performance. Extensive experiments demonstrate the superior performance of our method with both qualitative and quantitative results.

Abstract Multi‐agent simulation has attracted much attention in the field of computer animation in the last decades for its ability to model interaction between autonomous micro level entities. It widely uses deep reinforcement learning (DRL), which allows us to model environments and its agents approaching real‐world and human‐level complexity, with applications in robotics and computer animation, among others. However, DRL multi‐agent simulation faces additional challenges: they have to be able to generalize high‐dimensional observations and relate them with a high‐dimensional action space, maximizing long‐term cumulated reward. Due to this, DRL systems with numerous interacting agents seldom consider skeleton‐level action spaces. To this end, we present skeleton‐level control for multi‐agent simulation with DRL. With our method, we are able to procedurally generate real‐time collision‐free simulations directly on individual agents with a high‐dimensional skeleton‐level action space. The state in our DRL system includes the velocity of the agent, its destination, and the status of its joints, as well as visual‐based information about the environment and other agents. Our reward function encourages motion into the target destinations, and enalizes collision. We provide extensive experimentation to show the ability of agents to reach their goal through its skeleton motion while successfully avoiding inter‐collisions.

Separating the dominant person from the complex background is significant to the human-related research and photo-editing based applications. Existing segmentation algorithms are either too general to separate the person region accurately, or not capable of achieving real-time speed. In this paper, we introduce the multi-domain learning framework into a novel baseline model to construct the Multi-domain TriSeNet Networks for the real-time single person image segmentation. We first divide training data into different subdomains based on the characteristics of single person images, then apply a multi-branch Feature Fusion Module (FFM) to decouple the networks into the domain-independent and the domain-specific layers. To further enhance the accuracy, a self-supervised learning strategy is proposed to dig out domain relations during training. It helps transfer domain-specific knowledge by improving predictive consistency among different FFM branches. Moreover, we create a large-scale single person image segmentation dataset named MSSP20k, which consists of 22,100 pixel-level annotated images in the real world. The MSSP20k dataset is more complex and challenging than existing public ones in terms of scalability and variety. Experiments show that our Multi-domain TriSeNet outperforms state-of-the-art approaches on both public and the newly built datasets with real-time speed.

Detecting points of interest on 3D shapes is a fundamental research problem in geometry processing. Due to the complicated relationship between points of interest and their geometric features, detecting points of interest on any given 3D shape remains challenging. Due to the lack of training data, previous data-driven methods for detecting 3D points of interest mainly focus on utilizing hand-crafted geometric features to predict the probabilities of each point being a POI, which greatly limits detection performance. In this paper, we propose a novel algorithm for detecting 3D points of interest by using projective neural networks. Our method first projects the labeled training 3D shapes into multiple 2D views and then learns the required features from the 2D views in an end-to-end fashion. The points of interest on test 3D shapes are then automatically detected by applying the learned neural network and our improved density peak clustering. Our method relies neither on hand-crafted feature descriptors nor a large quantity of expensive 3D training data to obtain satisfactory results. Experimental results show significantly superior detection performance of our method over the state-of-the-art methods.

Butterflies are not only ubiquitous around the world but are also widely known for inspiring thrill resonance, with their elegant and peculiar flights. However, realistically modeling and simulating butterfly flights—in particular, for real-time graphics and animation applications—remains an under-explored problem. In this article, we propose an efficient and practical model to simulate butterfly flights. We first model a butterfly with parametric maneuvering functions, including wing-abdomen interaction. Then, we simulate dynamic maneuvering control of the butterfly through our force-based model, which includes both the aerodynamics force and the vortex force. Through many simulation experiments and comparisons, we demonstrate that our method can efficiently simulate realistic butterfly flight motions in various real-world settings.

Real-time in-between motion generation is universally required in games and highly desirable in existing animation pipelines. Its core challenge lies in the need to satisfy three critical conditions simultaneously: quality, controllability and speed , which renders any methods that need offline computation (or post-processing) or cannot incorporate (often unpredictable) user control undesirable. To this end, we propose a new real-time transition method to address the aforementioned challenges. Our approach consists of two key components: motion manifold and conditional transitioning. The former learns the important low-level motion features and their dynamics; while the latter synthesizes transitions conditioned on a target frame and the desired transition duration. We first learn a motion manifold that explicitly models the intrinsic transition stochasticity in human motions via a multi-modal mapping mechanism. Then, during generation, we design a transition model which is essentially a sampling strategy to sample from the learned manifold, based on the target frame and the aimed transition duration. We validate our method on different datasets in tasks where no post-processing or offline computation is allowed. Through exhaustive evaluation and comparison, we show that our method is able to generate high-quality motions measured under multiple metrics. Our method is also robust under various target frames (with extreme cases).

Virtual marine scene authoring plays an important role in generating large-scale 3D scenes and it has a wide range of applications in computer animation and simulation. Existing marine scene authoring methods either produce periodic patterns or generate unnatural group distributions when tiling marine entities such as schools of fish and groups of reefs. To this end, we propose a new large-scale marine scene authoring method based on real examples in order to create more natural and realistic results. Our method first extracts the distribution of multiple marine entities from real images to create Octahedral Blocks, and then we use a modified Wang Cubes algorithm to quickly tile the 3D marine scene. As a result, our method is able to generate aperiodic tiling results with diverse distributions of density and orientation of entities. We validate the effectiveness of our method through intensive comparative experiments. User study results show that our method can generate satisfactory results which are in accord with human preferences.

Sharing short personalized videos to various social media networks has become quite popular in recent years. This raises the need for digital retouching of portraits in videos. However, applying portrait image editing directly on portrait video frames cannot generate smooth and stable video sequences. To this end, we present a robust and easy-to-use parametric method to reshape the portrait in a video to produce smooth retouched results. Given an input portrait video, our method consists of two main stages: stabilized face reconstruction, and continuous video reshaping. In the first stage, we start by estimating face rigid pose transformations across video frames. Then we jointly optimize multiple frames to reconstruct an accurate face identity, followed by recovering face expressions over the entire video. In the second stage, we first reshape the reconstructed 3D face using a parametric reshaping model reflecting the weight change of the face, and then utilize the reshaped 3D face to guide the warping of video frames. We develop a novel signed distance function based dense mapping method for the warping between face contours before and after reshaping, resulting in stable warped video frames with minimum distortions. In addition, we use the 3D structure of the face to correct the dense mapping to achieve temporal consistency. We generate the final result by minimizing the background distortion through optimizing a content-aware warping mesh. Extensive experiments show that our method is able to create visually pleasing results by adjusting a simple reshaping parameter, which facilitates portrait video editing for social media and visual effects.

Abstract Fluid‐like motion and linear wave propagation behavior will emerge when we impose boundary constraints and polarized conditions on crowds. To this end, we present a Lagrangian hydrodynamics method to simulate the fluid‐like motion of crowd and a triggering approach to generate the linear stop‐and‐go wave behavior. Specifically, we impose a self‐propulsion force on the leading agents of the crowd to push the crowd to move forward and introduce a Smoothed Particle Hydrodynamics‐based model to simulate the dynamics of dense crowds. Besides, we present a motion signal propagation approach to trigger the rest of the crowd so that they respond to the immediate leaders linearly, which can lead to the linear stop‐and‐go wave effect of the fluid‐like motion for the crowd. Our experiments demonstrate that our model can simulate large‐scale dense crowds with linear wave propagation.

Abstract We present a simplified force‐based heterogeneous traffic simulation model to facilitate consistent adjustment of the parameters involved. Different from previous work which requires the adjustment of multiple ad hoc parameters to produce satisfactory results, our approach can achieve similar results by using clear and meaningful parameters to simulate interactions between various kinds of road users. To simulate diverse and realistic motions of road users, we parameterize the coefficients of the force model for better detailed motion control. Our approach is also scalable to new types of road users, and facilitates an object‐oriented implementation with high performance. We validate our framework with extensive experiments.

Facial structure editing of portrait images is challenging given the facial variety, the lack of ground-truth, the necessity of jointly adjusting color and shape, and the requirement of no visual artifacts. In this paper, we investigate how to perform chin editing as a case study of editing facial structures. We present a novel method that can automatically remove the double chin effect in portrait images. Our core idea is to train a fine classification boundary in the latent space of the portrait images. This can be used to edit the chin appearance by manipulating the latent code of the input portrait image while preserving the original portrait features. To achieve such a fine separation boundary, we employ a carefully designed training stage based on latent codes of paired synthetic images with and without a double chin. In the testing stage, our method can automatically handle portrait images with only a refinement to subtle misalignment before and after double chin editing. Our model enables alteration to the neck region of the input portrait image while keeping other regions unchanged, and guarantees the rationality of neck structure and the consistency of facial characteristics. To the best of our knowledge, this presents the first effort towards an effective application for editing double chins. We validate the efficacy and efficiency of our approach through extensive experiments and user studies.

Eyelashes play a crucial part in the human facial structure and largely affect the facial attractiveness in modern cosmetic design. However, the appearance and structure of eyelashes can easily induce severe artifacts in high-fidelity multi-view 3D face reconstruction. Unfortunately it is highly challenging to remove eyelashes from portrait images using both traditional and learning-based matting methods due to the delicate nature of eyelashes and the lack of eyelash matting dataset. To this end, we present EyelashNet, the first eyelash matting dataset which contains 5,400 high-quality eyelash matting data captured from real world and 5,272 virtual eyelash matting data created by rendering avatars. Our work consists of a capture stage and an inference stage to automatically capture and annotate eyelashes instead of tedious manual efforts. The capture is based on a specifically-designed fluorescent labeling system. By coloring the eyelashes with a safe and invisible fluorescent substance, our system takes paired photos with colored and normal eyelashes by turning the equipped ultraviolet (UVA) flash on and off. We further correct the alignment between each pair of photos and use a novel alpha matte inference network to extract the eyelash alpha matte. As there is no prior eyelash dataset, we propose a progressive training strategy that progressively fuses captured eyelash data with virtual eyelash data to learn the latent semantics of real eyelashes. As a result, our method can accurately extract eyelash alpha mattes from fuzzy and self-shadow regions such as pupils, which is almost impossible by manual annotations. To validate the advantage of EyelashNet, we present a baseline method based on deep learning that achieves state-of-the-art eyelash matting performance with RGB portrait images as input. We also demonstrate that our work can largely benefit important real applications including high-fidelity personalized avatar and cosmetic design.

Face-top coordination, which exists in most clothes-fitting scenarios, is challenging due to varieties of attributes, implicit correlations, and tradeoffs between general preferences and individual preferences. We present a Deep-Based Self-Refined (DBSR) system to simulate face-top coordination based on intuition evaluation. To this end, we first establish a well-coordinated face-top (WCFT) dataset from fashion databases and communities. Then, we use a jointly trained CNN Deep Canonical Correlation Analysis (DCCA) method to bridge the semantic face-top gap based on the WCFT dataset to deal with general preferences. Subsequently, an irrelevance-based Optimum-path Forest (OPF) method is developed to adapt the results to individual preferences iteratively. Experimental results and user study demonstrate the effectiveness of our method.

Interactive multi-agent simulation algorithms are used to compute the trajectories and behaviors of different entities in virtual reality scenarios. However, current methods involve considerable parameter tweaking to generate plausible behaviors. We introduce a novel approach (Heter-Sim) that combines physics-based simulation methods with data-driven techniques using an optimization-based formulation. Our approach is general and can simulate heterogeneous agents corresponding to human crowds, traffic, vehicles, or combinations of different agents with varying dynamics. We estimate motion states from real-world datasets that include information about position, velocity, and control direction. Our optimization algorithm considers several constraints, including velocity continuity, collision avoidance, attraction, direction control. Other constraints are implemented by introducing a novel energy function to control the motions of heterogeneous agents. To accelerate the computations, we reduce the search space for both collision avoidance and optimal solution computation. Heter-Sim can simulate tens or hundreds of agents at interactive rates and we compare its accuracy with real-world datasets and prior algorithms. We also perform user studies that evaluate the plausible behaviors generated by our algorithm and a user study that evaluates the plausibility of our algorithm via VR.

We present a CPU-based real-time cloth animation method for dressing virtual humans of various shapes and poses. Our approach formulates the clothing deformation as a high-dimensional function of body shape parameters and pose parameters. In order to accelerate the computation, our formulation factorizes the clothing deformation into two independent components: the deformation introduced by body pose variation (Clothing Pose Model) and the deformation from body shape variation (Clothing Shape Model). Furthermore, we sample and cluster the poses spanning the entire pose space and use those clusters to efficiently calculate the anchoring points. We also introduce a sensitivity-based distance measurement to both find nearby anchoring points and evaluate their contributions to the final animation. Given a query shape and pose of the virtual agent, we synthesize the resulting clothing deformation by blending the Taylor expansion results of nearby anchoring points. Compared to previous methods, our approach is general and able to add the shape dimension to any clothing pose model. Furthermore, we can animate clothing represented with tens of thousands of vertices at 50+ FPS on a CPU. We also conduct a user evaluation and show that our method can improve a user's perception of dressed virtual agents in an immersive virtual environment (IVE) compared to a realtime linear blend skinning method.

Feature matters for salient object detection. Existing methods mainly focus on designing a sophisticated structure to incorporate multi-level features and filter out cluttered features. We present Progressive Feature Polishing Network (PFPN), a simple yet effective framework to progressively polish the multi-level features to be more accurate and representative. By employing multiple Feature Polishing Modules (FPMs) in a recurrent manner, our approach is able to detect salient objects with fine details without any post-processing. A FPM parallelly updates the features of each level by directly incorporating all higher level context information. Moreover, it can keep the dimensions and hierarchical structures of the feature maps, which makes it flexible to be integrated with any CNN-based models. Empirical experiments show that our results are monotonically getting better with increasing number of FPMs. Without bells and whistles, PFPN outperforms the state-of-the-art methods significantly on five benchmark datasets under various evaluation metrics. Our code is available at: https://github.com/chenquan-cq/PFPN.

We present deep shapely portraits, a novel method based on deep learning, to automatically reshape an input portrait to be better proportioned and more shapely while keeping personal facial characteristics. Different from existing methods that may suffer from irrational face artifacts when dealing with portraits with large pose variations or reshaping adjustments, we utilize dense 3D face information and constraints instead of sparse facial landmarks based on 3D morphable models, resulting in better reshaped faces lying in rational face space. To this end, we first estimate the best shapely degree for the input portrait using a convolutional neural network (CNN) trained on our newly developed ShapeFaceNet dataset. Then the best shapely degree is used as the control parameter to reshape the 3D face reconstructed from the input portrait image. After that, we render the reshaped 3D face back to 2D and generate a seamless portrait image using a fast image warping optimization. Our work can deal with pose and expression free (PE-Free) portrait images and generate plausible shapely faces without noticeable artifacts, which cannot be achieved by prior work. We validate the effectiveness, efficiency, and robustness of the proposed method by extensive experiments and user studies.

Abstract Curve skeleton is an important shape descriptor with many potential applications in computer graphics, visualization and machine intelligence. We present a curve skeleton expression based on the set of the cross‐section centroids from a point cloud model and propose a corresponding extraction approach. We first provide the substitution of a distance field for a 3D point cloud model, and then combine it with curvatures to capture hybrid feature points. By introducing relevant facets and points, we shift these hybrid feature points along the skeleton‐guided normal directions to approach local centroids, simplify them through a tensor‐based spectral clustering and finally connect them to form a primary connected curve skeleton. Furthermore, we refine the primary skeleton through pruning, trimming and smoothing. We compared our results with several state‐of‐the‐art algorithms including the rotational symmetry axis (ROSA) and L1‐medial methods for incomplete point cloud data to evaluate the effectiveness and accuracy of our method.

The cover image is based on the Original Article Sketch-based Shape-constrained Fireworks Simulation in Head Mounted Virtual Reality by Xiaogang Jin et al., https://doi.org/10.1002/cav.1920.

Abstract In this paper we present a novel shape‐constrained fireworks simulation method with rich textures in an HMD (Helmet Mounted Display) virtual environment using sketched feature lines as input. Our approach first retrieves an object from a three‐dimensional (3D) model database using a sketch‐based 3D shape retrieval algorithm. Then, in order to approximate models with complex structures, we introduce a novel point sampling algorithm based on Gaussian curvatures, which stores not only the positions of the selected vertices but also the texture (UV) coordinates information for texture display. In addition, we introduce a multilevel explosion process so that the fireworks can dynamically form specific, visually pleasing shapes. Through our experiments, we demonstrate that our approach can produce better results than state‐of‐the‐art approaches.

The cover image is based on the Original Article FASTSWARM: A Data-driven FrAmework for Real-time Flying InSecT SWARM Simulation by Wei Xiang et al., https://doi.org/10.1002/cav.1957.

Abstract Insect swarms are common phenomena in nature and therefore have been actively pursued in computer animation. Realistic insect swarm simulation is difficult due to two challenges: high‐fidelity behaviors and large scales, which make the simulation practice subject to laborious manual work and excessive trial‐and‐error processes. To address both challenges, we present a novel data‐driven framework, FASTSWARM, to model complex behaviors of flying insects based on real‐world data and simulate plausible animations of flying insect swarms. FASTSWARM has a linear time complexity and achieves real‐time performance for large swarms. The high‐fidelity behavior model of FASTSWARM explicitly takes into consideration the most common behaviors of flying insects, including the interactions among insects such as repulsion and attraction, self‐propelled behaviors such as target following and obstacle avoidance, and other characteristics such as random movements. To achieve scalability, an energy minimization problem is formed with different behaviors modeled as energy terms, where the minimizer is the desired behavior. The minimizer is computed from the real‐world data, which ensures the plausibility of the simulation results. Extensive simulation results and evaluations show that FASTSWARM is versatile in simulating various swarm behaviors, high fidelity measured by various metrics, easily controllable in inducing user controls and highly scalable.

With the recent advances in data acquisition techniques, the compression of various dynamic mesh sequence data has become an important topic in the computer graphics community. In this article, we present a new spatio-temporal segmentation-based approach for the adaptive compression of the dynamic mesh sequences. Given an input dynamic mesh sequence, we first compute an initial temporal cut to obtain a small subsequence by detecting the temporal boundary of dynamic behavior. Then, we apply a two-stage vertex clustering on the resulting subsequence to classify the vertices into groups with optimal intra-affinities. After that, we design a temporal segmentation step based on the variations of the principal components within each vertex group prior to performing a PCA-based compression. Furthermore, we apply an extra step on the lossless compression of the PCA bases and coefficients to gain more storage saving. Our approach can adaptively determine the temporal and spatial segmentation boundaries to exploit both temporal and spatial redundancies. We have conducted extensive experiments on different types of 3D mesh animations with various segmentation configurations. Our comparative studies show the advantages of our approach for the compression of 3D mesh animations.

Aiming at objectively measuring the realism of virtual traffic flows and evaluating the effectiveness of different traffic simulation techniques, this paper introduces a general, dictionary-based learning method to evaluate the fidelity of any traffic trajectory data. First, a traffic pattern dictionary that characterizes common patterns of real-world traffic behavior is built offline from pre-collected ground truth traffic data. The corresponding learning error is set as the benchmark of the dictionary-based traffic representation. With the aid of the constructed dictionary, the realism of input simulated traffic flow data can be evaluated by comparing its dictionary-based reconstruction error with the dictionary error benchmark. This evaluation metric can be robustly applied to any simulated traffic flow data; in other words, it is independent of how the traffic data are generated. We demonstrated the effectiveness and robustness of this metric through many experiments on real-world traffic data and various simulated traffic data, comparisons with the state-of-the-art entropy-based similarity metric for aggregate crowd motions, and perceptual evaluation studies.

Recent failures in real-world self-driving tests have suggested a paradigm shift from directly learning in real-world roads to building a high-fidelity driving simulator as an alternative, effective, and safe tool to handle intricate traffic environments in urban areas. To date, traffic simulation can construct virtual urban environments with various weather conditions, day and night, and traffic control for autonomous vehicle testing. However, mutual interactions between autonomous vehicles and pedestrians are rarely modeled in existing simulators. Besides vehicles and pedestrians, the usage of personal mobility devices is increasing in congested cities as an alternative to the traditional transport system. A simulator that considers all potential road-users in a realistic urban environment is urgently desired. In this work, we propose a novel, extensible, and microscopic method to build heterogenous traffic simulation using the force-based concept. This force-based approach can accurately replicate the sophisticated behaviors of various road users and their interactions through a simple and unified way. Furthermore, we validate our approach through simulation experiments and comparisons to the popular simulators currently used for research and development of autonomous vehicles.

With the recent advances of data acquisition techniques, the compression of various 3D mesh animation data has become an important topic in computer graphics community. In this paper, we present a new spatio-temporal segmentation-based approach for the compression of 3D mesh animations. Given an input mesh sequence, we first compute an initial temporal cut to obtain a small subsequence by detecting the temporal boundary of dynamic behavior. Then, we apply a two-stage vertex clustering on the resulting subsequence to classify the vertices into groups with optimal intra-affinities. After that, we design a temporal segmentation step based on the variations of the principle components within each vertex group prior to performing a PCA-based compression. Our approach can adaptively determine the temporal and spatial segmentation boundaries in order to exploit both temporal and spatial redundancies. We have conducted many experiments on different types of 3D mesh animations with various segmentation configurations. Our comparative studies show the competitive performance of our approach for the compression of 3D mesh animations.

Abstract During the past decades, high‐fidelity realistic simulations of various flying insects exhibiting collective behavior have been broadly used in entertainment industries and virtual reality applications. However, due to the intrinsic complexity and high computational cost, shape constrained simulation of collective behaviors remains a challenging topic. In this paper, we present a robust multi‐agent model for large‐scale controllable shape constrained simulation of flying insects. Specifically, we design an internal force model to biologically mimic an individual insect. We also propose an external force model based on a trade‐off mechanic to guide the insects smoothly deforming into a target shape. Our experimental results and comparative studies show our method is able to simulate realistic and dynamic flying insects with various user‐specified shape constraints.

Abstract We present a unified biologically inspired approach to simulate ant colonies inspired by the key observation of collective behaviors of ants in nature. To generate the trajectories of virtual ants, we construct a motion controller to determine the motion states and the paths of virtual ants, considering dynamic internal and external interactions. The motion controller computes a target position for each ant at every time step according to its motion states. The motion states include four states: basic movement, the stop state, and two dynamic interactions (i.e., internal and external , respectively referring to interaction with neighbors for necessary information transfer about the destination, and interaction with surroundings such as food sources, nests, and obstacles) to represent basic exploration, casual or intentional stop, and purposeful movement, respectively. Based on the motion states, the motion controller plans an optimal path for each virtual ant. Through many simulation experiments, we demonstrate that our method is controllable, scalable, and flexible to simulate hybrid colonies with a large number of ants.

Abstract Although different approaches of segmenting and assembling geometric models for 3D printing have been proposed, it is difficult to find any research studies, which investigate model segmentation and assembly in head‐mounted display (HMD) virtual reality (VR) environments for 3D printing. In this work, we propose a novel and interactive segmentation method for screwing assembly in the environments to tackle this problem. Our approach divides a large model into semantic parts with a screwing interface for repeated tight assembly. Specifically, after a user places the cutting interface, our algorithm computes the bounding box of the current part automatically for subsequent multicomponent semantic Boolean segmentations. Afterwards, the bolt is positioned with an improved K3M image thinning algorithm and is used for merging paired components with union and subtraction Boolean operations respectively. Moreover, we introduce a swept Boolean‐based rotation collision detection and location method to guarantee a collision‐free screwing assembly. Experiments show that our approach provides a new interactive multicomponent semantic segmentation tool that supports not only repeated installation and disassembly but also tight and aligned assembly.

In the same vein of discriminative one-shot learning, Siamese networks allow recognizing an object from a single exemplar with the same class label. However, they do not take advantage of the underlying structure of the data and the relationship among the multitude of samples as they only rely on the pairs of instances for training. In this paper, we propose a new quadruplet deep network to examine the potential connections among the training instances, aiming to achieve a more powerful representation. We design a shared network with four branches that receive a multi-tuple of instances as inputs and are connected by a novel loss function consisting of pair loss and triplet loss. According to the similarity metric, we select the most similar and the most dissimilar instances as the positive and negative inputs of triplet loss from each multi-tuple. We show that this scheme improves the training performance. Furthermore, we introduce a new weight layer to automatically select suitable combination weights, which will avoid the conflict between triplet and pair loss leading to worse performance. We evaluate our quadruplet framework by model-free tracking-by-detection of objects from a single initial exemplar in several visual object tracking benchmarks. Our extensive experimental analysis demonstrates that our tracker achieves superior performance with a real-time processing speed of 78 frames/s. Our source code is available.

In this paper, we propose a framework of maximizing quadratic submodular energy with a knapsack constraint approximately, to solve certain computer vision problems. The proposed submodular maximization problem can be viewed as a generalization of the classic 0/1 knapsack problem. Importantly, maximization of our knapsack constrained submodular energy function can be solved via dynamic programing. We further introduce a range-reduction step prior to dynamic programing as a two-stage procedure for more efficient maximization. In order to demonstrate the effectiveness of the proposed energy function and its maximization algorithm, we apply it to two representative computer vision tasks: image segmentation and motion trajectory clustering. Experimental results of image segmentation demonstrate that our method outperforms the classic segmentation algorithms of graph cuts and random walks. Moreover, our framework achieves better performance than state-of-the-art methods on the motion trajectory clustering task.

Abstract We present a novel and intuitive sketch‐based tree animation technique, targeting on generating a new type of special effect of smoothly transforming leafy trees into morphologically different new shapes. Both topological consistencies of branches and meaningful in‐between crown shapes are preserved during the transformation. Specifically, it takes a leafy tree and a user's sketch describing the silhouette of the desired crown shape under a certain viewpoint as the input. Based on a self‐adaptive multiscale cage tree representation, branches are locally transformed through a series of topology‐aware deformations, and the resulting tree conforms to the user‐designed shape, demonstrating better aesthetics compared to global single‐cage‐based methods. By interpolating the transformations, we are able to create visually pleasing shape‐preserving animations of trees transforming between two crown shapes. Our proposed framework also provides an efficient way to interactively edit leafy trees toward desired shapes, demonstrating its potential to leverage existing tree modeling frameworks by providing flexible and intuitive tree editing operations.

Abstract We present a fast and automatic method to fit a given 3D garment onto a human model with various shapes and poses, without using a reference human model. Our approach uses a novel skeleton‐based error metric to find the pose that best fits the input garment. Specifically, we first generate the skeleton of the given human model and its corresponding skinning weights. Then, we iteratively rotate each bone to find its best position to fit the garment. After that, we rig the surface of the human model according to the transformations of the skeleton. Potential penetrations are resolved using collision handling and physically based simulation. Finally, we restore the human model back to the original pose in order to obtain the desired fitting result. Our experiment results show that besides its efficiency and automation, our method is about two orders of magnitudes faster than existing approaches, and it can handle various garments, including jacket, trousers, skirt, a suit of clothing, and even multilayered clothing.

We present a novel data-driven approach to populate virtual road networks with realistic traffic flows. Specifically, given a limited set of vehicle trajectories as the input samples, our approach first synthesizes a large set of vehicle trajectories. By taking the spatio-temporal information of traffic flows as a 2D texture, the generation of new traffic flows can be formulated as a texture synthesis process, which is solved by minimizing a newly developed traffic texture energy. The synthesized output captures the spatio-temporal dynamics of the input traffic flows, and the vehicle interactions in it strictly follow traffic rules. After that, we position the synthesized vehicle trajectory data to virtual road networks using a cage-based registration scheme, where a few traffic-specific constraints are enforced to maintain each vehicle's original spatial location and synchronize its motion in concert with its neighboring vehicles. Our approach is intuitive to control and scalable to the complexity of virtual road networks. We validated our approach through many experiments and paired comparison user studies.

Abstract We present an approach to fabricate shell objects with thickness parameters, which are computed to maintain the user‐specified structural stability. Given a boundary surface and user‐specified external forces, we optimize the thickness parameters according to stress constraints to extrude the surface. Our approach mainly consists of two technical components: First, we develop a patch‐based shell simulation technique to efficiently support the static simulation of extruded shell objects using finite element methods. Second, we analytically compute the derivative of stress required in the sensitivity analysis technique to turn the optimization into a sequential linear programming problem. Experimental results demonstrate that our approach can optimize the thickness parameters for arbitrary surfaces in a few minutes and well predict the physical properties, such as the deformation and stress of the fabricated object.

Abstract We present a novel morphing technique to generate pleasing visual effects between 2 topologically varying trees while preserving the topological consistency and botanical meanings of any in‐between shapes as natural trees. Specifically, we first efficiently convert leafy trees into botanically inspired chain‐lobe representations in an automatic way. With the aid of branching‐pattern aware, one‐to‐many correspondences between branches and leaves, we hierarchically interpolate branches of in‐between trees while maintaining their topological consistencies. Finally, we simultaneously interpolate foliage, specifically every single leaf, during the morphing process, avoiding the generation of unpleasant “floating” leaves. We demonstrate the effectiveness of our approach by creating visually compelling tree morphing animations, even between cross‐species.

Abstract In this paper, we present an innovative, animated traffic simulation method that we designed to feature an enhanced sense of reality and diversity of traffic flows. Instead of the typical one‐off initialization, our simulation method includes continuous, real trajectory data input providing an interactive control function that maximizes the characteristics of real‐world traffic flows. Our fusion models represent a comprehensive integration of the interactions among real‐data‐driven and virtual vehicles, thus depicting accurately the irregularity of traffic flows. Test results showed that animations generated via our proposed method depict inverse and irregular vehicle driving behaviors throughout the entire traffic flow.

Abstract Easy and efficient 3D modelling in virtual environments is an important and unsolved topic. This paper proposes a new modelling approach to tackle this issue. It invents convolution surface‐based brushes to directly draw 3D models in head‐mounted display‐based virtual environments. In order to maximize the efficiency, flexibility, and capacity of our proposed modelling approach, we propose three different skeleton‐based convolution surfaces to tackle different modelling tasks: point skeleton‐based convolution surfaces for metaball shapes, line skeleton‐based convolution surfaces for cylindrical shapes, and polygon skeleton‐based convolution surfaces for planar surfaces. Their combination makes 3D modelling more flexible and powerful. The high efficiency is further raised by our developed closed‐form solutions for point skeletons, ends of line skeletons, and edges of polygonal skeletons. Different user‐friendly sweeping schemes are provided to facilitate intuitive inputs for various complex shape generation. Unlike Google's Tilt Brush, which is used to create disconnected sheet‐like surfaces only, our proposed convolution surface‐based brushes can produce smoothly blended manifold surfaces, and novice users can easily learn and use them to create various interesting 3D models efficiently.

We present a new method to efficiently generate a set of morphologically diverse and inspiring virtual trees through hierarchical topology-preserving blending, aiming to facilitate designers' creativity production. By maintaining the topological consistency of the tree branches, sequences of similar yet different trees and novel intermediate trees with encouragingly interesting structures are generated by performing inner-species and cross-species blending, respectively. Hierarchical fuzzy correspondences are automatically established between two or multiple trees based on the multi-scale topology tree representations. Fundamental blending tasks including morph, grow and wilt are introduced and organized into a tree-structured blending scheduler, which not only introduces the randomness into the blending procedure but also wisely schedules the tasks to generate topology-aware blending sequences, contributing to a variety of resulting trees that exhibit diversities in both geometry and topology. Most significantly, multiple batches of blending can be executed in parallel, resulting in a rapid creation of a large repository of diverse trees.

We present a general solution for steering microrobotic swarm by dynamic actuating fields. In our approach, the motion of micro-robots is controlled by changing the actuating direction of a field applied to them. The time-series sequence of actuating field's directions can be computed automatically. Given a target position in the domain of swarm, a governing field is first constructed to provide optimal moving directions at every points. Following these directions, a robot can be driven to the target efficiently. However, when working with a crowd of micro-robots, the optimal moving directions on different agents can contradict with each other. To overcome this difficulty, we develop a novel steering algorithm to compute a statistically optimal actuating direction at each time frame. Following a sequence of these actuating directions, a crowd of micro-robots can be transported to the target region effectively. Our steering strategy of swarm has been verified on a platform that generates magnetic fields with unique actuating directions. Experimental tests taken on aggregated magnetic micro-particles are quite encouraging.

Abstract Several applications in shape modeling and exploration require identification and extraction of a 3D shape part matching a 2D sketch. We present CustomCut, an on‐demand part extraction algorithm. Given a sketched query, CustomCut automatically retrieves partially matching shapes from a database, identifies the region optimally matching the query in each shape, and extracts this region to produce a customized part that can be used in various modeling applications. In contrast to earlier work on sketch‐based retrieval of predefined parts, our approach can extract arbitrary parts from input shapes and does not rely on a prior segmentation into semantic components. The method is based on a novel data structure for fast retrieval of partial matches: the randomized compoundk‐NN graph built on multi‐view shape projections. We also employ a coarse‐to‐fine strategy to progressively refine part boundaries down to the level of individual faces. Experimental results indicate that our approach provides an intuitive and easy means to extract customized parts from a shape database, and significantly expands the design space for the user. We demonstrate several applications of our method to shape design and exploration.

Abstract Hidden images contain one or several concealed foregrounds which can be recognized with the assistance of clues preserved by artists. Experienced artists are trained for years to be skilled enough to find appropriate hidden positions for a given image. However, it is not an easy task for amateurs to quickly find these positions when they try to create satisfactory hidden images. In this paper, we present an interactive framework to suggest the hidden positions and corresponding results. The suggested results generated by our approach are sequenced according to the levels of their recognition difficulties. To this end, we propose a novel approach for assessing the levels of recognition difficulty of the hidden images and a new hidden image synthesis method that takes spatial influence into account to make the foreground harmonious with the local surroundings. During the synthesis stage, we extract the characteristics of the foreground as the clues based on the visual attention model. We validate the effectiveness of our approach by performing two user studies, including the quality of the hidden images and the suggestion accuracy.

Abstract In the paper, we present an online real‐time method for automatically transforming a basic locomotive motion to a desired motion of the same type, based on biomechanical results. Given an online request for a motion of a certain type with desired moving speed and turning angle, our method first extracts a basic motion of the same type from a motion graph, and then transforms it to achieve the desired moving speed and turning angle by exploiting the following biomechanical observations: contact‐driven center‐of‐mass control, anticipatory reorientation of upper body segments, moving speed adjustment, and whole‐body leaning. Exploiting these observations, we propose a simple but effective method to add physical and behavioral naturalness to the resulting locomotive motions without preprocessing. Through experiments, we show that our method enables a character to respond agilely to online user commands while efficiently generating walking, jogging, and running motions with a compact motion library. Our method can also deal with certain dynamical motions such as forward roll. Copyright © 2016 John Wiley & Sons, Ltd.

We present a stable and efficient simulator for deformable objects with collisions and contacts. For stability, an optimization derived from the implicit time integrator is solved in each timestep under the inequality constraints coming from collisions. To achieve fast convergence, we extend the MPRGP based solver from handling box constraints only to handling general linear constraints and prove its convergence. This generalization introduces a cost of solving dense linear systems in each step, but these systems can be reduced into diagonal ones for efficiency without affecting the general stability via pruning redundant collisions. Our solver is an order of magnitude faster, especially for elastic objects under large deformation compared with iterative constraint anticipation method (ICA), a typical method for stability. The efficiency, robustness and stability are further verified by our results.

Abstract Simulation of real‐world traffic scenarios is widely needed in virtual environments. Different from many previous works on simulating vehicles or pedestrians separately, our approach aims to capture the realistic process of vehicle–pedestrian interaction for mixed traffic simulation. We model a decision‐making process for their interaction based on a gap acceptance judging criterion and then design a novel environmental feedback mechanism for both vehicles' and pedestrians' behavior‐control models to drive their motions. We demonstrate that our proposed method can soundly model vehicle–pedestrian interaction behaviors in a realistic and efficient manner and is convenient to be plugged into various traffic simulation systems. Copyright © 2015 John Wiley & Sons, Ltd.

Abstract Creation of detailed character models is a very challenging task in animation production. Sketch‐based character model creation from a 3D template provides a promising solution. However, how to quickly find correct correspondences between user's drawn sketches and the 3D template model, how to efficiently deform the 3D template model to exactly match user's drawn sketches, and realize real‐time interactive modeling is still an open topic. In this paper, we propose a new approach and develop a user interface to effectively tackle this problem. Our proposed approach includes using user's drawn sketches to retrieve a most similar 3D template model from our dataset and marrying human's perception and interactions with computer's highly efficient computing to extract occluding and silhouette contours of the 3D template model and find correct correspondences quickly. We then combine skeleton‐based deformation and mesh editing to deform the 3D template model to fit user's drawn sketches and create new and detailed 3D character models. The results presented in this paper demonstrate the effectiveness and advantages of our proposed approach and usefulness of our developed user interface. Copyright © 2015 John Wiley & Sons, Ltd.

Abstract We present a novel approach to efficiently modelling branch structures with high‐quality meshes. Our approach has the following advantages. First, the limit surface can fit the target skeleton models as tightly as possible by reversely calculating the control vertices of subdivision surfaces. Second, high performance is achieved through our proposed analytical solutions and the parallel subdivision scheme on a graphics processing unit. Third, a smooth manifold quad‐only mesh is produced from the adopted Catmull–Clark scheme. A number of examples are given to demonstrate applications of our approach in various branch structures, such as tree branches, animal torsos, and vasculatures. Copyright © 2013 John Wiley & Sons, Ltd.

To solve the problem of generating segmentations of meaningful parts from scanned models with freeform surfaces, we explore a compact shape prior-based segmentation approach in this paper. Our approach is inspired by an observation that a variety of natural objects consist of meaningful components in the form of compact shape and these components with compact shape are usually separated with each other by salient features. The segmentation for multiregions is performed in two phases in our framework. First, the segmentation is taken in low-level with the help of discrete Morse complex enhanced by anisotropic filtering. Second, we extract components with compact shape by using agglomerative clustering to optimize the normalized cut metric, in which the affinities of boundary compatibility, 2D shape compactness and 3D shape compactness are incorporated. The practical functionality of our approach is proved by applying it to the application of customized dental treatment. Note to Practitioners-The research work presented in this paper is to support the procedure of customized design and manufacturing. As a very important preprocessing step for the industrial design of many applications, the 3D shape of real objects must be scanned and reconstructed in computer systems. To assign semantic information to the reconstructed mesh surface, the surface are segmented into meaningful components which, however, is not a well-defined problem. There is no general segmentation approach that has good performance for scanned models with freeform surfaces. According to the observation that models in many industrial applications (e.g., customized dental treatment) have meaningful components in the form of compact shape (e.g., teeth) separating from other regions (e.g., gum), a segmentation method is developed in this paper by using the compact shape prior. The techniques developed here can speedup the design and manufacturing of devices for customized dental treatment (e.g., orthodontic braces).

We present style-aware image cloning, a novel image editing approach for artworks, which allows users to seamlessly insert any photorealistic or artificial objects into an artwork to create a new image that shares the same artistic style with the original artwork. To this end, a real-time image transfer algorithm is developed to stylize the cloned object according to a distance metric based on the artistic styles and semantic information. Several interactive functions, such as layering, shadowing, semantic labeling, and direction field editing, are provided to enhance the harmonization of the composite image. Extensive experimental results demonstrate the effectiveness of our method.

This paper investigates the relationship between the extent of exaggeration in a caricature and its face identification ability. As face recognition is largely influenced by facial deformations, we focused on finding the borderline between likeness and unlikeness by applying gradual alterations to the face shape of the subject being studied. Suggestions on manipulating the degree of similarity when generating a caricature will be given. The experimental environment in this research can be used as a user-friendly caricature generation system based on Exaggerating the Difference From the Mean face, which allows a user to freely control each generation step and design his or her own unique caricature portrait.

Abstract In this paper, we present an on‐line real‐time physics‐based approach to motion control with contact repositioning based on a low‐dimensional dynamics model using example motion data. Our approach first generates a reference motion in run time according to an on‐line user request by transforming an example motion extracted from a motion library. Guided by the reference motion, it repeatedly generates an optimal control policy for a small time window one at a time for a sequence of partially overlapping windows, each covering a couple of footsteps of the reference motion, which supports an on‐line performance. On top of this, our system dynamics and problem formulation allow to derive closed‐form derivative functions by exploiting the low‐dimensional dynamics model together with example motion data. These derivative functions and their sparse structures facilitate a real‐time performance. Our approach also allows contact foot repositioning so as to robustly respond to an external perturbation or an environmental change as well as to perform locomotion tasks such as stepping on stones effectively.

Abstract Collective behaviour of winged insects is a wondrous and familiar phenomenon in the real world. In this paper, we introduce a highly efficient field‐based approach to simulate various insect swarms. Its core idea is to construct a smooth yet noise‐aware governing velocity field that can be further decomposed into two sub‐fields: (i) a divergence‐free curl‐noise field to model noise‐induced movements of individual insects in a swarm, and (ii) an enhanced global velocity field to control navigational paths in a complex environment along which all the insects in a swarm fly. Through simulation experiments and comparisons with existing crowd simulation approaches, we demonstrate that our approach is effective to simulate various insect swarm behaviours including aggregation, positive phototaxis, sedation, mass‐migrating, and so on. Besides its high efficiency, our approach is very friendly to parallel implementation on GPUs (e.g. the speedup achieved through GPU acceleration is higher than 50 if the number of simulated insects is more than 10 000 on an off‐the‐shelf computer). Our approach is the first multi‐agent modelling system that introduces curl‐noise into agents' velocity field and uses its non‐scattering nature to maintain non‐colliding movements in 3D crowd simulation.

ABSTRACT We propose a new animation technique, called flock morphing, to create special morphing effects between two arbitrary 3D objects by combining the features of 3D morphing and flock animation. Its core idea is first to tetrahedralize the source 3D mesh and regard each tetrahedron as an agent in a flock and then continually generate the flock morphing animation until the target mesh emerges, formed by the same set of tetrahedra. By applying plausible trajectory planning scheme and smooth deformation algorithm, we demonstrate that our proposed method can simultaneously achieve visually desired morphing effects. Copyright © 2014 John Wiley & Sons, Ltd.

ABSTRACT We present a novel procedural synthesis method to improve small‐scale turbulence details for controllable smoke animation constrained by shapes and paths. In order to enhance fluid details without introducing unpleasing fluid control effects, we propose a spatial–temporal varying synthesis parameter to control turbulence behaviors and compute it from control force and the vorticity velocity. Our approach can control enhanced turbulence behaviors efficiently and produces visually plausible realistic fine‐scale details while reducing artifacts of large‐scale noises on fluid control. We compare our algorithm to existing procedural synthesis ones to validate its efficiency and controllability. Copyright © 2014 John Wiley & Sons, Ltd.

We propose a 2D shape deformation method to fit technical drawings of garments (“flats”) to body figure drawings with a diversity of fashion poses. We first dress a flat onto a body figure in a standard standing pose using Radial Basis Function (RBF) mapping. For different types of clothing, we suggest two levels of treatment to determine handles automatically. We deform the flats using the selected handles to create realistic fashion sketches that fit the garments onto fashion figures in different poses. Shape deformation is performed to minimize the distortion of all the triangles of the garment mesh and preserve garment properties in the deformation. Finally, the garment details, such as style lines and seams, are deformed accordingly for realistic deformation results. Experimental results have shown that our method can deform various garment flats to fit fashion figures in different poses.

We present a novel method for elastic animation editing with space-time constraints. In a sharp departure from previous approaches, we not only optimize control forces added to a linearized dynamic model, but also optimize material properties to better match user constraints and provide plausible and consistent motion. Our approach achieves efficiency and scalability by performing all computations in a reduced rotation-strain (RS) space constructed with both cubature and geometric reduction, leading to two orders of magnitude improvement over the original RS method. We demonstrate the utility and versatility of our method in various applications, including motion editing, pose interpolation, and estimation of material parameters from existing animation sequences.

We present a real-time solution for generating detailed clothing deformations from pre-computed clothing shape examples. Given an input pose, it synthesizes a clothing deformation by blending skinned clothing deformations of nearby examples controlled by the body skeleton. Observing that cloth deformation can be well modeled with sensitivity analysis driven by the underlying skeleton, we introduce a sensitivity based method to construct a pose-dependent rigging solution from sparse examples. We also develop a sensitivity based blending scheme to find nearby examples for the input pose and evaluate their contributions to the result. Finally, we propose a stochastic optimization based greedy scheme for sampling the pose space and generating example clothing shapes. Our solution is fast, compact and can generate realistic clothing animation results for various kinds of clothes in real time.

We present stego-marbling-texture, a new and unique texture design method which allows users to deliver personalized messages with beautiful marbling textures. Our approach is inspired by the success of the recent work on modeling traditional marbling operations as mathematical functions. The encrypter transforms an input image or a text message into an intricate marbling pattern using marbling operations defined as reversible functions, and the decrypter recovers the input image or message through reversing the process of marbling operations. When applying marbling operations, the parameters of operations are automatically recorded, encrypted, and then invisibly embedded into the marbling pattern to create a stego-marbling-texture. In this way, the decrypter can be implemented as a stand along software, enabling the receiver to extract the hidden message from the stego-marbling-texture without requiring any extra information from the sender. To ensure that the message is unnoticeably and beautifully covered by the marbling texture, we propose a new technique for automatically creating a background which is harmonious with the input message based on a set of visual perception cues.

Partial differential equation-based surface modelling is a new approach of creating and manipulating three-dimensional geometric models. It uses the solution to a vector-valued partial differential equation subjected to suitably defined boundary constraints to carry out surface modeling. This paper provides a survey on this approach which summarizes various mathematical models of partial differential equation-based surface modelling, accurate and approximate analytical solutions as well as numerical solutions of the mathematical models, and the applications of partial differential equation-based surface modelling. It also discusses some future research directions of partial differential equation-based surface modelling.

ABSTRACT We present an intuitive and interactive approach for motion editing through space–time constraints on positions. Given an input motion of an elastic body, our approach enables the user to interactively edit node positions in order to alter and fine‐tune the motion. We formulate our motion editing as an optimization problem with dynamics constraints to enforce a physically plausible result. Through linearization of the editing around the input trajectory, we simplify this constrained optimal control problem into an unconstrained quadratic optimization. The optimal motion thus becomes the solution of a dense linear system, which we solve efficiently by applying the adjoint method in each iteration of a conjugate gradient solver. We demonstrate the efficiency and quality of our motion editing technique on a series of examples. Copyright © 2013 John Wiley & Sons, Ltd.

This paper presents a single-view hair modeling technique for generating visually and physically plausible 3D hair models with modest user interaction. By solving an unambiguous 3D vector field explicitly from the image and adopting an iterative hair generation algorithm, we can create hair models that not only visually match the original input very well but also possess physical plausibility (e.g., having strand roots fixed on the scalp and preserving the length and continuity of real strands in the image as much as possible). The latter property enables us to manipulate hair in many new ways that were previously very difficult with a single image, such as dynamic simulation or interactive hair shape editing. We further extend the modeling approach to handle simple video input, and generate dynamic 3D hair models. This allows users to manipulate hair in a video or transfer styles from images to videos.

Abstract We present an alternative approach to create digital camouflage images which follows human's perception intuition and complies with the physical creation procedure of artists. Our method is based on a two‐scale decomposition scheme of the input images. We modify the large‐scale layer of the background image by considering structural importance based on energy optimization and the detail layer by controlling its spatial variation. A gradient correction is presented to prevent halo artifacts. Users can control the difficulty level of perceiving the camouflage effect through a few parameters. Our camouflage images are natural and have less long coherent edges in the hidden region. Experimental results show that our algorithm yields visually pleasing camouflage images.

We present an engine for enhancing the geometry of a 3D face mesh model while making the enhanced version share close similarity with the original. After obtaining the feature points of a given scanned 3D face model, we first perform a local and global symmetrization on the key facial features. We then apply an overall proportion optimization to the frontal face based on Neoclassical Canons and golden ratios. A nonlinear least-squares solution is adopted to adjust the feature points so that the face profile complies with the aesthetic criteria, which are derived from the profile cosmetology. Through the above processes, we obtain the optimized feature points, which will lead to a more attractive face. According to the original feature points and the optimized ones, we perform Laplacian deformation to adjust the remaining points of the face in order to preserve the geometric details. The analysis of user study in this paper validates the effectiveness of our 3D face geometry enhancement engine.

This paper propose a new real-time image smoothing algorithm based on true edges. We observed that the pixels marked as edge points by the well-known Canny operator are as close as possible to the center of true edges. This gives us an opportunity to judge how well edges are preserved for an edge-preserving smoothing filter. We therefore propose a nonlinear edge-preserving interpolation technique guided by the Canny true edges in an image. Our new edge-preserving smoothing algorithm smoothes high-contrast details and, at the same time, effectively preserves the true edges in the input image. In order to achieve a real-time performance, we further implement the algorithm using NVIDIA CUDA by taking advantage of the high parallelism of modern graphics hardware. The proposed algorithm smoothes low-contrast regions while effectively preserving true edges, and can also be used to produce vivid abstracted effects from realistic photographs.

In this paper, we present a novel color-mood-aware technique to re-texture clothing in a photograph. An efficient classification algorithm is developed to classify clothing textures using color mood scheme. To re-texture the clothing, our approach first computes the gradient maps for the cloth region to be replaced and then calculates the texture distortion coordinates on the projected cloth region according to the gradient maps. After the user selects a target clothing texture from the classified clothing texture database, the lighting and shading effects on the original photograph is transferred using the HSV color space. Experimental results show that the proposed approach successfully re-textures the clothes in photographs while preserving the geometry and lighting features.

Abstract Adding interactive haptic‐constraint sensations is important in interactive computer gaming and 3D shape design. Usually constraints are set on vertices of the object to drive the deformation. How to simulate dynamic force constraints in interactive design is still a challenging task. In this paper, we propose a novel haptic‐constraint modeling method based on interactive metaballs, during which the haptic‐constraint tools are attracted to the target location and then control the touch‐enabled deformation within the constrained areas. The interactive force feedbacks facilitate designers to accurately deform the target regions and fine carve the details as their intention on the objects. Our work studies how to apply touch sensation in such constrained deformations using interactive metaballs, thus users can truly feel and control the soft‐touch objects during the deforming interactions. Experimental results show that the dynamic sense of touch during the haptic manipulation is intuitively simulated to users, via the interacting interface we have developed. Copyright © 2010 John Wiley & Sons, Ltd.

Abstract This paper develops an efficient, physically based shape manipulation technique. It defines a 3D model with profile curves, and uses spine curves generated from the profile curves to control the motion and global shape of 3D models. Profile and spine curves are changed into profile and spine wires by specifying proper material and geometric properties together with external forces. The underlying physics is introduced to deform profile and spine wires through the closed form solution to ordinary differential equations for axial and bending deformations. With the proposed approach, global shape changes are achieved through manipulating spine wires, and local surface details are created by deforming profile wires. A number of examples are presented to demonstrate the applications of our proposed approach in shape manipulation. Copyright © 2010 John Wiley & Sons, Ltd.

The 3D clothing fitting on a body model is an important research topic in the garment computer aided design (GCAD). During the fitting process, the match between the clothing and body models is still a problem for researchers. In this paper, we provide a 3D clothing fitting method based on the feature point match. We firstly use a new cubic-order weighted fitting patch to estimate the geometric properties of each vertex on two mesh models. Feature points are then extracted from two models and a new matching function is constructed to match them according to curvature and torsion. We interactively select several key feature points from two limited feature point sets to compute the transformation matrix of the clothing model. Finally the second match is performed to achieve the precise match between the clothing and body models. The experimental results show that our 3D clothing fitting method is simple and effective.

This paper presents an algorithm to approximate a solid model by a hierarchical set of bounding ellipsoids having optimal shape and volume approximation errors. The ellipsoid-tree is constructed in a top-down splitting framework. Starting from the root of hierarchy the volume occupied by a given model is divided into k sub-volumes where each is approximated by a volume bounding ellipsoid and will be later subdivided into k ellipsoids for the next level in hierarchy. The difficulty for implementing this algorithm comes from how to evaluate the volume of an ellipsoid outside the given model effectively and efficiently (i.e., the outside-volume-error). A new method - analytical computation based - is presented in this paper to compute the outside-volume-error. One application of ellipsoid-tree approximation has also been given at the end of the paper.

This paper presents a novel bilateral filtering using fuzzy-median for image manipulations such as denoising and tone mapping. Our proposed bilateral filtering consists of the standard bilateral filter and the estimation of the pixel values by the fuzzy median filter. We have applied the proposed fuzzy filtering for image denoising with both the impulse and Gaussian random noise, which achieves better results than the bilateral filtering based denoising approaches, the Perona-Maliks anisotropic diffusion filter, the fuzzy vector median filter and the non-local means filter. Further, we develop the tone mapping algorithm of high dynamic range images incorporating the proposed fuzzy filtering, which does not introduce unpleasant visual halo artifacts.

This paper presents a novel approach for real-time photo style transfer. The automatic image manipulation technique is performed in the oRGB color space, which is a new color model based on the psychologically opponent color theory. We transfer color from an appropriate source image to the target image using a simple statistical analysis. In addition, we match the global luminance histogram to achieve better photographic look. Note that the whole pipeline is highly parallel, enabling a GPU-based real-time implementation. Several experimental results are shown to demonstrate the effectiveness and efficiency of the proposed method.

In this paper we present a novel tone mapping algorithm for high dynamic range (HDR) images using the retinal adaptation model (RAM). The physiological evidence suggests that the RAM is obtained by measuring intensity-response functions to flashes of light presented under varying adaptation conditions, which leads to a theoretic-sound model that can be flexibly adapted for tone reproduction. The multiplicative-subtractive process of the model can provide high quality tone mapping results for rendering the HDR images. The experimental results demonstrate that our RAM-based tone mapping approach is effective to produce pleasing results on HDR images in a wide range of real-world scenarios.

In this paper, we extend Razdan and Bae's second-order local fitting method [11] to construct an effective third-order fitting patch. Compared to other estimation algorithms, this weighted bicubic Bézier patch more accurately obtains the normal vector and curvature estimation of a triangular mesh model. Furthermore, we define the principal geodesic torsion of each vertex on the mesh model and estimate it through this local fitting patch. In the end of this paper, we apply the third-order fitting patch for the mesh smoothing and hole-filling which can get the satisfactory results.

Abstract This paper presents the fireworks controller, a novel real‐time shape‐constrained fireworks animation system. We depict the shape of a firework by a 3D mesh. In order to approximate the mesh using evenly distributed points, we propose a fast point sampling method by extending the dual depth peeling algorithm. The samples are then taken as input to shape‐constrained fireworks whose physically plausible animations are based on inverse dynamics. We present a highly parallel iterative clustering algorithm to support multi‐level fireworks explosion. In order to simulate natural fuzzy fireworks, we impose extra random particles with a parallel random number generator. Several novel intuitive user interfaces are introduced to improve the usability of the system. Experimental results demonstrate the prettiness and efficiency of the proposed approach. Copyright © 2009 John Wiley & Sons, Ltd.

High dynamic range rendering attempts to take an HDR image and produce a more realistic representation on a limited range computer monitor. Although several tone mapping operators have been proposed in recent years, no evaluation has yet been undertaken to explore which operator is more suitable for hardware implementation. In this paper, we begin with our novel GPU implementations of two state-of-the-art operators in real time. Then several experimental results using eight GPU-based tone mapping operators are presented to evaluate which one is better with regard to running efficiency. Our GPU implementation of the Pattanaik operator can achieve real-time performance even on high-resolution HDR images. In addition, we believe that many real-time applications, including HDR video player and environment mapping with HDR textures in games, will benefit from our novel approach.

Abstract We propose a novel shape‐constrained flock animation system for interactively controlling flock navigation in virtual environments. This system is capable of making the spatial distribution of a flock meet static or deforming shape constraints while performing flock simulation. Such a capability can find many applications in the entertainment industry. Given a 3D constraining shape, our system first draws a set of uniform sample points through a 3D surface mosaicing process or a stratified point sampling strategy. Once correspondences between flock members and sample points have been established, points on the target shape are used as homing destinations to guide flock migration. Under a global path control scheme, an effective fuzzy control logic, which dynamically adjusts steering forces and control forces, has been developed to create visually pleasing shape‐constrained flock animations. Copyright © 2008 John Wiley & Sons, Ltd.

Abstract This paper addresses the problem of simulating the mechanical behavior of cloth in computer animation, which is very important and challenging. The micro‐structure of woven fabrics leads to significantly different shear reaction from other sheet materials, which has been neglected in previous approaches of cloth simulation. Therefore, it is beneficial for cloth simulation to model the shear buckling and structural bending separately. We analyze the shear buckling yielded by the micro‐structure of woven and the dynamic bending based on the thin‐shell theory, and develop a compact implementation of the new model on mass‐spring systems. Experimental results show that the animations generated using this technique are with wrinkles and folds appearing and vanishing in a more natural way than other approaches. Copyright © 2008 John Wiley & Sons, Ltd.

This paper presents a new approach for the mesh composition on models with arbitrary boundary topology. After cutting the needed parts from existing mesh models and putting them into the right pose, an implicit surface is adopted to smoothly interpolate the boundaries of models under composition. An interface is developed to control the shape of the implicit transient surface by using sketches to specify the expected silhouettes. After that, a localized Marching Cubes algorithm is investigated to tessellate the implicit transient surface so that the mesh surface of composed model is generated. Different from existing approaches in which the models under composition are required to have pairwise merging boundaries, the framework developed based on our techniques have the new function to fuse models with arbitrary boundary topology.

This paper presents a novel mesh denoising and smoothing method. We first estimate the principal curvatures and mesh saliency value of each vertex of the mesh, the uniform principal curvatures of a vertex is then calculated based on the weighted average of local mesh saliency values. The weighted bi-cubic Bezier surface is used to fit the neighborhood of a vertex by the least square method, and the new vertex position is obtained by adjusting the parameters of the local fitting surface. Experiments show that the smoothing method has the following advantages. (1). It can preferably keep the geometry feature of the original mesh model. (2). It can prevent the volume shrinkage of mesh efficiently. (3). The method can attain the smoothing boundaries of non-closed mesh model.

As ellipsoids have been employed in the collision handling of many applications in physical simulation and robotics systems, we present a novel algorithm for generating a bounding volume hierarchy (BVH) from a given model with ellipsoids as primitives. Our algorithm approximates the given model by a hierarchical set of optimized bounding ellipsoids. The ellipsoid-tree is constructed by a top-down splitting. Starting from the root of hierarchy, the volume occupied by a given model is divided into k sub-volumes where each is approximated by a volume bounding ellipsoid. Recursively, each sub-volume is then subdivided into ellipsoids for the next level in the hierarchy. The k ellipsoids at each hierarchy level for a sub-volume bounding is generated by a bottom-up algorithm - simply, the sub-volume is initially approximated by m spheres (m » k), which will be iteratively merged into k volume bounding ellipsoids and globally optimized to minimize the approximation error. Benefited from the anisotropic shape of primitives, the ellipsoid-tree constructed in our approach gives tighter volume bound and higher shape fidelity than another widely used BVH, sphere-tree.

Interactive control is one of the key issues when simulating crowd navigation in virtual environment. In this paper, we propose a simple but practical method for authoring crowd scenes in an effective and intuitive way. Radial Basis Functions (RBF) based vector field is employed as the governing tool to drive the motion flow. With this basic mathematical tool, users can easily control the motions of crowd by simply sketching velocities on a few points in the scene. Our approach is fast enough to allow on-the-fly modification of the vector field. Besides, the behavior of an individual in a crowd can be interactively adjusted by changing the ratio between its autonomous and governed movements.

Abstract It is crucial for the users to touch, grasp and manipulate the interested objects through our sense of touch in many interactive applications, such as on‐line computer games, interactive cartoon design, and virtual prototyping. In this paper, we propose an interactive haptic deformation approach which incorporates the dynamic simulation of mass–spring systems and flexible control of free‐form deformation in the touch‐enabled soft‐object deformation. Through distributing mass, spring and damping coefficients of the object to the bounded Bezier volume lattice, the deformation of the object related to the haptic avatar follows the physical laws and has high working rate. Both homogenous and inhomogenous materials are simulated. The anchor nodes of haptic input are specified to create amazing special effects during the interactive haptic deformation. Interactive haptic deformations of three‐type tropic fishes, Angel, Demekin, and GuppyBlueGrass, have been experimented to simulate vivid fish swimming processes in the virtual ocean scene. Our proposed approach provides touch‐enabled input and efficient performance in the flexible deforming controls, letting the objects move in a dynamic, cartoon‐style deforming manner. Copyright © 2007 John Wiley & Sons, Ltd.

We present a new high quality tessellation method for implicit surfaces in this paper. The approach can handle arbitrary implicit functions and dynamic implicit surfaces based on skeletal primitives. We first samples the implicit surface uniformly using particle fission and floating, then reconstructs a triangular mesh from the sample points using ball pivoting algorithm (BPA). Finally, we subdivide the reconstructed surface using a 1 to 4 subdivision scheme to obtain the high quality implicit surface tessellation.

Physical-based cloth modeling and animation is a hot topic in computer graphics. Recently the semi-implicit integration scheme has been widely employed in cloth animation because of its high stability. In this paper, we propose an efficient and simple iterative scheme to create realistic cloth animation. The linear system created in the semi-implicit scheme can be solved in approximately linear time complexity in one time-step. This greatly benefits real-time applications. We also improve the traditional mass-spring model by avoiding the shrink of the cloth model.

Abstract In this paper, we propose a novel mesh morphing approach based on polycubic cross‐parameterization. We compose parameterizations over the surfaces of the polycubes whose shape is similar to that of the given meshes. Because the polycubes capture the large‐scale features, we can easily preserve the shape of the models, mapping legs to legs, head to head, and so on. For the finer features that are not reflected by the shape of the polycubes, we split the polycubes into matching patches and optimize them to get a low‐distortion bijection that satisfies user‐prescribed constraints. Our approach works well for meshes with arbitrary genus as long as the polycubes capture this feature and transfers texture seamlessly. We can also build maps with singularities between models with different genus. Copyright © 2005 John Wiley & Sons, Ltd.

Abstract In this paper, we propose a novel practical method for blob‐based liquid 3D morphing. Firstly, blobby objects are employed to approximate a given polygonal surface. The primitives in the medial axis sphere‐tree of a polygonal model are utilized as initial blobs—this greatly improves the robustness and efficiency of the blob‐based approximation. Secondly, we establish the blob correspondences between two models by sphere cellular matching and hierarchical matching. Finally, we interpolate the parameters of the implicit representation to get the intermediate shapes. Experiments show our method can produce visually pleasing liquid morphing effects. Copyright © 2005 John Wiley & Sons, Ltd.

Space deformation is an important tool in computer animation and shape design. We propose a new local deformation model based on generalized metaballs. The user specifies a series of constraints, which can be made up of points, lines, surfaces and volumes, their effective radii and maximum displacements; the deformation model creates a generalized metaball for each constraint. Each generalized metaball is associated with a potential function centered on the constraint, the potential function drops from 1 on the constraint to 0 on the effective radius. This deformation model operates on the local space and is independent of the underlining representation of the object to be deformed. The deformation can be finely controlled by adjusting the parameters of the generalized metaballs. We also present some extensions and the extended deformation model to include scale and rotation constraints. Experiments show that this deformation model is efficient and intuitive. It can deal with various constraints, which is difficult for traditional deformation model.

In real life, some objects may deform along axial curves and the lengths of their skeletons usually remain constant during the axial deformation, such as a swimming fish, a swaying tree, etc. This paper presents a practical approach of arc-length-based axial deformation and axial-length-preserved animation. The space spanned by the arc-length parameter and the rotation-minimizing frame on the axis is taken as the embedding space. During animation, the keyframe axial curves are consistently approximated by polylines after sufficient subdivisions and both the edge lengths and the directional vertex angles of the keyframe polylines (or unit edge vectors) are then interpolated to generate the intermediate polylines which are regarded as the discrete expressions of the intermediate axes. Experiments show that our method is very useful, intuitive and easy to control.