Detecting Schelling Points—salient 3D mesh landmarks that serve as natural reference points for shape analysis—is a challenging problem in geometry processing. While existing CNN-based methods struggle with limited receptive fields and poor geometric context modeling, this paper proposes {\em SchellingFormer}, a novel Laplacian matrix-guided Geometric Transformer that effectively captures long-range dependencies and discriminative geometric features for robust Schelling point prediction. Our framework consists of two key components: (i) a hybrid geometric feature embedding module that integrates handcrafted descriptors (coordinates, Gaussian curvature, and curvature differences) to encode local geometry, and (ii) a Laplacian-driven vector attention mechanism, where spatial relationships encoded by the Laplacian matrix guide feature aggregation with the Transformer. This approach enables adaptive, geometry-aware message passing and contextual representation learning. Extensive experiments demonstrate that SchellingFormer outperforms state-of-the-art methods across multiple evaluation metrics. Our work bridges the gap between spectral mesh analysis and Transformer-based learning, offering a powerful tool for 3D shape understanding tasks such as shape matching and saliency detection.

Reconstructing precise CAD modeling sequences from point clouds remains a challenging task, especially for objects with complex geometry and topology. In this paper, by formulating the CAD sequence reconstruction as a Markov decision process, we introduce ReACT, a novel Reward-informed Autoregressive decision Cad Transformer architecture for robust CAD sequence prediction. Beyond previous imitation-only approaches, our key innovation is to frame the CAD Transformer under a reinforcement learning paradigm and thereby integrate reward-inspired heuristic learning into our architecture. This allows ReACT to effectively leverage shape-aware long-term reward feedback to guide the inference of (nearly) optimal CAD commands. Specifically, conditioned on past tokens, comprising the historical CAD states, sketch-extrude commands (i.e., actions) and associated geometric rewards, ReACT autoregressively outputs the most promising CAD commands in a causal manner. In particular, we develop a novel scaffold-aware CAD state representation that integrates global point-command features with an incrementally constructed surface point scaffold, enabling fine-grained geometric reasoning for subsequent reconstruction prediction. Moreover, an effective local barrel points-guided dense reward function is designed to jointly evaluate surface fidelity and command efficiency for reliable reward guidance. Extensive evaluations on the DeepCAD and Fusion360 benchmarks demonstrate that ReACT can achieve superior CAD reconstruction quality, even for objects with complex shapes.

This paper addresses the challenge of estimating local surface differential properties, specifically surface normals and curvatures, from raw 3D point clouds. Traditional methods either rely on fitting pre-defined analytic surfaces risking model bias, or directly regress normals and curvatures overlooking their intrinsic geometric correlation. We propose a learning-based approach that locally fits osculating implicit quadrics to recover both normals and curvatures simultaneously. Drawing on classical differential geometry, we exploit the fact that every point on a C² surface admits an osculating quadric in Monge form that exactly reproduces local differential properties. However, the Monge frame itself depends on the very differential quantities being estimated. To bypass this circularity, we reformulate the Monge-form quadric as an implicit representation in a canonical local frame derived solely from point coordinates, enabling supervised learning without requiring Monge frame alignment. This reformulation allows us to construct a ground-truth dataset of such local-frame quadrics and train a neural network to predict per-point weights and offsets for a robust weighted least squares fitting process. The learned offsets account for the deviations of neighboring points from the idealized osculating surface. We further incorporate stable curvature formulations into the training loss alongside normal supervision to enhance estimation fidelity. Extensive experiments on diverse datasets demonstrate that our method outperforms prior approaches in normal and curvature estimation from raw point clouds.

The orientation of surface normals in 3D point cloud is a fundamental problem in computer vision and graphics. Determining a globally consistent orientation solely from the point cloud is however challenging due to the global scope of the problem and the discrete nature of point cloud, particularly in the presence of noise, outliers, holes, thin structures, and complex topologies. This paper presents an efficient, robust, and global algorithm for generating consistent normal orientation of a dense 3D point cloud. The basic idea is to transform the original binary normal orientation problem to finding a relaxed sign field on a Delaunay graph, which can be achieved by solving a sparse linear system. The Delaunay graph is constructed by triangulating a level set of an implicit function defined from the input point cloud. The shape diameter function is estimated to serve as a prior for determining an appropriate level value such that the level set implicitly defines the inner and outer shells enclosing the input point clouds. As such, our algorithm leverages the strengths of the shape diameter function, Delaunay triangulation, and the least-square techniques, making the underlying processes take both geometry and topology into consideration, and thus provides an efficient and robust solution for handling point clouds with complicated geometry and topology. Extensive experiments on various shapes with noise and outliers confirm the effectiveness and robustness of our algorithm.

This paper introduces ZeroMatch, a novel zero-shot RGB-D point cloud registration framework, aimed at achieving robust 3D matching on unseen data without any task-specific training. Our core idea is to utilize the powerful zero-shot image representation of Stable Diffusion, achieved through extensive pre-training on large-scale data, to enhance point-cloud geometric descriptors for robust matching. Specifically, we combine the handcrafted geometric descriptor FPFH with Stable-Diffusion features to create point descriptors that are both locally and contextually aware, enabling reliable RGB-D registration with zero-shot capability. This approach is based on our observation that Stable-Diffusion features effectively encode discriminative global contextual cues, naturally alleviating the feature ambiguity that FPFH often encounters in scenes with repetitive patterns or low overlap. To further enhance cross-view consistency of Stable-Diffusion features for improved matching, we propose a coupled-image input mode that concatenates the source and target images into a single input, replacing the original single-image mode. This design achieves both inter-image and prompt-to-image consistency attentions, facilitating robust cross-view feature interaction and alignment. Finally, we leverage feature nearest neighbors to construct putative correspondences for hypothesize-and-verify transformation estimation. Extensive experiments on 3DMatch, ScanNet, and ScanLoNet verify the excellent zero-shot matching ability of our method. [Code]

In this paper, we propose a novel 3D registration paradigm, Generative Point Cloud Registration, which bridges advanced 2D generative models with 3D matching tasks to enhance registration performance. Our key idea is to generate cross-view consistent image pairs that are well-aligned with the source and target point clouds, enabling geometric-color feature fusion to facilitate robust matching. To ensure high-quality matching, the generated image pair should feature both 2D-3D geometric consistency and cross-view texture consistency. To achieve this, we introduce Match-ControlNet, a matching-specific, controllable 2D generative model. Specifically, it leverages the depth-conditioned generation capability of ControlNet to produce images that are geometrically aligned with depth maps derived from point clouds, ensuring 2D-3D geometric consistency. Additionally, by incorporating a coupled conditional denoising scheme and coupled prompt guidance, Match-ControlNet further promotes cross-view feature interaction, guiding texture consistency generation. Our generative 3D registration paradigm is general and could be seamlessly integrated into various registration methods to enhance their performance. Extensive experiments on 3DMatch and ScanNet datasets verify the effectiveness of our approach.

This paper presents PartCAD, a novel framework for reconstructing CAD modeling sequences directly from point clouds by projection-guided, part-aware geometry reasoning. It consists of (1) an autoregressive approach that decomposes point clouds into part-aware latent representations, serving as interpretable anchors for CAD generation; (2) a projection guidance module that provides explicit cues about underlying design intent via triplane projections; and (3) a non-autoregressive decoder to generate sketch-extrusion parameters in a single forward pass, enabling efficient and structurally coherent CAD instruction synthesis. By bridging geometric signals and semantic understanding, PartCAD tackles the challenge of reconstructing editable CAD models—capturing underlying design processes—from 3D point clouds. Extensive experiments show that PartCAD significantly outperforms existing methods for CAD instruction generation in both accuracy and robustness. The work sheds light on part-driven reconstruction of interpretable CAD models, opening new avenues in reverse engineering and CAD automation.

The 3D-printed mesh screen with dense penetrating pores is a new structure for massive manufacturing of molded pulp package products. However, some of the pores may be clogged by the printing material powder during the printing process. Such defects negatively affect the quality of the pulp packages produced using the mesh screen mold. To pinpoint the defects, we design a model-based robotic visual sensing system, called RoboCam, which uses a robotic arm to carry a high-resolution camera for full inspection of a mold consisting of joined mesh screens. To inspect the entire mold, RoboCam plans the camera poses to capture multiple images of the mold and render synthesized images as references for identifying the clogged pores. In particular, we propose novel designs to rectify the inherent run-time pose errors of the robotic system for ensuring the reference quality and to accelerate the reference rendering for reducing inspection latency. Extensive evaluation shows that RoboCam’s design outperforms various baselines, including three existing computer vision and convolution neural network-based inspection systems. RoboCam achieves a recall rate of 94.95% within 528 seconds latency for inspecting an entire mold with 13,000 designed pores.

We propose a new method for computing smooth and integrable cross fields on 2D and 3D surfaces. our approach first computes smooth cross fields by minimizing the Dirichlet energy. Unlike existing optimization-based methods, our technique determines the singularity configuration-i.e., the number, locations, and indices of singularities-by iteratively adjusting them. Singularities can move, merge and split, akin to the behavior of like charges repelling and unlike charges attracting. Once all singularities stop moving, we obtain a cross field with (locally) the lowest Dirichlet energy. In simply connected domains, this cross field is guaranteed to be integrable. However, this property does not hold in multiply connected domains. To make a smooth cross field integrable, we construct a vector field $\bf c$c that characterizes the deviation of the cross field from a curl-free field. We then optimize the locations of singularities by moving them along the field lines of $\bf c$c. Our method is fundamentally different from existing integer programming-based approaches, as it avoids combinatorial optimization. It is fully automatic and includes a parameter to control the number of singularities. Our method is well suited for smooth models where exact boundary alignment and sparse hard directional constraints are desired, and can guide seamless conformal parameterization and T-junction-free quadrangulation.

The article presents a new method for constructing self-supporting surfaces using arch beams that are designed to convert their thrust into supporting force, thereby eliminating shear stress and bending moments. Our method allows for the placement of the arch beams on the boundary or within a surface and partitions the surface into multiple self-supporting parts. The use of arch beams enhances stability and durability, adds aesthetic appeal, and allows for greater flexibility in the design process. We develop an iterative algorithm for designing self-supporting surfaces with arch beams that enables the user to control the shape of the beams and surface through intuitive parameters and specify the desired location of the arch beams. We verify the physical stability of the structure using finite element analysis. Experimental results show that our method can produce visually pleasing self-supporting surfaces that satisfy the equilibrium equation with high accuracy.

Rendering and reconstruction are long-standing topics in computer vision and graphics. Achieving both high rendering quality and accurate geometry is a challenge. Recent advancements in 3D Gaussian Splatting (3DGS) have enabled high-fidelity novel view synthesis at real-time speeds. However, the noisy and discrete nature of 3D Gaussian primitives hinders accurate surface estimation. Previous attempts to regularize 3D Gaussian normals often degrade rendering quality due to the fundamental disconnect between normal vectors and the rendering pipeline in 3DGS-based methods. Therefore, we introduce Normal-GS, a novel approach that integrates normal vectors into the 3DGS rendering pipeline. The core idea is to model the interaction between normals and incident lighting using the physically-based rendering equation. Our approach re-parameterizes surface colors as the product of normals and a designed Integrated Directional Illumination Vector (IDIV). To optimize memory usage and simplify optimization, we employ an anchor-based 3DGS to implicitly encode locally-shared IDIVs. Additionally, Normal-GS leverages optimized normals and Integrated Directional Encoding (IDE) to accurately model specular effects, enhancing both rendering quality and surface normal precision. Extensive experiments demonstrate that Normal-GS achieves near state-of-the-art visual quality while obtaining accurate surface normals and preserving real-time rendering performance.

Optimizing heterogeneous elastic material distribution on a 3D part to achieve desired deformation behavior is an important task in computer-aided design and additive manufacturing . This paper presents a solution to this problem, which involves interactive design, automatic deformation generation, and optimization of spatial distribution of heterogeneous elastic materials. Our method improves previous techniques in three aspects. First, we incorporates a geometric deformation-based interactive design into FEM-based optimization, which makes the solution less dependent of initial guesses of Young’s modulus values and it more likely to produce the target design even with sparse user input of displacements and forces at a limited set of mesh vertices. Second, we formulate the problem as an L 2 - or L 0 -optimization problem. The L 2 formulation outputs smoothly varying heterogeneous material distribution that accommodates multiple functions within a single part. The L 0 formulation achieves the computation of sparse material distribution in one step, which is beneficial for additive manufacturing with multi-material printers. Third, we utilize the adjoint method to derive formulae for efficiently computing the gradient of the objective functions, making it possible to quickly solve the optimization problem in the full-dimensional space of materials, which was previously infeasible. The experiments demonstrate the robustness and efficiency of our approach.

A model free adaptive iterative learning control based fault-tolerant control (MFAILC-FTC) scheme for subway train speed tracking with speed sensor fault and over-speed protection is proposed. Firstly, the train dynamics is transformed into a compact form dynamic linearization (CFDL) data model by applying the concept of pseudo-partial derivative (PPD). If speed sensor fault occurs, the fault function is approximated by the trained RBFNNs under normal condition and the output data of the train system with fault, which serves as a compensation for the proposed MFAILC-FTC scheme. Then, over-speed protection mechanism is developed to ensure that the train operates within safe speed range. Furthermore, the constraint on traction/braking force is also taken into account. Through rigorous mathematical analysis, it is proved that the proposed MFAILC-FTC method with over-speed protection mechanism can ensure the train speed tracking error converges along the iteration axis, which implies the train operates safely and reliably. Finally, the simulation results further demonstrate the effectiveness of the proposed algorithm. Note to Practitioners—Subway train as a practical engineering system with short distance between two stations, starts and stops frequently, has the outstanding repetitive operation pattern, and it is unavoidable subject to speed sensor fault, aerodynamic issues, constraint on output speed and traction/braking force. Nevertheless, few works have considered these factors simultaneously, and a lot of data contain valuable operation information are generated during the train operation, this motives the work of this note. On account of the repetitive operation features of subway trains, the control schemes of speed trajectory tracking are handled under MFAILC framework, which is a pure data-driven model free control methodology. By constructing the RBFNNs-based fault function estimation mechanism, a robust compensation term is designed in the fault-tolerant controller. Taking the safe operation of subway trains into account, an over-speed protection term with trigger mechanism is added to the fault-tolerant controller. To further enhance the application, the constraint on traction/braking force is addressed as well. Without requirement of the train dynamics model, the theoretical analyses and simulation results have confirmed the effectiveness and the feasibility of the proposed data-driven control approach. In the future work, we will focus on verifying the proposed control strategy and addressing some other practical problems, for instance, the energy-efficiency and exogenous disturbances during the train operation.

A mechanism is an assembly of moving parts interconnected by joints to transfer an input motion to a desired output motion. Traditionally, to generate a complex motion, mechanisms are modeled by selecting and combining a number of mechanical parts with simple shapes such as links, gears, and cams. Combining multiple mechanical parts results in a mechanism with an intricate topology, which not only complicates assembly and maintenance but also deteriorates the functionality of generating motions due to accumulation of manufacturing imprecisions. To get rid of these limitations, we study mechanisms with a single pair of moving parts for generating complex motions. We model the pair of moving parts as a pair of conjugate surfaces with multiple conjugation points, forming a multi-point conjugation mechanism. To study this new mechanism, we establish a connection between conjugate surface pairs and form-closure grasps to formulate a dynamic form closure condition under which one conjugate surface is able to continuously transfer the motion to the other conjugate surface by utilizing multiple conjugation points. Guided by the condition, we propose an optimization-based approach to model the geometry of a multi-point conjugation mechanism for exactly generating a user-specified motion, in 1-, 2-, or 3-DOF motion space. The core of our approach is to model multiple conjugate curve pairs that satisfy various requirements in multi-point conjugation, dynamic form closure, and surface fabricability. We demonstrate the effectiveness of our approach by modeling different classes of multi-point conjugation mechanisms to generate various motions, evaluating the mechanisms' kinematic performance with 3D printed prototypes, and presenting three applications of these mechanisms.

With the development of 3D digital geometry technology, 3D triangular meshes are becoming more useful and valuable in industrial manufacturing and digital entertainment. A high quality triangular mesh can be used to represent a real world object with geometric and physical characteristics. While anisotropic meshes have advantages of representing shapes with sharp features (such as trimmed surfaces) more efficiently and accurately, isotropic meshes allow more numerically stable computations. When there is no anisotropic mesh requirement, isotropic triangles are always a good choice. In this paper, we propose a remeshing method to convert an input mesh into an adaptively isotropic one based on a curvature smoothed field (CSF). With the help of the CSF, adaptively isotropic remeshing can retain the curvature sensitivity, which enables more geometric features to be kept, and avoid the occurrence of obtuse triangles in the remeshed model as much as possible. The remeshed triangles with locally isotropic property benefit various geometric processes such as neighbor-based feature extraction and analysis. The experimental results show that our method achieves better balance between geometric feature preservation and mesh quality improvement compared to peers. We provide the implementation codes of our resampling method at github.com/vvvwo/Adaptively-Isotropic-Remeshing.

Abstract Curve reconstruction is a fundamental task in many visual computing applications. In this paper, a data-driven approach for curve reconstruction is proposed. We present an inception layered deep neural network structure, capable of learning simultaneously the number of control points and their positions in order to reconstruct the curve. To train the network, a large set of general synthetic data is generated. The reconstructed uniform B-spline closely approximates any arbitrary input curve, with or without intersections. Because the network predicts the number of control points required for the B-spline reconstruction, redundancy is reduced in the curve representation. We demonstrate our approach on various examples.

In recent years, neural implicit surface reconstruction has emerged as a popular paradigm for multi-view 3D reconstruction. Unlike traditional multi-view stereo approaches, the neural implicit surface-based methods leverage neural networks to represent 3D scenes as signed distance functions (SDFs). However, they tend to disregard the reconstruction of individual objects within the scene, which limits their performance and practical applications. To address this issue, previous work ObjectSDF introduced a nice framework of object-composition neural implicit surfaces, which utilizes 2D instance masks to supervise individual object SDFs. In this paper, we propose a new framework called ObjectSDF++ to overcome the limitations of ObjectSDF. First, in contrast to ObjectSDF whose performance is primarily restricted by its converted semantic field, the core component of our model is an occlusion-aware object opacity rendering formulation that directly volume-renders object opacity to be supervised with instance masks. Second, we design a novel regularization term for object distinction, which can effectively mitigate the issue that ObjectSDF may result in unexpected reconstruction in invisible regions due to the lack of constraint to prevent collisions. Our extensive experiments demonstrate that our novel framework not only produces superior object reconstruction results but also significantly improves the quality of scene reconstruction. Code and more resources can be found in https://qianyiwu.github.io/objectsdf++.

Many real-world problems, such as airfoil design, involve optimizing a black-box expensive objective function over complex-structured input space (e.g., discrete space or non-Euclidean space). By mapping the complex-structured input space into a latent space of dozens of variables, a two-stage procedure labeled as generative model-based optimization (GMO), in this article, shows promise in solving such problems. However, the latent dimension of GMO is hard to determine, which may trigger the conflicting issue between desirable solution accuracy and convergence rate. To address the above issue, we propose a multiform GMO approach, namely, generative multiform optimization (GMFoO), which conducts optimization over multiple latent spaces simultaneously to complement each other. More specifically, we devise a generative model which promotes a positive correlation between latent spaces to facilitate effective knowledge transfer in GMFoO. And furthermore, by using Bayesian optimization (BO) as the optimizer, we propose two strategies to exchange information between these latent spaces continuously. Experimental results are presented on airfoil and corbel design problems and an area maximization problem as well to demonstrate that our proposed GMFoO converges to better designs on a limited computational budget.

An extended state observer based model-free adaptive iterative learning energy-efficient control (ESO-based MFAILEEC) scheme for subway train speed tracking with external disturbances and over-speed protection under the constraint on traction/braking force is proposed. Firstly, the continuous-time train motion dynamics is formulated into a discrete-time data model with consideration of external disturbances by applying the iterative dynamic linearization. Meanwhile, the external disturbances and the unknown nonlinear uncertainties of the train are transformed into a new state, which is estimated by an ESO designed in the iteration domain. Then, the ESO-based energy-efficient controller with learning ability is designed, which enables the train to achieve the purpose of energy efficiency by reducing the input force. Further, over-speed protection with trigger mechanism is developed to ensure the train operates within safe speed range. All the control strategies are designed under the constraint on traction/braking force by considering the practical limitation of the train system. No model information is involved in the whole design processes and it is a pure data-driven iterative learning approach. Rigorous mathematical analysis proves the feasibility and the robustness of the proposed method, which can guarantee the train operates safely and reliably. Finally, the simulation results further demonstrate the effectiveness of the proposed algorithm.

A data-driven spatial adaptive terminal iterative learning predictive control (SATILPC) scheme with actuator saturation is proposed for automatic stop control of the subway train. Considering the outstanding repetitive operation pattern and the unavailable accurate model of a subway train, the unknown train dynamics is firstly transformed into a nonlinear discrete form in spatial domain via the spatial differential operator. Since the train automatic stop control (TASC) only concentrates on the tracking performances of the terminal position and terminal speed, an iterative dynamic linearization approach considering the terminal operation point is devised to formulate the relationship of the train input and output (I/O) into a linear affine form. Then, a terminal iterative learning prediction mechanism is introduced to reconstruct the developed train data model to forecast the future train behaviors through rolling optimization process. As a result, by removing the constraints on the unimportant points, the optimal control input (braking force) can be obtained by minimizing the objective function through terminal I/O data. Further, actuator saturation is considered to address the passenger comfort and the reliable operation of the train. The proposed SATILPC approach is a pure data-driven iterative learning control scheme and no model information is involved in the whole design processes. By utilizing a newly space-based contraction mapping method, the convergence of the proposed approach is strictly proved. Finally, the simulation results further demonstrate the feasibility of the proposed algorithm.

Estimating personality traits of a person through visual or multimodal signals has attracted increasing attention in cognitive multimodal interfaces and human factors in XR. Existing methods place a great emphasis on individual’s facial features and use these features to predict the apparent personality, but overlook the importance of environment and real personality. In this paper, we propose a deep learning approach to predict both real and apparent personality based on pure visual information in dyadic human-human interaction scenarios. We use the nonverbal information of both the target person and the interlocutor to learn their body and facial representations through a multi-branch ResNet-Attention network, and output real and apparent personality prediction in the form of five-dimensional personality traits (openness, conscientiousness, extraversion, agreeableness, and neuroticism). We conduct experiments to evaluate the proposed method. The experimental results show that the proposed method achieves good performance for both real and appearance personality prediction.

While using virtual backgrounds has recently become a very popular feature in videoconferencing, there often exists a jarring mismatch between the lighting of the user and the illumination condition of the virtual background. Existing portrait relighting methods can alleviate the problem, but do not have the capacity to deal with difficult shadow effects. In this paper, we present a new shadow-aware portrait relighting system that can relight an input portrait to be consistent with a given desired background image with shadow effects. Our system consists of four major components: portrait neutralization, illumination estimation, shadow generation and hierarchical neural rendering, which are all based on deep neural networks, and the whole system is end-to-end trainable. In addition, we created a large-scale photorealistic synthetic dataset with shadow, illumination and depth annotations for training, which allows our model to generalize well to real images. The extensive experiments demonstrate that our shadow-aware relight system outperforms the state-of-the-art portrait relighting solutions in terms of producing more lighting-consistent relighted images with shadow effects.

We present a simple yet effective method for constructing 3D self-supporting surfaces with planar quadrilateral (PQ) elements. Starting with a triangular discretization of a self-supporting surface, we first compute the principal curvatures and directions of each triangular face using a new discrete differential geometry approach, yielding more accurate results than existing methods. Then, we smooth the principal direction field to reduce the number of singularities. Next, we partition all faces into two groups in terms of principal curvature difference. For each face with small curvature difference, we compute a stretch matrix that turns the principal directions into a pair of conjugate directions. For the remaining triangular faces, we simply keep their smoothed principal directions. Finally, applying a mixed-integer programming solver to the mixed principal and conjugate direction field, we obtain a planar quadrilateral mesh. Experimental results show that our method is computationally efficient and can yield high-quality PQ meshes that well approximate the geometry of the input surfaces and maintain their self-supporting properties.

Abstract The use of social robots in healthcare systems or nursing homes to assist the elderly and their caregivers will be becoming common, where robots' understanding of engagement of the elderly is important. Traditional engagement estimation (EE) often requires expert involvement in a controlled dyadic interaction environment. In this article, we propose a supervised machine learning method to estimate the engagement state of the elderly in a multiparty human–robot interaction (HRI) scenario from the real‐world video recording as input. The method is built upon the basic concept of engagement in geriatric psychiatry and HRI video representations. It adapts pretrained models to extract behavior, affective, and visual signals to form the multi‐modal features. These features are then fed into a neural network made of a self‐attention mechanism and average pooling for individual learning, a graph attention network for group learning and a fully connected layer to estimate the engagement. We tested the proposed method using 43 wild multiparty elderly robot interaction (ERI) videos. The experimental results show that our method is capable of detecting the key participants and estimating the engagement state of the elderly effectively. Also our study demonstrates the signals from side‐participants in the main interaction group considerably contribute to the EE of the elderly in the multiparty ERI.

Facial expression retargeting from humans to virtual characters is a useful technique in computer graphics and animation. Traditional methods use markers or blendshapes to construct a mapping between the human and avatar faces. However, these approaches require a tedious 3D modeling process, and the performance relies on the modelers' experience. In this article, we propose a brand-new solution to this cross-domain expression transfer problem via nonlinear expression embedding and expression domain translation. We first build low-dimensional latent spaces for the human and avatar facial expressions with variational autoencoder. Then we construct correspondences between the two latent spaces guided by geometric and perceptual constraints. Specifically, we design geometric correspondences to reflect geometric matching and utilize a triplet data structure to express users' perceptual preference of avatar expressions. A user-friendly method is proposed to automatically generate triplets for a system allowing users to easily and efficiently annotate the correspondences. Using both geometric and perceptual correspondences, we trained a network for expression domain translation from human to avatar. Extensive experimental results and user studies demonstrate that even nonprofessional users can apply our method to generate high-quality facial expression retargeting results with less time and effort.

Neural implicit reconstruction is emerging as a promising approach to constructing 3D geometry from point clouds due to its ability to model geometry with complicated topology and unrestricted resolution. Current methods in this category usually deliver smooth and good quality results, but suffer from defective details and generalization issues. The major reason is that these methods use either a global code or interpolated feature on 3D grids of limited resolution to estimate implicit surface, therefore may cause distortion in feature discretization. This paper presents a neighborhood-aware neural implicit reconstruction framework that consists of an encoder network, a feature aggregation module, and a decoder network to learn implicit surface. The method can easily incorporate an off-the-shelf 3D point-based or volume-based neural network as an encoder. At the heart of our framework is the aggregation module that fuses the learnt contextual features on neighbor inputs so that the method can directly exploit local features of neighboring inputs for geometry detail recovery as well as cross-domain generalization. Experimental results demonstrate that our method significantly outperforms the state-of-the-art methods (about 4.0 points IoU improvements in ShapeNet dataset and 9.0 points IoU improvements in DFAUST dataset). Furthermore, our method preserves finer shape details and can be successfully transferred to a novel category without fine-tuning.

Generating an interpretable and compact representation of 3D shapes from point clouds is an important and challenging problem. This paper presents CSG-Stump Net, an unsupervised end-to-end network for learning shapes from point clouds and discovering the underlying constituent modeling primitives and operations as well. At the core is a three-level structure called CSG-Stump, consisting of a complement layer at the bottom, an intersection layer in the middle, and a union layer at the top. CSG-Stump is proven to be equivalent to CSG in terms of representation, therefore inheriting the interpretable, compact and editable nature of CSG while freeing from CSG’s complex tree structures. Particularly, the CSG-Stump has a simple and regular structure, allowing neural networks to give outputs of a constant dimensionality, which makes itself deep-learning friendly. Due to these characteristics of CSG-Stump, CSG-Stump Net achieves superior results compared to previous CSG-based methods and generates much more appealing shapes, as confirmed by extensive experiments.

As the applications of intelligent agents (IAs) are gradually increasing in daily life, they are expected to have reasonable social intelligence to interact with people by appropriately interpreting human behavior and intention. This paper presents a method to estimate whether people have willingness to join in a conversation, which helps to endow IAs with the capability of detecting potential participants. The method is built on the CNN-LSTM network, which takes image features and social signals as input, making use of general information conveyed in images, semantic social cues proven by social psychology studies, and temporal information in the sequence of inputs. The network is designed to have a multi-branch structure with the flexibility of accommodating different types of inputs. We also discuss the signal transition in multiparty human-robot interaction scenarios. The method is evaluated on three datasets with social signals and/or images as inputs. The results show that the proposed method can infer human engagement intention well.

Abstract We present a neural‐based model for relighting a half‐body portrait image by simply referring to another portrait image with the desired lighting condition. Rather than following classical inverse rendering methodology that involves estimating normals, albedo and environment maps, we implicitly encode the subject and lighting in a latent space, and use these latent codes to generate relighted images by neural rendering. A key technical innovation is the use of a novel overcomplete lighting representation, which facilitates lighting interpolation in the latent space, as well as helping regularize the self‐organization of the lighting latent space during training. In addition, we propose a novel multiplicative neural render that more effectively combines the subject and lighting latent codes for rendering. We also created a large‐scale photorealistic rendered relighting dataset for training, which allows our model to generalize well to real images. Extensive experiments demonstrate that our system not only outperforms existing methods for referral‐based portrait relighting, but also has the capability generate sequences of relighted images via lighting rotations.

In recent years, Massively Multiplayer Online Games (MMOGs) are becoming popular, partially due to their sophisticated graphics and broad virtual world, and cloud gaming is demanded more than ever especially when entertaining with light and portable devices. This article considers the problem of server allocation for running MMOG on cloud, aiming to reduce the cost on cloud gaming service and meanwhile enhance the quality of service. The problem is formulated into minimizing an objective function involving the cost of server rental, the cost of data transfer and the network latency during the gaming time. A genetic algorithm is developed to solve the minimization problem for processing simultaneous server allocation for the players who log into the system at the same time while many existing players are playing the same game. Extensive experiments based on the player behavior in “World of Warcraft” are conducted to evaluate the proposed method and compare with the state-of-the-art as well. The experimental results show that the method gives a lower cost and a shorter network latency in most of the time.

3D instance segmentation plays a predominant role in environment perception of robotics and augmented reality. Many deep learning based methods have been presented recently for this task. These methods rely on either a detection branch to propose objects or a grouping step to assemble same-instance points. However, detection based methods do not ensure a consistent instance label for each point, while the grouping step requires parameter-tuning and is computationally expensive. In this paper, we introduce a novel framework to enable end-to-end instance segmentation without detection and a separate step of grouping. The core idea is to convert instance segmentation to a candidate assignment problem. At first, a set of instance candidates is sampled. Then we propose an assignment module for candidate assignment and a suppression module to eliminate redundant candidates. A mapping between instance labels and instance candidates is further sought to construct an instance grouping loss for the network training. Experimental results demonstrate that our method is more effective and efficient than previous approaches.

Existing methods of unstructured point cloud compression usually exploit the spatial sparseness of point clouds using hierarchical tree data structures for spatial encoding. However, such methods can be inefficient when very deep octrees are applied to sparse point cloud data to maintain low level of geometric error during compression. This paper proposes a novel octree structure called truncated octree that improves the compression ratio by representing the deep octree with a set of shallow sub-octrees which can save storage without losing the original structure. We also propose a variable length addressing scheme, to adaptively choose the length of an octree's node address based on the truncation level-shorter (resp. longer) address when octree is truncated near the leaf (resp. root) which leads to further compression. The method is able to achieve 40% to 90% compression ratio on our tested models for point clouds of different spatial distributions. For extremely sparse point clouds, the method achieves approximately 7 times higher compression ratio than previous methods. Moreover, the method is designed to run in parallel for octree construction, encoding and decoding.

Corbel is a common category of decorative architectural geometry that has clear structure and aesthetic design. This paper presents a method for automatically generating a group of new corbel models from one selected by the user in the dataset. The method consists of offline learning and online generation. The offline learning trains two VAE models (2D CurveVAE and 3D VoxelVAE) for learning the feature representation of corbel parts. The online generation includes a generation algorithm by evolution that evolves to product new generation of models by crossing over and mutating features, and a feature-driven deformation that synthesizes 3D mesh representation of corbel models. By integrating these technical components, we develop a creative corbel modeling tool capable of generating new corbel models that are both “more of the same” and “surprising”, which is demonstrated by experiments.

Multi-robot systems are widely used in environmental exploration and modeling, especially in hazardous environments. However, different types of robots are limited by different mobility, battery life, sensor type, etc. Heterogeneous robot systems are able to utilize various types of robots and provide solutions where robots are able to compensate each other with their different capabilities. In this paper, we consider the problem of sampling and modeling environmental characteristics with a heterogeneous team of robots. To utilize heterogeneity of the system while remaining computationally tractable, we propose an environmental partitioning approach that leverages various robot capabilities by forming a uniformly defined heterogeneity cost space. We combine with the mixture of Gaussian Processes model-learning framework to adaptively sample and model the environment in an efficient and scalable manner. We demonstrate our algorithm in field experiments with ground and aerial vehicles.

The problem of deforming an artist-drawn caricature according to a given normal face expression is of interest in applications such as social media, animation and entertainment. This paper presents a solution to the problem, with an emphasis on enhancing the ability to create desired expressions and meanwhile preserve the identity exaggeration style of the caricature, which imposes challenges due to the complicated nature of caricatures. The key of our solution is a novel method to model caricature expression, which extends traditional 3DMM representation to caricature domain. The method consists of shape modelling and texture generation for caricatures. Geometric optimization is developed to create identity-preserving blendshapes for reconstructing accurate and stable geometric shape, and a conditional generative adversarial network (cGAN) is designed for generating dynamic textures under target expressions. The combination of both shape and texture components makes the non-trivial expressions of a caricature be effectively defined by the extension of the popular 3DMM representation and a caricature can thus be flexibly deformed into arbitrary expressions with good results visually in both shape and color spaces. The experiments demonstrate the effectiveness of the proposed method.

Human bodies exhibit various shapes for different identities or poses, but the body shape has certain similarities in structure and thus can be embedded in a low-dimensional space. This article presents an autoencoder-like network architecture to learn disentangled shape and pose embedding specifically for the 3D human body. This is inspired by recent progress of deformation-based latent representation learning. To improve the reconstruction accuracy, we propose a hierarchical reconstruction pipeline for the disentangling process and construct a large dataset of human body models with consistent connectivity for the learning of the neural network. Our learned embedding can not only achieve superior reconstruction accuracy but also provide great flexibility in 3D human body generation via interpolation, bilinear interpolation, and latent space sampling. The results from extensive experiments demonstrate the powerfulness of our learned 3D human body embedding in various applications.

This work addresses the problem of 3D human shape reconstruction from point clouds. Considering that human shapes are of high dimensions and with large articulations, we adopt the state-of-the-art parametric human body model, SMPL, to reduce the dimension of learning space and generate smooth and valid reconstruction. However, SMPL parameters, especially pose parameters, are not easy to learn because of ambiguity and locality of the pose representation. Thus, we propose to incorporate skeleton awareness into the deep learning based regression of SMPL parameters for 3D human shape reconstruction. Our basic idea is to use the state-of-the-art technique PointNet++ to extract point features, and then map point features to skeleton joint features and finally to SMPL parameters for the reconstruction from point clouds. Particularly, we develop an end-to-end framework, where we propose a graph aggregation module to augment PointNet++ by extracting better point features, an attention module to better map unordered point features into ordered skeleton joint features, and a skeleton graph module to extract better joint features for SMPL parameter regression. The entire framework network is first trained in an end-to-end manner on synthesized dataset, and then online fine-tuned on unseen dataset with unsupervised loss to bridges gaps between training and testing. The experiments on multiple datasets show that our method is on par with the state-of-the-art solution.

While learning based depth estimation from images/videos has achieved substantial progress, there still exist intrinsic limitations. Supervised methods are limited by a small amount of ground truth or labeled data and unsupervised methods for monocular videos are mostly based on the static scene assumption, not performing well on real world scenarios with the presence of dynamic objects. In this paper, we propose a new learning based method consisting of DepthNet, PoseNet and Region Deformer Networks (RDN) to estimate depth from unconstrained monocular videos without ground truth supervision. The core contribution lies in RDN for proper handling of rigid and non-rigid motions of various objects such as rigidly moving cars and deformable humans. In particular, a deformation based motion representation is proposed to model individual object motion on 2D images. This representation enables our method to be applicable to diverse unconstrained monocular videos. Our method can not only achieve the state-of-the-art results on standard benchmarks KITTI and Cityscapes, but also show promising results on a crowded pedestrian tracking dataset, which demonstrates the effectiveness of the deformation based motion representation. Code and trained models are available at https://github.com/haofeixu/rdn4depth.

Abstract This paper presents subdivision‐based representations for both lighting and geometry in shape‐from‐shading. A very recent shading‐based method introduced a per‐vertex overall illumination model for surface reconstruction, which has advantage of conveniently handling complicated lighting condition and avoiding explicit estimation of visibility and varied albedo. However, due to its discrete nature, the per‐vertex overall illumination requires a large amount of memory and lacks intrinsic coherence. To overcome these problems, in this paper we propose to use classic subdivision to define the basic smooth lighting function and surface, and introduce additional independent variables into the subdivision to adaptively model sharp changes of illumination and geometry. Compared to previous works, the new model not only preserves the merits of the per‐vertex illumination model, but also greatly reduces the number of variables required in surface recovery and intrinsically regularizes the illumination vectors and the surface. These features make the new model very suitable for multi‐view stereo surface reconstruction under general, unknown illumination condition. Particularly, a variational surface reconstruction method built upon the subdivision representations for lighting and geometry is developed. The experiments on both synthetic and real‐world data sets have demonstrated that the proposed method can achieve memory efficiency and improve surface detail recovery.

Abstract Light field (LF) reconstruction is a fundamental technique in light field imaging and has applications in both software and hardware aspects. This paper presents an unsupervised learning method for LF‐oriented view synthesis, which provides a simple solution for generating quality light fields from a sparse set of views. The method is built on disparity estimation and image warping. Specifically, we first use per‐view disparity as a geometry proxy to warp input views to novel views. Then we compensate the occlusion with a network by a forward‐backward warping process. Cycle‐consistency between different views are explored to enable unsupervised learning and accurate synthesis. The method overcomes the drawbacks of fully supervised learning methods that require large labeled training dataset and epipolar plane image based interpolation methods that do not make full use of geometry consistency in LFs. Experimental results demonstrate that the proposed method can generate high quality views for LF, which outperforms unsupervised approaches and is comparable to fully‐supervised approaches.

With the powerfulness of convolution neural networks (CNN), CNN based face reconstruction has recently shown promising performance in reconstructing detailed face shape from 2D face images. The success of CNN-based methods relies on a large number of labeled data. The state-of-the-art synthesizes such data using a coarse morphable face model, which however has difficulty to generate detailed photo-realistic images of faces (with wrinkles). This paper presents a novel face data generation method. Specifically, we render a large number of photo-realistic face images with different attributes based on inverse rendering. Furthermore, we construct a fine-detailed face image dataset by transferring different scales of details from one image to another. We also construct a large number of video-type adjacent frame pairs by simulating the distribution of real video data.11.All these coarse-scale and fine-scale photo-realistic face image datasets can be downloaded from https://github.com/Juyong/3DFace. With these nicely constructed datasets, we propose a coarse-to-fine learning framework consisting of three convolutional networks. The networks are trained for real-time detailed 3D face reconstruction from monocular video as well as from a single image. Extensive experimental results demonstrate that our framework can produce high-quality reconstruction but with much less computation time compared to the state-of-the-art. Moreover, our method is robust to pose, expression and lighting due to the diversity of data.

This paper discusses automatic path planning for tower crane lifting in highly complex environments to be digitized using point cloud representation. A mathematical optimization technique is developed to identify the lifting path with GPU accelerated massively parallel genetic algorithm. A continuous collision detection method is designed for real time application of collision avoidance during the crane lifting process.

This paper presents a novel GPU-based parallel algorithm to simplify triangular meshes. Existing GPU based methods usually produce simplified meshes with lower quality. This is generally because they put more emphasis on parallelism than mesh quality. After a thorough analysis of the existing methods, we propose a GPU based multiple-choice mechanism, which combines multiple-choice decimation and GPU-based parallel simplification, to balance the mesh quality and computational speed. As a result, our algorithm improves the quality of the simplified meshes and also achieves high speed-up provided by GPU. The experiments and the comparison with the prior art confirm the performance of the proposed algorithm.

When building robotic hands, researchers are always face with two main issues of how to make robotic hands look human-like and how to make robotic hands function like real hands. Most existing solutions solve these issues by manually modelling the robotic hand [10-18]. However, the design processes are long, and it is difficult to duplicate the geometry shape of a human hand. To solve these two issues, this paper presents a simple and effective method that combines 3D printing and digitization techniques to create a 3D printable cable-driven robotic hand from scanning a physical hand. The method involves segmenting the 3D scanned hand model, adding joints, and converting it into a 3D printable model. Comparing to other robotic solutions, our solution retains more than 90% geometry information of a human hand1, which is attained from 3D scanning. Our modelling progress takes around 15 minutes that include 10 minutes of 3D scanning and five minutes for changing the scanned model to an articulated model by running our algorithm. Compared to other articulated modelling solutions [19, 20], our solution is compatible with an actuation system which provides our robotic hand with the ability to mimic different gestures. We have also developed a way of representing hand skeletons based on the hand anthropometric. As a proof of concept, we demonstrate our robotic hand's performance in the grasping experiments.

Caricature is an art form that expresses subjects in abstract, simple and exaggerated views. While many caricatures are 2D images, this paper presents an algorithm for creating expressive 3D caricatures from 2D caricature images with minimum user interaction. The key idea of our approach is to introduce an intrinsic deformation representation that has the capability of extrapolation, enabling us to create a deformation space from standard face datasets, which maintains face constraints and meanwhile is sufficiently large for producing exaggerated face models. Built upon the proposed deformation representation, an optimization model is formulated to find the 3D caricature that captures the style of the 2D caricature image automatically. The experiments show that our approach has better capability in expressing caricatures than those fitting approaches directly using classical parametric face models such as 3DMM and FaceWareHouse. Moreover, our approach is based on standard face datasets and avoids constructing complicated 3D caricature training sets, which provides great flexibility in real applications.

Sketch-based modeling uses freeform strokes as basic modeling metaphor and provides an intuitive way for shape modeling, for instance, for cyberworlds. This paper presents a new method to enhance sketch-based modeling. The core idea of the method is to enhance the sketching process by allowing the user to iteratively sketch to progressively create initial shapes that interpolate the sketched strokes. This process considers all the sketches and the up-to-date constructed 3D shape, which enables the user to be aware of the shape of the sketched model. The key underlying technique that supports this process is a novel surface construction algorithm, which generates 3D triangular mesh models with gradual shape changes during iterative sketching. Experiments demonstrate that the presented method can allow users to intuitively and flexibly create and edit 3D models even with complex topology, which is usually difficult in existing sketch-based modeling systems.

Negative symptoms of schizophrenia are often associated with the blunting of emotional affect which creates a serious impediment in the daily functioning of the patients. Affective prosody is almost always adversely impacted in such cases, and is known to exhibit itself through the low-level acoustic signals of prosody. To automate and simplify the process of assessment of severity of emotion related symptoms of schizophrenia, we utilized these low-level acoustic signals to predict the expert subjective ratings assigned by a trained psychologist during an interview with the patient. Specifically, we extract acoustic features related to emotion using the openSMILE toolkit from the audio recordings of the interviews. We analysed the interviews of 78 paid participants (52 patients and 26 healthy controls) in this study. The subjective ratings could be accurately predicted from the objective openSMILE acoustic signals with an accuracy of 61-85% using machine-learning algorithms with leave-one-out cross-validation technique. Furthermore, these objective measures can be reliably utilized to distinguish between the patient and healthy groups, as supervised learning methods can classify the two groups with 79-86% accuracy.

Teleconference or telepresence based on virtual reality (VR) head-mount display (HMD) device is a very interesting and promising application since HMD can provide immersive feelings for users. However, in order to facilitate face-to-face communications for HMD users, real-time 3D facial performance capture of a person wearing HMD is needed, which is a very challenging task due to the large occlusion caused by HMD. The existing limited solutions are very complex either in setting or in approach as well as lacking the performance capture of 3D eye gaze movement. In this paper, we propose a convolutional neural network (CNN) based solution for real-time 3D face-eye performance capture of HMD users without complex modification to devices. To address the issue of lacking training data, we generate massive pairs of HMD face-label dataset by data synthesis as well as collecting VR-IR eye dataset from multiple subjects. Then, we train a dense-fitting network for facial region and an eye gaze network to regress 3D eye model parameters. Extensive experimental results demonstrate that our system can efficiently and effectively produce in real time a vivid personalized 3D avatar with the correct identity, pose, expression and eye motion corresponding to the HMD user.

This paper presents a spatial-varying illumination model for shading-based depth refinement that based on a smooth Spherical Harmonics (SH) lighting field. The proposed lighting model is able to recover shading under challenging unknown lighting conditions, thus improving the quality of recovered surface detail. To avoid over-parameterization, local lighting coefficients are treated as a vector-valued function which is represented by subdivided surfaces using Catmull-Clark subdivision. We solve our lighting model utilizing a highly parallelized scheme that recovers lighting in a few milliseconds. A real-time shading-based depth recovery system is implemented with the integration of our proposed lighting model. We conduct quantitative and qualitative evaluations on both synthetic and real world datasets under challenging illumination. The experimental results show our method outperforms the state-of-the-art real-time shading-based depth refinement system.

Reconstructing the shape of a 3D object from multi-view images under unknown, general illumination is a fundamental problem in computer vision. High quality reconstruction is usually challenging especially when fine detail is needed and the albedo of the object is non-uniform. This paper introduces vertex overall illumination vectors to model the illumination effect and presents a total variation (TV) based approach for recovering surface details using shading and multi-view stereo (MVS). Behind the approach are the two important observations: (1) the illumination over the surface of an object often appears to be piecewise smooth and (2) the recovery of surface orientation is not sufficient for reconstructing the surface, which was often overlooked previously. Thus we propose to use TV to regularize the overall illumination vectors and use visual hull to constrain partial vertices. The reconstruction is formulated as a constrained TV-minimization problem that simultaneously treats the shape and illumination vectors as unknowns. An augmented Lagrangian method is proposed to quickly solve the TV-minimization problem. As a result, our approach is robust, stable and is able to efficiently recover high-quality surface details even when starting with a coarse model obtained using MVS. These advantages are demonstrated by extensive experiments on the state-of-the-art MVS database, which includes challenging objects with varying albedo.

Cooperative dual-crane lifting is an important but challenging process involved in heavy and critical lifting tasks. This paper considers the path planning for the cooperative dual-crane lifting. It aims to automatically generate optimal dual-crane lifting paths under multiple constraints, i.e., collision avoidance, coordination between the two cranes, and balance of the lifting target. Previous works often used oversimplified models for the dual-crane lifting system, the lifting environment, and the motion of the lifting target. They were thus limited to simple lifting cases and might even lead to unsafe paths in some cases. We develop a novel path planner for dual-crane lifting that can quickly produce optimized paths in complex 3-D environments. The planner has fully considered the kinematic structure of the lifting system. Therefore, it is able to robustly handle the nonlinear movement of the suspended target during lifting. The effectiveness and efficiency of the planner are enabled by three novel aspects: 1) a comprehensive and computationally efficient mathematical modeling of the lifting system; 2) a new multiobjective parallel genetic algorithm designed to solve the path planning problem; and 3) a new efficient approach to perform continuous collision detection for the dual-crane lifting target. The planner has been tested in complex industrial environments. The results show that the planner can generate dual-crane lifting paths that are easy for conductions and optimized in terms of costs for complex environments. Comparisons with two previous methods demonstrate the advantages of the planner, including safer paths, higher success rates, and the ability to handle general lifting cases.

Shading is a tedious process for artists involved in 2D cartoon and manga production given the volume of contents that the artists have to prepare regularly over tight schedule. While we can automate shading production with the presence of geometry, it is impractical for artists to model the geometry for every single drawing. In this work, we aim to automate shading generation by analyzing the local shapes, connections, and spatial arrangement of wrinkle strokes in a clean line drawing. By this, artists can focus more on the design rather than the tedious manual editing work, and experiment with different shading effects under different conditions. To achieve this, we have made three key technical contributions. First, we model five perceptual cues by exploring relevant psychological principles to estimate the local depth profile around strokes. Second, we formulate stroke interpretation as a global optimization model that simultaneously balances different interpretations suggested by the perceptual cues and minimizes the interpretation discrepancy. Lastly, we develop a wrinkle-aware inflation method to generate a height field for the surface to support the shading region computation. In particular, we enable the generation of two commonly-used shading styles: 3D-like soft shading and manga-style flat shading.

We present a variational method for subdivision surface reconstruction from a noisy dense mesh. A new set of subdivision rules with continuous sharpness control is introduced into Loop subdivision for better modeling subdivision surface features such as semi-sharp creases, creases, and corners. The key idea is to assign a sharpness value to each edge of the control mesh to continuously control the surface features. Based on the new subdivision rules, a variational model with L1 norm is formulated to find the control mesh and the corresponding sharpness values of the subdivision surface that best fits the input mesh. An iterative solver based on the augmented Lagrangian method and particle swarm optimization is used to solve the resulting non-linear, non-differentiable optimization problem. Our experimental results show that our method can handle meshes well with sharp/semi-sharp features and noise.

An accurate and efficient clothoid approximation approach is presented in this paper using Bézier curves based on the minimization of curvature profile difference. Compared with existing methods, the proposed approach is able to guarantee higher order geometric continuity with smaller approximation error in terms of position, orientation, and curvature. The approximation scheme takes place in three stages. First, a subset of clothoids with specific winding angle constraints referred to as elementary clothoids is approximated using quintic Bézier curves. Then, a basic clothoid defined in the first quadrant is formulated, which is composed of a series of transformed elementary clothoids. An adaptive sampling stra-tegy is applied to ensure that the resulting Bézier segments are computed within a specified accuracy and all the required information can be obtained offline and stored in a lookup table. Finally, a general clothoid with arbitrary parameters can be conveniently approximated based on the lookup table through appropriate geometric transformations. A comparison with the recent circular interpolation and rational Bézier curve based approximation shows that the proposed approach is able to achieve equivalent or greater computational efficiency in most scenarios.

In the coming era of social companions, many researches have been pursuing natural dialog interactions and long-term relations between social companions and users. With respect to the quick decrease of user interests after the first few interactions, various emotion and memory models are developed and integrated with social companions for better user engagement. This paper reviews related works in the effort of combining memory and emotion with natural language dialog on social companions. We separate these works into three categories: (1) Affective system with dialog, (2) Task-driven memory with dialog, (3) Chat-driven memory with dialog. In addition, we discussed limitations and challenging issues to be solved. Finally, we also introduced our framework of social companions.

The state-of-the-art photometric stereo (PS) methods typically apply shading cues on each vertex and represent a vertex normal as a non-linear function of its neighboring vertices. Such vertex-based representation leads to huge computational cost and limits it from processing dense meshes. In this paper, we propose a PS based surface reconstruction using mesh face based representation. In particular, we propose to apply the shading cue on each mesh face instead of each vertex and optimize face normals instead of vertex normals. We develop a two-step approach to solve the surface recovery problem, where we first optimize face normals using shading cues and then update vertices by the optimized face normals. Experimental results show that, compared with the state-of-the-art, such a two-step approach is able to reduce the runtime significantly as well as handling much denser meshes.

Localizing, identifying, and extracting humans with consistent appearance jointly from a personal photo stream is an important problem and has wide applications. The strong variations in foreground and background and irregularly occurring foreground humans make this realistic problem challenging. Inspired by advancements in object detection, scene understanding, and image cosegmentation, we explore explicit constraints to label and segment human objects rather than other nonhuman objects and “stuff.” We refer to such a problem as multiple human identification and cosegmentation (MHIC). To identify specific human subjects, we propose an efficient human instance detector by combining an extended color line model with a poselet-based human detector. Moreover, to capture high-level human shape information, a novel soft shape cue is proposed. It is initialized by the human detector, then further enhanced through a generalized geodesic distance transform, and finally refined with a joint bilateral filter. We also propose to capture the rich feature context around each pixel by using an adaptive cross-region data structure, which gives a higher discriminative power than a single pixel-based estimation. The high-level object cues from the detector and the shape are then integrated with the low-level pixel cues and midlevel contour cues into a principled conditional random field (CRF) framework, which can be efficiently solved by using fast graph cut algorithms. We evaluate our method over a newly created NTU-MHIC human dataset, which contains 351 images with manually annotated groundtruth segmentation. Both visual and quantitative results demonstrate that our method achieves state-of-the-art performance for the MHIC task.

Home healthcare becomes more and more important with the increased aging population. The advent of various low-cost sensing devices makes it tempting to develop low-cost, non-intrusive systems to monitor the variations of human being. In this paper, we describe a Kinect-based gait analysis and visualization system as a case study in this direction. The system uses depth images and skeleton captured by the Kinect to generate a BVH file recording the motion information, extracts features of gait for detecting abnormal gait, and customizes the 3D body model for personalized motion visualization. Compared to previous work in this area, the proposed system has advantages since it integrates gait classification and visualization which may bring new possibilities in healthcare. The experiments show that our proposed system achieves accurate gait classification as well as flexible personalized 3D visualization.

This paper presents a variational algorithm for feature-preserved mesh denoising. At the heart of the algorithm is a novel variational model composed of three components: fidelity, regularization and fairness, which are specifically designed to have their intuitive roles. In particular, the fidelity is formulated as an L1 data term, which makes the regularization process be less dependent on the exact value of outliers and noise. The regularization is formulated as the total absolute edge-lengthed supplementary angle of the dihedral angle, making the model capable of reconstructing meshes with sharp features. In addition, an augmented Lagrange method is provided to efficiently solve the proposed variational model. Compared to the prior art, the new algorithm has crucial advantages in handling large scale noise, noise along random directions, and different kinds of noise, including random impulsive noise, even in the presence of sharp features. Both visual and quantitative evaluation demonstrates the superiority of the new algorithm.

Abstract Example‐based material allows simulating complex material behaviors in an art‐directed way. This paper presents a method for fast subspace integration for example‐based elastic material, which is suitable for real‐time simulation in computer graphics. At the core of the method is the formulation of a new potential using example‐based Green strain tensors. By using this potential, the deformation can be attracted towards the example‐based deformation feature space, the example weights can be explicitly obtained and the internal force can be decomposed into the conventional one and an additional one induced by the examples. The real‐time subspace integration is then developed with subspace integration costs independent of geometric complexity, and both the reduced conventional internal force and additional one being cubic polynomials in reduced coordinates. Experiments demonstrate that our method can achieve real‐time simulation while providing comparable quality with the prior art.

Abstract How to endow the virtual characters with personalized behavior patterns remains a challenging problem. Instead of heuristically designing behaviors for certain personalities, this paper bridges the gap between personalities and behaviors using a medium concept, mood, to make the behaviors of the characters consistent enough to convey their personalities, while flexible enough to make appropriate response in various situations. We propose a personality‐characterized mood dynamics model, in which the emotion weights are computed as a solution of a convex optimization problem that is constructed to make the overall mood approaches the personality after sufficient interactions. The convergence of the algorithm is demonstrated by numerical simulations. The implementation of the personality‐characterized mood dynamics model enables an emotion‐oriented virtual human, Sophie, to show personalized behaviors in the emotional interactions with users. Copyright © 2015 John Wiley & Sons, Ltd.

Considering that the existing depth recovery approaches have different limitations when applied to Kinect depth data, in this article, we propose to integrate their effective features including adaptive support region selection, reliable depth selection, and color guidance together under an optimization framework for Kinect depth recovery. In particular, we formulate our depth recovery as an energy minimization problem, which solves the depth hole filling and denoising simultaneously. The energy function consists of a fidelity term and a regularization term, which are designed according to the Kinect characteristics. Our framework inherits and improves the idea of guided filtering by incorporating structure information and prior knowledge of the Kinect noise model. Through analyzing the solution to the optimization framework, we also derive a local filtering version that provides an efficient and effective way of improving the existing filtering techniques. Quantitative evaluations on our developed synthesized dataset and experiments on real Kinect data show that the proposed method achieves superior performance in terms of recovery accuracy and visual quality.

Mapping texture onto 3D meshes with positional constraints is a popular technique that can effectively enhance the visual realism of geometric models. Such a process usually requires constructing a valid mesh embedding satisfying a set of positional constraints, which is known to be a challenging problem. This paper presents a novel algorithm for computing a foldover-free piecewise linear mapping with exact positional constraints. The algorithm begins with an unconstrained planar embedding, followed by iterative constrained mesh transformations. At the heart of the algorithm are radial basis function (RBF)-based warping and the longest edge bisection (LEB)-based refinement. A delicate integration of the RBF-based warping and the LEB-based refinement provides a provably-foldover-free, smooth constrained mesh warping, which can handle a large number of constraints and output a visually pleasing mapping result without extra smoothing optimization. The experiments demonstrate the effectiveness of the proposed algorithm.

Localizing, identifying and extracting human groups with consistent appearance jointly from a personal photo stream is an important problem and has wide applications. Inspired by recent advances in object detection, scene understanding and image cosegmentation, in this paper we explore explicit constraints to label and segment human objects rather than other non-human objects and “stuff”. We propose a novel soft human shape cue, which is initialized by color line poselet-based human part detection, further processed through a generalized geodesic distance transform, and refined finally with a joint bilateral filter. Such a high-level object cue is then integrated with other low-level unary and pairwise terms into a principled conditional random field framework, which can be efficiently solved by fast graph cut algorithms. We evaluate our algorithm over the FlickrMFC human dataset, and show that it achieves state-of-the-art performance for this challenging task.

Localizing, recognizing, and segmenting multiple foreground objects jointly from a general user's photo stream that records a specific event is an important task with many useful applications. As argued in recent Multiple Foreground Cosegmentation (MFC) work by Kim and Xing, this task is very challenging in that it contrasts substantially from the classical cosegmentation problem, and aims to parse a set of realistic event photos but each containing irregularly occurring multiple foregrounds with high appearance and scene configuration variations. Inspired by the impressive advance in scene understanding and object recognition, this paper casts the multiple foreground recognition and cosegmentation (MFRC) problem within a conditional random fields (CRFs) framework in a principled manner. We capitalize centrally on the key objective that MFRC is to segment out and annotate foreground objects or “things” rather than “stuff”. To this end, we exploit a few complementary objectness cues (e.g. contours, object detectors and layout) and propose novel and efficient methods to capture object-level information. Integrating object potentials as soft constraints (e.g. robust higher-order potentials defined over detected object regions) with low-level unary and pairwise terms holistically, we solve the MFRC task with a probabilistic CRF model. The inference for such a CRF model is performed efficiently with graph cut based move making algorithms. With a minimal amount of user annotations on just a few example photos, the proposed approach produces spatially coherent, boundary-aligned segmentation results with correct and consistent object labeling. Experiments on the FlickrMFC dataset justify that our method achieves state-of-the-art performance.

This paper presents a subdivision-based vector graphics for image representation and creation. The graphics representation is a subdivision surface defined by a triangular mesh augmented with color attribute at vertices and feature attribute at edges. Special cubic B-splines are proposed to describe curvilinear features of an image. New subdivision rules are then designed accordingly, which are applied to the mesh and the color attribute to define the spatial distribution and piecewise-smoothly varying colors of the image. A sharpness factor is introduced to control the color transition across the curvilinear edges. In addition, an automatic algorithm is developed to convert a raster image into such a vector graphics representation. The algorithm first detects the curvilinear features of the image, then constructs a triangulation based on the curvilinear edges and feature attributes, and finally iteratively optimizes the vertex color attributes and updates the triangulation. Compared with existing vector-based image representations, the proposed representation and algorithm have the following advantages in addition to the common merits (such as editability and scalability): 1) they allow flexible mesh topology and handle images or objects with complicated boundaries or features effectively; 2) they are able to faithfully reconstruct curvilinear features, especially in modeling subtle shading effects around feature curves; and 3) they offer a simple way for the user to create images in a freehand style. The effectiveness of the proposed method has been demonstrated in experiments.

Surface reconstruction from point cloud is of great practical importance in computer graphics. Existing methods often realize reconstruction via a few phases with respective goals, whose integration may not give an optimal solution. In this paper, to avoid the inherent limitations of multi-phase processing in the prior art, we propose a unified framework that treats geometry and connectivity construction as one joint optimization problem. The framework is based on dictionary learning in which the dictionary consists of the vertices of the reconstructed triangular mesh and the sparse coding matrix encodes the connectivity of the mesh. The dictionary learning is formulated as a constrained ℓ 2,q -optimization (0 < q < 1), aiming to find the vertex position and triangulation that minimize an energy function composed of point-to-mesh metric and regularization. Our formulation takes many factors into account within the same framework, including distance metric, noise/outlier resilience, sharp feature preservation, no need to estimate normal, etc., thus providing a global and robust algorithm that is able to efficiently recover a piecewise smooth surface from dense data points with imperfections. Extensive experiments using synthetic models, real world models, and publicly available benchmark show that our method outperforms the state-of-the-art in terms of accuracy, robustness to noise and outliers, geometric feature and detail preservation, and mesh connectivity.

The placement of knot vector and the determination of control points are two fundamental issues in B-spline surface reconstruction. This paper presents a variational approach to construct B-spline surfaces from a set of data points. The approach finds the optimal placement of knots and control points simultaneously while most previous methods determine the knots heuristically or in a separate step. Moreover, different from most previous methods using least squares metric, our approach adapts L_1-norm with total variation (TV) as regularization in the fitting procedure, which enables the approach to handle both Gaussian noise and outliers in the same manner and is able to automatically optimize the placement of knot vector to faithfully reconstruct the sharp features. A numerical solver based on the augmented Lagrangian method is also proposed in the paper to efficiently solve the TV-L_1 optimization. Experimental results demonstrate the effectiveness and efficiency of the proposed variational B-spline surface reconstruction.

Warping a tetrahedral mesh driven by boundary surface deformation is useful in many applications. Although some methods have been developed to transform the mesh to conform to the deformed boundary surface, it is still a challenging problem to construct an inversion free warped mesh maintaining a compatible topology. In this paper, these two problems are solved by a novel method that combines radial basis function (RBF)-based warping and adaptive mesh refinement. We iteratively transform the mesh using RBF-based warping with a safe step size to ensure that no element is inverted. The use of the RBF-based warping ensures a smooth warping and thus generates a high-quality warped volumetric mesh. To avoid too small step sizes, we refine the elements that are potentially inverted. The refinement is performed on the original and the warped meshes in the same way so as to maintain compatible topology between them. The results of our method can be used in many areas such as finite element simulation and shape interpolation. We demonstrate the effectiveness of our method with a set of examples.

Though quite a few image segmentation benchmark datasets have been constructed, there is no suitable benchmark for semantic image segmentation. In this paper, we construct a benchmark for such a purpose, where the ground-truths are generated by leveraging the existing fine granular ground-truths in Berkeley Segmentation Dataset (BSD) as well as using an interactive segmentation tool for new images. We also propose a percept-tree-based region merging strategy for dynamically adapting the ground-truth for evaluating test segmentation. Moreover, we propose a new evaluation metric that is easy to understand and compute, and does not require boundary matching. Experimental results show that, compared with BSD, the generated ground-truth dataset is more suitable for evaluating semantic image segmentation, and the conducted user study demonstrates that the proposed evaluation metric matches user ranking very well.

Given any image input by users, how to automatically cutout the object-of-interest is a challenging problem due to lack of information of the object-of-interest and the background. Saliency detection techniques are able to provide some rough information about object-of-interest since they highlight high-contrast or high attention regions or pixels. However, the generated saliency map is often noisy and directly applying it for segmentation often leads to erroneous results. Motivated by the recent progress on image co-segmentation and internet image retrieval techniques, in this paper, we propose to use the user input image for segmentation as a query image to Google Images and then employ the top returned Google images to build up the knowledge about the object-of-interest in the user input image. Particularly, we develop a lightweight algorithm to learn the knowledge of the object-of-interest in the retrieved images to enhance the saliency map of the input image. Then, the enhanced saliency map is used to initialize the graph-cut to extract the object-of-interest. Experiments with the Mcgill dataset and multiple challenge cases demonstrate the effectiveness of our method in terms of producing a clean cutout.

Considering the existing depth recovery approaches that have different limitations when applying to Kinect depth data, in this paper, we propose to integrate their effective features including adaptive support region selection, reliable depth selection and color guidance together under a unified framework for Kinect depth recovery. In particular, we formulate our depth recovery as an energy minimization problem, which solves the depth hole-filling and denoising simultaneously. The energy function consists of a fidelity term and a regularization term. The fidelity term takes into account the characteristics of Kinect data. The regularization term is designed to incorporate the joint bilateral filtering (JBF) kernel and the joint trilateral filtering (JTF) kernel so as to facilitate both depth hole-filling and denoising. Moreover, the JBF kernel is modified to incorporate the structure information. Both simulations on the benchmark Middlebury dataset and experiments on real Kinect data show that our proposed method achieves state-of-the-art performance in terms of recovery accuracy and visual quality.

This paper considers the problem of automatically segmenting an image into a small number of regions that correspond to objects conveying semantics or high-level structure. Although such object-level segmentation usually requires additional high-level knowledge or learning process, we explore what low level cues can produce for this purpose. Our idea is to construct a feature vector for each pixel, which elaborately integrates spectral attributes, color Gaussian mixture models, and geodesic distance, such that it encodes global color and spatial cues as well as global structure information. Then, we formulate the Potts variational model in terms of the feature vectors to provide a variational image segmentation algorithm that is performed in the feature space. We also propose a heuristic approach to automatically select the number of segments. The use of feature attributes enables the Potts model to produce regions that are coherent in color and position, comply with global structures corresponding to objects or parts of objects and meanwhile maintain a smooth and accurate boundary. We demonstrate the effectiveness of our algorithm against the state-of-the-art with the data set from the famous Berkeley benchmark.

A reciprocal frame (RF) is a self-supported three-dimensional structure made up of three or more sloping rods, which form a closed circuit, namely an RF-unit. Large RF-structures built as complex grillages of one or a few similar RF-units have an intrinsic beauty derived from their inherent self-similar and highly symmetric patterns. Designing RF-structures that span over large domains is an intricate and complex task. In this paper, we present an interactive computational tool for designing RF-structures over a 3D guiding surface, focusing on the aesthetic aspect of the design. There are three key contributions in this work. First, we draw an analogy between RF-structures and plane tiling with regular polygons, and develop a computational scheme to generate coherent RF-tessellations from simple grammar rules. Second, we employ a conformal mapping to lift the 2D tessellation over a 3D guiding surface, allowing a real-time preview and efficient exploration of wide ranges of RF design parameters. Third, we devise an optimization method to guarantee the collinearity of contact joints along each rod, while preserving the geometric properties of the RF-structure. Our tool not only supports the design of wide variety of RF pattern classes and their variations, but also allows preview and refinement through interactive controls.

The state-of-the-art interactive image segmentation algorithms are often not able to produce accurate segmentation results with one-shot user input, and they frequently rely on laborious user editing to refine the segmentation boundary. In this paper, we propose a constrained active contour method for boundary refinement, which can be used to improve the segmentation results of many existing region-based interactive segmentation algorithms. Our constrained active contour model exhibits many desired properties for a good boundary refinement tool, including the robustness to user inputs, the ability to produce a smooth and accurate boundary contour, and the ability to handle topology changes. Experimental results show that the proposed refinement tool is highly effective and can significantly improve initial segmentation results without additional user inputs.

The state-of-the-art interactive image segmentation algorithms are sensitive to the user inputs and often unable to produce an accurate boundary with a small amount of user interaction. They frequently rely on laborious user editing to refine the segmentation boundary. In this paper, we propose a robust and accurate interactive method based on the recently developed continuous-domain convex active contour model. The proposed method exhibits many desirable properties of an effective interactive image segmentation algorithm, including robustness to user inputs and different initializations, the ability to produce a smooth and accurate boundary contour, and the ability to handle topology changes. Experimental results on a benchmark data set show that the proposed tool is highly effective and outperforms the state-of-the-art interactive image segmentation algorithms.

The problem of decomposing a 3D mesh into meaningful segments (or parts) is of great practical importance in computer graphics. This article presents a variational mesh decomposition algorithm that can efficiently partition a mesh into a prescribed number of segments. The algorithm extends the Mumford-Shah model to 3D meshes that contains a data term measuring the variation within a segment using eigenvectors of a dual Laplacian matrix whose weights are related to the dihedral angle between adjacent triangles and a regularization term measuring the length of the boundary between segments. Such a formulation simultaneously handles segmentation and boundary smoothing, which are usually two separate processes in most previous work. The efficiency is achieved by solving the Mumford-Shah model through a saddle-point problem that is solved by a fast primal-dual method. A preprocess step is also proposed to determine the number of segments that the mesh should be decomposed into. By incorporating this preprocessing step, the proposed algorithm can automatically segment a mesh into meaningful parts. Furthermore, user interaction is allowed by incorporating the user's inputs into the variational model to reflect the user's special intention. Experimental results show that the proposed algorithm outperforms competitive segmentation methods when evaluated on the Princeton Segmentation Benchmark.

The existing high-accuracy environment matting extraction methods usually require the capturing of thousands of sample images and spend several hours in data acquisition. In this paper, a fast environment matting algorithm is proposed to ex tract the environment matte data effectively and efficiently. In particular, we incorporate the recently developed compressive sensing theory to simplify the data acquisition process. More over, taking into account special properties of light refraction and reflection effects of transparent object, we further propose to use hierarchical sampling and group clustering based recovery to accelerate the matte extraction process. Compared with the state-of-the-art approaches, our proposed algorithm significantly accelerates the environment matting extraction process while still achieving high-accuracy results.

Abstract In digital image editing, environment matting and compositing are fundamental and interesting operations that can capture and simulate the refraction and reflection effects of light from an environment. The state‐of‐the‐art real‐time environment matting and compositing method is short of flexibility, in the sense that it has to repeat the entire complex matte acquisition process if the distance between the object and the background is different from that in the acquisition stage, and also lacks accuracy, in the sense that it can only remove noises but not errors. In this paper, we introduce the concept of refractive vector and propose to use a refractive vector field as a new representation for environment matte. Such refractive vector field provides great flexibility for transparent‐object environment matting and compositing. Particularly, with only one process of the matte acquisition and the refractive vector field extraction, we are able to composite the transparent object into an arbitrary background at any distance. Furthermore, we introduce a piecewise vector field fitting algorithm to simultaneously remove both noises and errors contained in the extracted matte data. Experimental results show that our method is less sensitive to artefacts and can generate perceptually good composition results for more general scenarios.

This paper considers the problem of interactively finding the cutting contour to extract components from an existing mesh. First, we propose a constrained random walks algorithm that can add constraints to the random walks procedure and thus allows for a variety of intuitive user inputs. Second, we design an optimization process that uses the shortest graph path to derive a nice cut contour. Then a new mesh cutting algorithm is developed based on the constrained random walks plus the optimization process. Within the same computational framework, the new algorithm provides a novel user interface for interactive mesh cutting that supports three typical user inputs and also their combinations: 1) foreground/background seed inputs: the user draws strokes specifying seeds for “foreground” (i.e., the part to be cut out) and “background” (i.e., the rest); 2) soft constraint inputs: the user draws strokes on the mesh indicating the region which the cuts should be made nearby; and 3) hard constraint inputs: the marks which the cutting contour must pass. The algorithm uses feature sensitive metrics that are based on surface geometric properties and cognitive theory. The integration of the constrained random walks algorithm, the optimization process, the feature sensitive metrics, and the varieties of user inputs makes the algorithm intuitive, flexible, and effective as well. The experimental examples show that the proposed cutting method is fast, reliable, and capable of producing good results reflecting user intention and geometric attributes.

Seeded image segmentation is a popular type of supervised image segmentation in computer vision and image processing. Previous methods of seeded image segmentation treat the image as a weighted graph and minimize an energy function on the graph to produce a segmentation. In this paper, we propose to conduct the seeded image segmentation according to the result of a heat diffusion process in which the seeded pixels are considered to be the heat sources and the heat diffuses on the image starting from the sources. After the diffusion reaches a stable state, the image is segmented based on the pixel temperatures. It is also shown that our proposed framework includes the RandomWalk algorithm for image segmentation as a special case which diffuses only along the two coordinate axes. To better control diffusion, we propose to incorporate the attributes (such as the geometric structure) of the image into the diffusion process, yielding an anisotropic diffusion method for image segmentation. The experiments show that the proposed anisotropic diffusion method usually produces better segmentation results. In particular, when the method is tested using the groundtruth dataset of Microsoft Research Cambridge (MSRC), an error rate of 4.42% can be achieved, which is lower than the reported error rates of other state-of-the-art algorithms.

This paper presents a sketch-based modeling system with auxiliary planes as references for 3D freeform shape design. The user first creates a rough 3D model of arbitrary topology by sketching some contours of the model. Then the user can use sketching to perform deformation, extrusion, etc, to edit the model. To regularize and interpret the user's inputs properly, we introduce some rules for the strokes into the system, which are based on both the semantic meaning of the sketched strokes and human psychology. Unlike other sketching systems, all the creation and editing operations in the presented system are performed with reference to some auxiliary planes that are automatically constructed based on the user’s sketches or default settings. The use of reference planes provides a heuristic solution to the problem of ambiguity of 2D interface for modeling in 3D space. Examples demonstrate that the presented system can allow the user to intuitively and intelligently create and edit 3D models even with complex topology, which is usually difficult in other similar sketch-based modeling systems.

Existing digital image composition algorithms neglect the out-of-gamut problem, i.e. some pixel values in a composited image exceed the displayable or printable range. The commonly used solutions, including hard clipping or linear scaling, result in either detail loss or global contrast reduction. Directly applying the existing high dynamic range (HDR) compression algorithms cannot achieve pleasant visual quality either. In our previous work, we proposed a gamut fitting algorithm by formulating gamut fitting as an energy-minimization problem and used the cubic polynomial in the Bernstein-Bézier form to compute the optimal mapping curve. Despite the good performance achieved, the problem of the previous algorithm lies in the necessity of fine tuning the weighting parameter in the proposed energy function. In this paper, we further improve our previous method by using piecewise mapping curves with multiple Bernstein polynomials to find the optimal mapping curve. The proposed approach can be regarded as a post-process to enhance the visual quality of the resulting images from image composition applications. The performance of the proposed method is compared with our previous method and the state-of-the-art HDR compression algorithms.

Patch-based image completion proceeds by iteratively filling the target (unknown) region by the best matching patches in the source image. In most existing such algorithms, the size of the patches is either fixed and specified by a default number or simply chosen to be inversely proportional to the spatial frequency. However, it is noted that the patch size affects how well the filled patch captures the local characteristics of the source image and thus the final completion accuracy. Thus in this paper we propose a new method to compute appropriate patch sizes for image completion to improve its performance. In particular, we formulate the patch size determination as an optimization problem that minimizes an objective function involving image gradients and distinct and homogenous features. Experimental results show that our method can provide a significant enhancement to patch-based image completion algorithms.

Abstract This paper considers the problem of interactively finding the cutting contour to extract components from a given mesh. Some existing methods support cuts of arbitrary shape but require careful and tedious input from the user. Others need little user input however they are sensitive to user input and need a postprocessing step to smooth the generated jaggy cutting contours. The popular geometric snake can be used to optimize the cutting contour, but it cannot deal with the topology change. In this paper, we propose a geodesic curvature flow based framework to overcome all these problems. Since in many cases the meaningful cutting contour on a 3D mesh is locally shortest in the sense of some weighted curve length, the geodesic curvature flow is an ideal tool for our problem. It evolves the cutting contour to the nearby local minimum. We should mention that the previous numerical scheme, discretized geodesic curvature flow (dGCF) is too slow and has not been applied to mesh segmentation. With a careful observation to dGCF, we devise here a fast computation scheme called fast geodesic curvature flow (FGCF), which only needs to solve a smaller and easier problem. The initial cutting contour is generated by a variant of random walks algorithm, which is very fast and gives reasonable cutting result with little user input. Experiment results on the benchmark mesh segmentation data set show that our proposed framework is robust to user input and capable of producing good results reflecting geometric features and human shape perception.

Abstract Triangular meshes are a popular geometric representation for 3D models used in computer graphics. Parameterization is a process that establishes a mapping between the surface of a model and a suitable domain. This paper considers the problem of parameterizing triangular meshes that have tubular shapes. Unlike an open mesh that is of plane topological type, a tubular mesh gives rise to some special issues in parameterization due to its mesh structure. This paper presents an edge‐based parameterization method, in which the edges rather than the vertices of the mesh are treated as the target for parameterization. It first parameterizes the edges on the two boundaries of the tubular mesh, then parameterizes the internal edges based on the mean value coordinates, and finally computes the parameters of the mesh vertices. The method does not need cutting of the mesh. It improves conventional cutting‐based algorithms, which cut the mesh to make it a disk topologically, and overcomes the problems of cutting paths that are the zigzag paths leading to suboptimal parameterizations and the difficulty in finding good cutting paths. Some applications such as surface fitting and texture mapping are also provided. Copyright © 2009 John Wiley & Sons, Ltd.

For transmitting complex 3-D models over bandwidth-limited networks, efficient mesh coding and transmission are indispensable. The state-of-the-art 3-D mesh transmission system employs a wavelet-based progressive mesh coder, which converts an irregular mesh into a semi-regular mesh and directly applies the zerotree-like image coders to compress the wavelet vectors, and view-dependent transmission, which saves the transmission bandwidth through only delivering the visible portions of a mesh model. We propose methods to improve both progressive mesh coding and transmission based on thorough rate-distortion analysis. In particular, by noticing that the dependency among the wavelet coefficients generated in remeshing is not being considered in the existing approaches, we propose to introduce a preprocessing step to scale up the wavelets so that the inherent dependency of wavelets can be truly understood by the zerotree-like image compression algorithms. The weights used in the scaling process are carefully designed through thoroughly analyzing the distortions of wavelets at different refinement levels. For the transmission part, we propose to incorporate the illumination effects into the existing view-depend progressive mesh transmission system to further improve the performance. We develop a novel distortion model that considers both illumination distortion and geometry distortion. Based on our proposed distortion model, given the viewing and lighting parameters, we are able to optimally allocate bits among different segments in real time. Simulation results show significant improvements in both progressive compression and transmission.

One weakness in the existing interactive image segmentation algorithms is the lack of more intelligent ways to understand the intention of user inputs. In this paper, we advocate the use of multiple intuitive user inputs to better reflect a user's intention. In particular, we propose a constrained random walks algorithm that facilitates the use of three types of user inputs: 1) foreground and background seed input, 2) soft constraint input, and 3) hard constraint input, as well as their combinations. The foreground and background seed input allows a user to draw strokes to specify foreground and background seeds. The soft constraint input allows a user to draw strokes to indicate the region that the boundary should pass through. The hard constraint input allows a user to specify the pixels that the boundary must align with. Our proposed method supports all three types of user inputs in one coherent computational framework consisting of a constrained random walks and a local editing algorithm, which allows more precise contour refinement. Experimental results on two benchmark data sets show that the proposed framework is highly effective and can quickly and accurately segment a wide variety of natural images with ease.

The major drawback of real-time environment matting method is that the extracted environment matte data often contains significant amount of noise and errors. Although some filtering methods have been employed to remove the noise and obtain acceptable composition results, they are incapable of removing potential errors. In this paper, we first establish a light motion field to better describe the environmental matting effect of transparent objects and propose a new vector field fitting algorithm to simultaneously remove both noise and errors in the extracted matte data by using energy minimization approach. Experimental results show that our method is less sensitive to noise and error and can generate perceptually better composition results than the existing real-time environment matting approaches.

Existing digital image composition algorithms neglect the out-of-gamut problem, i.e. some pixel values in a composited image exceed the displayable or printable range. In this paper, we show that the commonly used solution, i.e. hard clipping or linear scaling, results in either detail loss or global contrast reduction. Directly applying the existing high dynamic range (HDR) compression algorithms cannot achieve pleasant visual quality either. Thus, we propose a gamut fitting method to solve this out-of-gamut problem in image composition. In particular, we formulate gamut fitting as a multi-criteria optimization problem. Polynomials in the Bernstein-Bézier form are used to compute the optimal gamut mapping curve. The proposed approach can be regarded as a post-processing procedure to enhance the visual quality of the resulting images from image composition applications.

While the state-of-the-art image composition algorithms subtly handle the object boundary to achieve seamless image copy-and-paste, it is observed that they are unable to preserve the color fidelity of the source object, often require quite an amount of user interactions, and often fail to achieve realism when there exists salient discrepancy between the background textures in the source and destination images. These observations motivate our research towards color controlled natural and seamless image composition with least user interactions. In particular, based on the Poisson image editing framework, we first propose a variational model that considers both the gradient constraint and the color fidelity. The proposed model allows users to control the coloring effect caused by gradient domain fusion. Second, to have less user interactions, we propose a distance-enhanced random walks algorithm, through which we avoid the necessity of accurate image segmentation while still able to highlight the foreground object. Third, we propose a multiresolution framework to perform image compositions at different subbands so as to separate the texture and color components to simultaneously achieve smooth texture transition and desired color control. The experimental results demonstrate that our proposed framework achieves better and more realistic results for images with salient background color or texture differences, while providing comparable results as the state-of-the-art algorithms for images without the need of preserving the object color fidelity and without significant background texture discrepancy.

The latest wavelet based 3D mesh coding schemes convert an irregular mesh into a semi-regular mesh and directly apply the zerotree-like image coders to compress the wavelet vectors generated in the remeshing process. The major problem of such type of approaches is that the particular properties of semi-regular meshes are not being considered in the zerotree-like image coders. In this paper, we propose an improved wavelet based 3D mesh coder. The basic idea is to introduce a preprocessing step to scale up the vector wavelets generated in remeshing so that the inherent dependency of wavelets can be truly understood by the zerotree-like image compression algorithms. The weights used in the scaling process are carefully designed through thoroughly analyzing the distortions of wavelets at different refinement levels. Experimental results show that our proposed mesh coder significantly outperforms the state-of-the-art wavelet based 3D mesh compression scheme.

For wireless network based graphics applications, a key challenge is how to efficiently transmit complex 3-D models over bandwidth-limited wireless channels. Most existing 3-D mesh transmission systems do not consider such a view-dependent delivery issue, and thus transmit unnecessary portions of 3-D mesh models, which leads to the waste in precious wireless network bandwidth. In this paper, we propose a novel view-dependent 3-D model transmission scheme, where a 3-D model is partitioned into a number of segments, each segment is then independently coded using the MPEG-4 3DMC coding algorithm, and finally only the visible segments are selected and delivered to the client. Moreover, we also propose analytical models to find the optimal number of segments so as to minimize the average transmission size. Simulation results show that such a view-based 3-D model transmission is able to substantially save the transmission bandwidth and therefore has a significant impact on wireless graphics applications.

For network-based graphics applications, a key challenge is how to efficiently transmit complex three-dimensional (3D) models over bandwidth-limited communication channels such as wireless links. Most existing 3D mesh coding algorithms do not consider the view-dependent rendering issue, and therefore result in transmitting unnecessary portions of 3D mesh models which leads to the waste in precious network bandwidth. In this paper, we propose a novel view-dependent 3D model transmission scheme, where a 3D model is partitioned into a number of segments, each segment is then independently coded using the MPEG-4 3DMC coding algorithm, and finally only the visible segments are selected and delivered to the client. Such a view-based 3D model transmission is able to substantially save the transmission bandwidth and therefore has a significant impact on wireless network based graphics applications.

This paper considers the problems of how to introduce shape parameters into recursive subdivision schemes for additional shape control and how to find appropriate values of shape parameters to improve the quality of subdivision surface shapes. Following Brunet, we restrict our discussion to the algorithm that constructs a Doo-Sabin subdivision surface to interpolate the vertices of an input polyhedron with arbitrary topology. While Brunet defined one so-called "shape handle" for each vertex of the initial polyhedron, which is used to scale the type-V face obtained after the first step of the subdivision process, we introduce three shape parameters for each vertex: one for the scale and the other two for the orientation of the type-V face. This gives more degrees of freedom to optimize the shape of the result interpolatory surface. We develop a genetic algorithm to compute the optimal set of shape parameters such that a "fairness" measure of the surface is minimized. Examples are provided to demonstrate the effects of the optimal shape parameters on the final interpolatory surfaces.

The construction of a smooth surface interpolating a mesh of arbitrary topological type is an important problem in many graphics applications. This paper presents a two-phase process, based on a topological modification of the control mesh and a subsequent Catmull-Clark subdivision, to construct a smooth surface that interpolates some or all of the vertices of a mesh with arbitrary topology. It is also possible to constrain the surface to have specified tangent planes at an arbitrary subset of the vertices to be interpolated. The method has the following features: 1) It is guaranteed to always work and the computation is numerically stable, 2) there is no need to solve a system of linear equations and the whole computation complexity is O(K) where K is the number of the vertices, and 3) each vertex can be associated with a scalar shape handle for local shape control. These features make interpolation using Catmull-Clark surfaces simple and, thus, make the new method itself suitable for interactive free-form shape design.

A typical NURBS surface model has a large percentage of superfluous control points that significantly interfere with the design process. This paper presents an algorithm for eliminating such superfluous control points, producing a T-spline. The algorithm can remove substantially more control points than competing methods such as B-spline wavelet decomposition. The paper also presents a new T-spline local refinement algorithm and answers two fundamental open questions on T-spline theory.

This paper presents a generalization of non-uniform B-spline surfaces called T-splines. T-spline control grids permit T-junctions, so lines of control points need not traverse the entire control grid. T-splines support many valuable operations within a consistent framework, such as local refinement, and the merging of several B-spline surfaces that have different knot vectors into a single gap-free model. The paper focuses on T-splines of degree three, which are C 2 (in the absence of multiple knots). T-NURCCs (Non-Uniform Rational Catmull-Clark Surfaces with T-junctions) are a superset of both T-splines and Catmull-Clark surfaces. Thus, a modeling program for T-NURCCs can handle any NURBS or Catmull-Clark model as special cases. T-NURCCs enable true local refinement of a Catmull-Clark-type control grid: individual control points can be inserted only where they are needed to provide additional control, or to create a smoother tessellation, and such insertions do not alter the limit surface. T-NURCCs use stationary refinement rules and are C 2 except at extraordinary points and features.

This paper presents a method for determining a priori a constant parameter interval for tessellating a rational curve or surface such that the deviation of the curve or surface from its piecewise linear approximation is within a specified tolerance. The parameter interval is estimated based on information about second-order derivatives in the homogeneous coordinates, instead of using affine coordinates directly. This new step size can be found with roughly the same amount of computation as the step size in Cheng [1992], though it can be proven to always be larger than Cheng's step size. In fact, numerical experiments show the new step is typically orders of magnitude larger than the step size in Cheng [1992]. Furthermore, for rational cubic and quartic curves, the new step size is generally twice as large as the step size found by computing bounds on the Bernstein polynomial coefficients of the second derivatives function.

Doo-Sabin and Catmull-Clark subdivision surfaces are based on the notion of repeated knot insertion of uniform tensor product B-spline surfaces. This paper develops rules for non-uniform Doo-Sabin and Catmull-Clark surfaces that generalize non-uniform tensor product B-spline surfaces to arbitrary topologies. This added flexibility allows, among other things, the natural introduction of features such as cusps, creases, and darts, while elsewhere maintaining the same order of continuity as their uniform counterparts.