This paper presents a novel Contrastive Self-Attention Convolutional Neural Network (CSA-CNN) model with enhanced Difficulty Aware (DA) loss function to improve the segmentation of pupils in eye images. The incorporation of transformer-style self-attention and Difficulty-Aware loss in a UNET-style architecture allows for robust feature representation and promotes shape alignment. The novel model was trained on two public databases (LPW and RIT-Eyes) and evaluated on two other public datasets (ExCuSe and ElSe). When compared with seven state-of-the-art pupil center detection methods, the CSA-CNN showed improvement of over 6% in pupil center detection accuracy (detection within 5 pixels of the labeled center) and more than 9% in Intersection Over Union (IOU) accuracy, compared to the best performer among the other seven methods. Furthermore, when the CSA-CNN model was integrated into a glint-based eye tracking system that uses learning based methods to detect pupil-center, we saw a 25% improvement in gaze accuracy.

As video lectures are gaining more popularity, determining their effectiveness and obtaining valuable feedback have become necessary. To measure the learners’ attention state during video lectures, we specified the conceptual “with-me-ness” (WMN) as slide-level WMN (SL-WMN). The content domain on each slide was automatically extracted via an optical character recognition (OCR)-based method, while the eye gazing behaviors were analyzed through a Gaussian mixture modeling (GMM) fixation clustering method. Both domain-specific WMN and behavior-enriched WMN were then computed via OCR- and GMM-OCR-based methods to measure the learners’ attention levels. We conducted an experiment to collect in-lecture eye-tracking data, video recordings, and post-lecture test scores from 50 Grade 8 students. The results demonstrated that both OCR- and GMM-OCR-based SL-WMNs are reliable and compatible automatic measurements of learners’ attention states during video lectures. A survey from participating learners and lecturers also revealed highly favorable feedback for the developed SL-WMNs.

Abstract In physics‐based cloth animation, rich folds and detailed wrinkles are achieved at the cost of expensive computational resources and huge labor tuning. Data‐driven techniques make efforts to reduce the computation significantly by utilizing a preprocessed database. One type of methods relies on human poses to synthesize fitted garments, but these methods cannot be applied to general cloth animations. Another type of methods adds details to the coarse meshes obtained through simulation, which does not have such restrictions. However, existing works usually utilize coordinate‐based representations which cannot cope with large‐scale deformation, and requires dense vertex correspondences between coarse and fine meshes. Moreover, as such methods only add details, they require coarse meshes to be sufficiently close to fine meshes, which can be either impossible, or require unrealistic constraints to be applied when generating fine meshes. To address these challenges, we develop a temporally and spatially as‐consistent‐as‐possible deformation representation (named TS‐ACAP) and design a DeformTransformer network to learn the mapping from low‐resolution meshes to ones with fine details. This TS‐ACAP representation is designed to ensure both spatial and temporal consistency for sequential large‐scale deformations from cloth animations. With this TS‐ACAP representation, our DeformTransformer network first utilizes two mesh‐based encoders to extract the coarse and fine features using shared convolutional kernels, respectively. To transduct the coarse features to the fine ones, we leverage the spatial and temporal Transformer network that consists of vertex‐level and frame‐level attention mechanisms to ensure detail enhancement and temporal coherence of the prediction. Experimental results show that our method is able to produce reliable and realistic animations in various datasets at high frame rates with superior detail synthesis abilities compared to existing methods.

Recent learning-based approaches show promising performance improvement for the scene text removal task but usually leave several remnants of text and provide visually unpleasant results. In this work, a novel end-to-end framework is proposed based on accurate text stroke detection. Specifically, the text removal problem is decoupled into text stroke detection and stroke removal; we design separate networks to solve these two subproblems, the latter being a generative network. These two networks are combined as a processing unit, which is cascaded to obtain our final model for text removal. Experimental results demonstrate that the proposed method substantially outperforms the state-of-the-art for locating and erasing scene text. A new large-scale real-world dataset with 12,120 images has been constructed and is being made available to facilitate research, as current publicly available datasets are mainly synthetic so cannot properly measure the performance of different methods.

We introduce a new method to efficiently track complex interfaces among multi-phase immiscible fluids. Unlike existing techniques, we use a mesh-based representation for global liquid surfaces while selectively modeling some local surficial regions with regional level sets (RLS) to handle complex geometries that are difficult to resolve with explicit topology operations. Such a semi-explicit surface mechanism can preserve volume, fine features and foam-like thin films under a relatively low computational expenditure. Our method processes the surface evolution by sampling the fluid domain onto a spectrally refined grid (SRG) and performs efficient grid scanning, generalized interpolations and topology operations on the basis of this grid structure. For the RLS surface part, we propose an accurate advection scheme targeted at SRG. For the explicit mesh part, we develop a fast grid-scanning technique to voxelize the meshes and introduce novel strategies to detect grid cells that contain inconsistent mesh components. A robust algorithm is proposed to construct consistent local meshes to resolve mesh penetrations, and handle the coupling between explicit mesh and RLS surficial regions. We also provide further improvement on handling complicated topological variations, and strategies for remeshing mesh/RLS interconversions.

Abstract We present a novel isotropic surface remeshing algorithm that automatically aligns the mesh edges with an underlying directional field. The alignment is achieved by minimizing an energy function that combines both centroidal Voronoi tessellation (CVT) and the penalty enforced by a six‐way rotational symmetry field. The CVT term ensures uniform distribution of the vertices and high remeshing quality, and the field constraint enforces the directional alignment of the edges. Experimental results show that the proposed approach has the advantages of isotropic and field‐aligned remeshing. Our algorithm is superior to the representative state‐of‐the‐art approaches in various aspects.

Abstract We propose a novel deep learning‐based method, called mesh superresolution, to enrich low‐resolution (LR) cloth meshes with wrinkles. A pair of low and high‐resolution (HR) meshes are simulated, with the simulation of the HR mesh tracks with that of the LR mesh. The frame data are converted into geometry images and used as a training data set. A residual network, called SR residual network, is employed to train an image synthesizer that superresolves an LR image into an HR one. Once the HR image is converted back to an HR mesh, it is abundant in wrinkles compared with its coarse counterpart. The synthesizing is very efficient and is 24× faster than a full HR simulation. We demonstrate the performances of mesh superresolution with various simulation scenes.

Abstract We present an efficient and stable framework, called Unified Intersection Resolver (UIR), for cloth simulation systems where not only impending collisions but also pre‐existing penetrations often arise. These two types of collisions are handled in a unified manner, by detecting edge‐face intersections first and then forming penetration stencils to be resolved iteratively. A stencil is a quadruple of vertices and it reveals either a vertex‐face or an edge‐edge collision event happened. Each quadruple also implicitly defines a collision normal, through which the four stencil vertices can be relocated, so that the corresponding edge‐face intersection disappear. We deduce three different ways, i.e., from predefined surface orientation, from history data and from global intersection analysis, to determine the collision normals of these stencils robustly. Multiple stencils that constitute a penetration region are processed simultaneously to eliminate penetrations. Cloth trapped in pinched environmental objects can be handled easily within our framework. We highlight its robustness by a number of challenging experiments involving collisions.

Abstract The cloth simulation systems often suffer from excessive extension on the polygonal mesh, so an additional strain‐limiting process is typically used as a remedy in the simulation pipeline. A cloth model can be discretized as either a quadrilateral mesh or a triangular mesh, and their strains are measured differently. The edge‐based strain‐limiting method for a quadrilateral mesh creates anisotropic behaviour by nature, as discretization usually aligns the edges along the warp and weft directions. We improve this anisotropic technique by replacing the traditionally used equality constraints with inequality ones in the mathematical optimization, and achieve faster convergence. For a triangular mesh, the state‐of‐the‐art technique measures and constrains the strains along the two principal (and constantly changing) directions in a triangle, resulting in an isotropic behaviour which prohibits shearing. Based on the framework of inequality‐constrained optimization, we propose a warp and weft strain‐limiting formulation. This anisotropic model is more appropriate for textile materials that do not exhibit isotropic strain behaviour.

We present a new method for resolving some of the most frequently occurring penetration configurations in cloth simulation, with a history-free discrete collision detection (DCD). To be adapted to a wide range of applications, this method is also orientation-free as no inside/outside surface setting is assumed. Our method relies on a heuristic paradigm "small region is illegal" to identify penetrating (illegal) regions whenever possible. First, intersection contours are constructed and classified, followed by a global analysis of the collision configurations. Then for those surfaces having identifiable illegal regions, we use dynamic repulsive normal (DRN) to compute proper displacements to relocate incorrectly configured vertices to be intersection-free. For those configurations that do not clearly define legal/illegal regions, displacements are designed to push one mesh to the boundary of the other. The proposed method can also be used in the context of time-dependent simulation of complex deformable surfaces, making it an competitive alternative to the popular CCD-based approach.

Geometric flows have been successfully used for surface modeling and designing, largely because they are inherently good at controlling geometric shape evolution. Variational image segmentation approaches, on the other hand, detect objects of interest by deforming certain given shapes. This motivates us to revisit the minimal partition problem for segmentation of images, and propose a new geometric flow-based formulation and solution to it. Our model intends to derive a mapping that will evolve given contours or piecewise-constant regions toward objects in the image. The mapping is approximated by B-spline basis functions, and the positions of the control points are to be determined. Starting with the energy functional based on intensity averaging, we derive a Euler-Lagrange equation and then a geometric evolution equation. The linearized system of equations is efficiently solved via a special matrix-vector multiplication technique. Furthermore, we extend the piecewise-constant model to a piecewise-smooth model which effectively handles images with intensity inhomogeneity.

Abstract Computer simulation of cloth is often plagued by springs being over‐stretched. The evaluation of impulses to prevent over‐stretching is explained step‐by‐step. Our impulse approach controls the length of springs by a nonlinear system and then a novel linearization of it into a symmetric positive definite system. The cloth/solid collision handling is integrated into the linearized system seamlessly. Some results based on this method are also presented.