Zhou Ziheng, Huacong Tang, Mingjie Bi, Wanying He, Fang Sun, Yizhou Sun, Ying Nian Wu, Demetri Terzopoulos, Yipeng Kang, Fangwei Zhong. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

We introduce MPM Lite, a hybrid Lagrangian/Eulerian method that eliminates the need for particle-based quadrature at solve time. Standard Material Point Method (MPM) practices suffer from a performance bottleneck where expensive implicit solves are proportional to particle-per-cell (PPC) counts due to the the choices of particle-based quadrature and wide-stencil kernels. By contrast, MPM Lite treats particles primarily as carriers of kinematic state and material history. Conceptualizing the background Cartesian grid as a voxel hexahedral mesh, we resample particle states onto fixed-location quadrature points using efficient, compact linear kernels. This architectural shift allows force assembly and the entire time-integration process to proceed without accessing particles, thus making the solver's complexity independent of the particle count. At the core of our method is a novel stress transfer and stretch reconstruction strategy. To avoid non-physical averaging of deformation gradients, we resample the extensive Kirchhoff stress and derive a rotation-free deformation reference solution, which naturally supports an optimization-based incremental potential formulation. Consequently, MPM Lite can be implemented as modular resampling units coupled with an FEM-style integration module, enabling the direct use of off-the-shelf nonlinear solvers, preconditioners, and unambiguous boundary conditions. We demonstrate through extensive experiments that MPM Lite preserves the robustness and versatility of traditional MPM across diverse materials while delivering significant speedups in implicit settings while simultaneously improving explicit ones. Project page: https://mpmlite.github.io.

How can one efficiently generate high-quality, wide-scope 3D scenes from arbitrary single images? Existing methods suffer several drawbacks, such as requiring multi-view data, time-consuming per-scene optimization, distorted geometry in occluded areas, and low visual quality in backgrounds. Our novel 3D scene reconstruction pipeline overcomes these limitations to tackle the aforesaid challenge. Specifically, we introduce a large-scale reconstruction model that leverages latents from a video diffusion model to predict 3D Gaussian Splattings of scenes in a feed-forward manner. The video diffusion model is designed to create videos precisely following specified camera trajectories, allowing it to generate compressed video latents that encode multi-view information while maintaining 3D consistency. We train the 3D reconstruction model to operate on the video latent space with a progressive learning strategy, enabling the efficient generation of high-quality, wide-scope, and generic 3D scenes. Extensive evaluations across various datasets affirm that our model significantly outperforms existing single-view 3D scene generation methods, especially with out-of-domain images. Thus, we demonstrate for the first time that a 3D reconstruction model can effectively be built upon the latent space of a diffusion model in order to realize efficient 3D scene generation. Project page: https://snap-research.github.io/wonderland/

Video summarization, by selecting the most informative and/or user-relevant parts of original videos to create concise summary videos, has high research value and consumer demand in today’s video proliferation era. Multi-modal video summarization that accomodates user input has become a research hotspot. However, current multi-modal video summarization methods suffer from two limitations. First, existing methods inadequately fuse information from different modalities and cannot effectively utilize modality-unique features. Second, most multi-modal methods focus on video and text modalities, neglecting the audio modality, despite the fact that audio information can be very useful in certain types of videos. In this paper we propose CFSum, a transformer-based multi-modal video summarization framework with coarse-fine fusion. CFSum exploits video, text, and audio modal features as input, and incorporates a two-stage transformer-based feature fusion framework to fully utilize modality-unique information. In the first stage, multi-modal features are fused simultaneously to perform initial coarse-grained feature fusion, then, in the second stage, video and audio features are explicitly attended with the text representation yielding more fine-grained information interaction. The CFSum architecture gives equal importance to each modality, ensuring that each modal feature interacts deeply with the other modalities. Our extensive comparative experiments against prior methods and ablation studies on various datasets confirm the effectiveness and superiority of CFSum.

A major challenge for Multi-Agent Systems (MAS) is enabling agents to adapt dynamically to diverse environments in which opponents and teammates may continually change. Agents trained using conventional methods tend to excel only within the confines of their training cohorts; their performance drops significantly when confronting unfamiliar agents. To address this shortcoming, we introduce Inverse Attention Agents that adopt concepts from the Theory of Mind (ToM) implemented algorithmically using an attention mechanism trained in an end-to-end manner. Crucial to determining the final actions of these agents, the weights in their attention model explicitly represent attention to different goals. We furthermore propose an inverse attention network that deduces the ToM of agents based on observations and prior actions. The network infers the attentional states of other agents, thereby refining the attention weights to adjust the agent's final action. We conduct experiments in a continuous environment, tackling demanding tasks encompassing cooperation, competition, and a blend of both. They demonstrate that the inverse attention network successfully infers the attention of other agents, and that this information improves agent performance. Additional human experiments show that, compared to baseline agent models, our inverse attention agents exhibit superior cooperation with humans and better emulate human behaviors.

Robotic manipulation of slender objects is challenging, especially when the induced deformations are large and nonlinear. Traditionally, learning-based control approaches, such as imitation learning, have been used to address deformable material manipulation. These approaches lack generality and often suffer critical failure from a simple switch of material, geometric, and/or environmental (e.g., friction) properties. This article tackles a fundamental but difficult deformable manipulation task: forming a predefined fold in paper with only a single manipulator. A sim2real framework combining physically-accurate simulation and machine learning is used to train a deep neural network capable of predicting the external forces induced on the manipulated paper given a grasp position. We frame the problem using scaling analysis, resulting in a control framework robust against material and geometric changes. Path planning is then carried out over the generated “neural force manifold” to produce robot manipulation trajectories optimized to prevent sliding, with offline trajectory generation finishing 15$\times$faster than previous physics-based folding methods. The inference speed of the trained model enables the incorporation of real-time visual feedback to achieve closed-loop model-predictive control. Real-world experiments demonstrate that our framework can greatly improve robotic manipulation performance compared to state-of-the-art folding strategies, even when manipulating paper objects of various materials and shapes.Note to Practitioners—This article is motivated by the need for efficient robotic folding strategies for stiff materials such as paper. Previous robot folding strategies have focused primarily on soft materials (e.g., cloth) possessing minimal bending resistance or relied on multiple complex manipulators and sensors, significantly increasing computational and monetary costs. In contrast, we formulate a robust, sim2real, physics-based method capable of folding papers of varying stiffness with a single manipulator. The proposed folding scheme is limited to papers of homogeneous material and folding along symmetric centerlines. Future work will involve formulating efficient methods for folding along arbitrary geometries and preexisting creases.

A large portion of volumetric medical data, especially magnetic resonance imaging (MRI) data, is anisotropic, as the through-plane resolution is typically much lower than the in-plane resolution. Both 3D and purely 2D deep learning-based segmentation methods are deficient in dealing with such volumetric data since the performance of 3D methods suffers when confronting anisotropic data, and 2D methods disregard crucial volumetric information. Insufficient work has been done on 2.5D methods, in which 2D convolution is mainly used in concert with volumetric information. These models focus on learning the relationship across slices, but typically have many parameters to train. We offer a Cross-Slice Attention Module (CSAM) with minimal trainable parameters, which captures information across all the slices in the volume by applying semantic, positional, and slice attention on deep feature maps at different scales. Our extensive experiments using different network architectures and tasks demonstrate the usefulness and generalizability of CSAM. Associated code is available at https://github.com/aL3x-O-o-Hung/CSAM.

Abstract Spiking neural networks (SNNs) are comprised of artificial neurons that, like their biological counterparts, communicate via electrical spikes. SNNs have been hailed as the next wave of deep learning as they promise low latency and low-power consumption when run on neuromorphic hardware. Current deep neural network models for computer vision often require power-hungry GPUs to train and run, making them great candidates to replace with SNNs. We develop and train a biomimetic, SNN-driven, neuromuscular oculomotor controller for a realistic biomechanical model of the human eye. Inspired by the ON and OFF bipolar cells of the retina, we use event-based data flow in the SNN to direct the necessary extraocular muscle-driven eye movements. We train our SNN models from scratch, using modified deep learning techniques. Classification tasks are straightforward to implement with SNNs and have received the most research attention, but visual tracking is a regression task. We use surrogate gradients and introduce a linear layer to convert membrane voltages from the final spiking layer into the desired outputs. Our SNN foveation network enhances the biomimetic properties of the virtual eye model and enables it to perform reliable visual tracking. Overall, with event-based data processed by an SNN, our oculomotor controller successfully tracks a visual target while activating 87.3% fewer neurons than a conventional neural network.

Understanding the continuous states of objects is essential for task learning and planning in the real world. However, most existing task learning benchmarks assume discrete (e.g., binary) object goal states, which poses challenges for the learning of complex tasks and transferring learned policy from simulated environments to the real world. Furthermore, state discretization limits a robot’s ability to follow human instructions based on the grounding of actions and states. To tackle these challenges, we present ARNOLD, a benchmark that evaluates language-grounded task learning with continuous states in realistic 3D scenes. ARNOLD is comprised of 8 language-conditioned tasks that involve understanding object states and learning policies for continuous goals. To promote language-instructed learning, we provide expert demonstrations with template-generated language descriptions. We assess task performance by utilizing the latest language-conditioned policy learning models. Our results indicate that current models for language-conditioned manipulations continue to experience significant challenges in novel goal-state generalizations, scene generalizations, and object generalizations. These findings highlight the need to develop new algorithms that address this gap and underscore the potential for further research in this area. Project website: https://arnold-benchmark.github.io.

Prostate cancer is the second leading cause of cancer death among men in the United States. The diagnosis of prostate MRI often relies on accurate prostate zonal segmentation. However, state-of-the-art automatic segmentation methods often fail to produce well-contained volumetric segmentation of the prostate zones since certain slices of prostate MRI, such as base and apex slices, are harder to segment than other slices. This difficulty can be overcome by leveraging important multi-scale image-based information from adjacent slices, but current methods do not fully learn and exploit such cross-slice information. In this paper, we propose a novel cross-slice attention mechanism, which we use in a Transformer module to systematically learn cross-slice information at multiple scales. The module can be utilized in any existing deep-learning-based segmentation framework with skip connections. Experiments show that our cross-slice attention is able to capture cross-slice information significant for prostate zonal segmentation in order to improve the performance of current state-of-the-art methods. Cross-slice attention improves segmentation accuracy in the peripheral zones, such that segmentation results are consistent across all the prostate slices (apex, mid-gland, and base). The code for the proposed model is available at https://bit.ly/CAT-Net.

Image segmentation is a key task in computer vision and image processing with important applications such as scene understanding, medical image analysis, robotic perception, video surveillance, augmented reality, and image compression, among others, and numerous segmentation algorithms are found in the literature. Against this backdrop, the broad success of deep learning (DL) has prompted the development of new image segmentation approaches leveraging DL models. We provide a comprehensive review of this recent literature, covering the spectrum of pioneering efforts in semantic and instance segmentation, including convolutional pixel-labeling networks, encoder-decoder architectures, multiscale and pyramid-based approaches, recurrent networks, visual attention models, and generative models in adversarial settings. We investigate the relationships, strengths, and challenges of these DL-based segmentation models, examine the widely used datasets, compare performances, and discuss promising research directions.

The retinal vasculature provides important clues in the diagnosis and monitoring of systemic diseases including hypertension and diabetes. The microvascular system is of primary involvement in such conditions, and the retina is the only anatomical site where the microvasculature can be directly observed. The objective assessment of retinal vessels has long been considered a surrogate biomarker for systemic vascular diseases, and with recent advancements in retinal imaging and computer vision technologies, this topic has become the subject of renewed attention. In this paper, we present a novel dataset, dubbed RAVIR, for the semantic segmentation of Retinal Arteries and Veins in Infrared Reflectance (IR) imaging. It enables the creation of deep learning-based models that distinguish extracted vessel type without extensive post-processing. We propose a novel deep learning-based methodology, denoted as SegRAVIR, for the semantic segmentation of retinal arteries and veins and the quantitative measurement of the widths of segmented vessels. Our extensive experiments validate the effectiveness of SegRAVIR and demonstrate its superior performance in comparison to state-of-the-art models. Additionally, we propose a knowledge distillation framework for the domain adaptation of RAVIR pretrained networks on color images. We demonstrate that our pretraining procedure yields new state-of-the-art benchmarks on the DRIVE, STARE, and CHASE_DB1 datasets. Dataset link: https://ravirdataset.github.io/data.

The transfer of facial expressions from people to 3D face models is a classic computer graphics problem. In this paper, we present a novel, learning-based approach to transferring facial expressions and head movements from images and videos to a biomechanical model of the face-head-neck musculoskeletal complex. Specifically, leveraging the Facial Action Coding System (FACS) as an intermediate representation of the expression space, we train a deep neural network to take in FACS Action Units (AUs) and output suitable facial muscle and jaw activations for the biomechanical model. Through biomechanical simulation, the activations deform the face, thereby transferring the expression to the model. The success of our approach is demonstrated through experiments involving the transfer of a range of expressive facial images and videos onto our biomechanical face-head-neck complex.

Annotation-efficient deep learning refers to methods and practices that yield high-performance deep learning models without the use of massive carefully labeled training datasets. This paradigm has recently attracted attention from the medical imaging research community because (1) it is difficult to collect large, representative medical imaging datasets given the diversity of imaging protocols, imaging devices, and patient populations, (2) it is expensive to acquire accurate annotations from medical experts even for moderately sized medical imaging datasets, and (3) it is infeasible to adapt data-hungry deep learning models to detect and diagnose rare diseases whose low prevalence hinders data collection.

To tackle the problem of limited annotated data, semi-supervised learning is attracting attention as an alternative to fully supervised models. Moreover, optimizing a multiple-task model to learn “multiple contexts” can provide better generalizability compared to single-task models. We propose a novel semi-supervised multiple-task model leveraging self-supervision and adversarial training—namely, self-supervised, semi-supervised, multi-context learning (S4MCL)—and apply it to two crucial medical imaging tasks, classification and segmentation. Our experiments on spine X-rays reveal that the S4MCL model significantly outperforms semi-supervised single-task, semi-supervised multi-context, and fully-supervised single-task models, even with a 50% reduction of classification and segmentation labels.

Scoliosis is a congenital disease in which the spine is deformed from its normal shape. Radiography is the most cost-effective and accessible modality for imaging the spine. Conventional spinal assessment, diagnosis of scoliosis, and treatment planning relies on tedious and time-consuming manual analysis of spine radiographs that is susceptible to observer variation. A reliable, fully-automated method that can accurately identify vertebrae, a crucial step in image-guided scoliosis assessment, is presently unavailable in the literature. Leveraging a novel, deep-learning-based image segmentation model, we develop an end-to-end spine radiograph analysis pipeline that automatically provides an accurate segmentation and identification of the vertebrae, culminating in the reliable estimation of the Cobb angle, the most widely used measurement to quantify the magnitude of scoliosis. Our experimental results with anterior-posterior spine X-ray images indicate that our system is effective in the identification and labeling of vertebrae, and can potentially provide assistance to medical practitioners in the assessment of scoliosis.

Semi-supervised learning has recently been attracting attention as an alternative to fully supervised models that require large pools of labeled data. Moreover, optimizing a model for multiple tasks can provide better generalizability than single-task learning. Leveraging self-supervision and adversarial training, we propose a novel, general purpose semi-supervised, multiple-task model-namely, self-supervised, semi-supervised, multi-task learning (S4MTL)-for accomplishing two important medical image analysis tasks: segmentation and diagnostic classification. Experimental results on chest and spine X-ray datasets confirm that our S4MTL model significantly outperforms semi-supervised single-task, semi/fully-supervised multi-task, and fully-supervised single-task models, even with a 50% reduction in class and segmentation labels.

Imaging anisotropy poses a critical challenge in applying deep learning models to 3D medical image analysis. Anisotropy downgrades model performance, especially when slice spacing varies significantly between training and clinical datasets. We propose a transformer-based model to tackle the anisotropy problem. It is adaptable to different levels of anisotropy and is computationally efficient. Our model outperforms baseline models in 3D lung cancer segmentation experiments.

Medical image computing has advanced rapidly with the advent of deep learning techniques. Deep convolutional neural networks can perform well given full supervision. However, the success of such fully-supervised models in various image analysis tasks (e.g., anatomy or lesion segmentation from medical images) depends on the availability of massive quantities of labeled data. Given small sample sizes, such models are prohibitively data biased with large domain shifts. To tackle this problem, we propose a novel end - to-end medical image segmentation model, namely Progressive Adversarial Semantic Segmentation (PASS), which can make improved and consistent pixel-wise segmentation predictions without requiring any domain-specific data during training. Our extensive experimentation with 8 public diabetic retinopathy and chest X-ray datasets confirms the effectiveness of PASS in accurate vascular and pulmonary segmentation, both for in-domain and cross-domain evaluations.

An advanced simulation framework has recently been introduced for exploring human perception and visuomotor control. In this context, we investigate locally-connected, irregular deep neural networks (liNets) for biomimetic active vision. Like commonly used CNNs, liNets are locally-connected, forming receptive fields, but unlike CNNs, they are suitable for spatially irregular photoreceptor distributions inspired by those found in foveated biological retinas. Compared to fully-connected deep neural networks, liNets accommodate a much greater number of retinal photoreceptors to enhance visual acuity without intractable memory consumption. LiNets serve well in the biomimetic active vision system embodied in a simulated human that learns active visuomotor control and active appearance-based recognition.

We introduce a framework for simulating a variety of nontrivial, socially motivated behaviors that underlie the orderly passage of pedestrians through doorways, especially the common courtesy of opening and holding doors open for others, an important etiquette that has been overlooked in the literature on autonomous multi-human animation. Emulating such social activity requires serious attention to the interplay of visual perception, navigation in constrained doorway environments, manipulation of a variety of door types, and high-level decision making based on social considerations. To tackle this complex human simulation problem, we take an artificial life approach to modeling autonomous pedestrians, proposing a layered architecture comprising mental, behavioral, and motor layers. The behavioral layer couples two stages: (1) a decentralized, agent-based strategy for dynamically determining the well-mannered ordering of pedestrians around doorways, and (2) a state-based model that directs and coordinates a pedestrian's interactions with the door. The mental layer is a Bayesian network decision model that dynamically selects appropriate door-holding behaviors by considering both internal and external social factors pertinent to pedestrians interacting with one another in and around doorways. Our framework addresses the various door types in common use and supports a variety of doorway etiquette scenarios with efficient, real-time performance.

The unsupervised training of GANs and VAEs has enabled them to generate realistic images mimicking real-world distributions and perform unsupervised clustering or semi-supervised classification of images. Combining the power of these two generative models, we introduce a novel network architecture, Multi-Adversarial Variational autoEncoder Networks (MAVENs), which incorporate an ensemble of discriminators in a combined VAE-GAN network, with simultaneous adversarial learning and variational inference. We apply MAVENs to the generation of synthetic images and propose a new distribution measure to evaluate the quality of the generated images. Our experimental results using the computer vision datasets SVHN and CIFAR-10 demonstrate competitive performance against state-of-the-art semi-supervised models both in image generation and classification tasks.

We present a novel, biomimetic model of the eye for realistic virtual human animation. We also introduce a deep learning approach to oculomotor control that is compatible with our biomechanical eye model. Our eye model consists of the following functional components: (i) submodels of the 6 extraocular muscles that actuate realistic eye movements, (ii) an iris submodel, actuated by pupillary muscles, that accommodates to incoming light intensity, (iii) a corneal submodel and a deformable, ciliary-muscle-actuated lens submodel, which refract incoming light rays for focal accommodation, and (iv) a retina with a multitude of photoreceptors arranged in a biomimetic, foveated distribution. The light intensity captured by the photoreceptors is computed using ray tracing from the photoreceptor positions through the finite aperture pupil into the 3D virtual environment, and the visual information from the retina is output via an optic nerve vector. Our oculomotor control system includes a foveation controller implemented as a locally-connected, irregular Deep Neural Network (DNN), or "LiNet", that conforms to the nonuniform retinal photoreceptor distribution, and a neuromuscular motor controller implemented as a fully-connected DNN, plus auxiliary Shallow Neural Networks (SNNs) that control the accommodation of the pupil and lens. The DNNs are trained offline through deep learning from data synthesized by the eye model itself. Once trained, the oculomotor control system operates robustly and efficiently online. It innervates the intraocular muscles to perform illumination and focal accommodation and the extraocular muscles to produce natural eye movements in order to foveate and pursue moving visual targets. We additionally demonstrate the operation of our eye model (binocularly) within our recently introduced sensorimotor control framework involving an anatomically-accurate biomechanical human musculoskeletal model.

The arrangement of objects into a layout can be challenging for non-experts, as is affirmed by the existence of interior design professionals. Recent research into the automation of this task has yielded methods that can synthesize layouts of objects respecting aesthetic and functional constraints that are non-linear and competing. These methods usually adopt a stochastic optimization scheme, which samples from different layout configurations, a process that is slow and inefficient. We introduce an physics-motivated, continuous layout synthesis technique, which results in a significant gain in speed and is readily scalable. We demonstrate our method on a variety of examples and show that it achieves results similar to conventional layout synthesis based on Markov chain Monte Carlo (McMC) state-search, but is faster by at least an order of magnitude and can handle layouts of unprecedented size as well as tightly-packed layouts that can overwhelm McMC.

Future generations of advanced, autonomous virtual humans will likely require artificial vision systems that more accurately model the human biological vision system. With this in mind, we propose a strongly biomimetic model of visual perception within a novel framework for human sensorimotor control. Our framework features a biomechanically simulated, musculoskeletal human model actuated by numerous skeletal muscles, with two human-like eyes whose retinas have spatially nonuniform distributions of photoreceptors not unlike biological retinas. The retinal photoreceptors capture the scene irradiance that reaches them, which is computed using ray tracing. Within the sensory subsystem of our model, which continuously operates on the photoreceptor outputs, are 10 automatically-trained, deep neural networks (DNNs). A pair of DNNs drive eye and head movements, while the other 8 DNNs extract the sensory information needed to control the arms and legs. Thus, exclusively by means of its egocentric, active visual perception, our biomechanical virtual human learns, by synthesizing its own training data, efficient, online visuomotor control of its eyes, head, and limbs to perform tasks involving the foveation and visual pursuit of target objects coupled with visually-guided reaching actions to intercept the moving targets.

We introduce a biomimetic framework for human sensorimotor control, which features a biomechanically simulated human musculoskeletal model actuated by numerous muscles, with eyes whose retinas have nonuniformly distributed photoreceptors. The virtual human's sensorimotor control system comprises 20 trained deep neural networks (DNNs), half constituting the neuromuscular motor subsystem, while the other half compose the visual sensory subsystem. Directly from the photoreceptor responses, 2 vision DNNs drive eye and head movements, while 8 vision DNNs extract visual information required to direct arm and leg actions. Ten DNNs achieve neuromuscular control---2 DNNs control the 216 neck muscles that actuate the cervicocephalic musculoskeletal complex to produce natural head movements, and 2 DNNs control each limb; i.e., the 29 muscles of each arm and 39 muscles of each leg. By synthesizing its own training data, our virtual human automatically learns efficient, online, active visuomotor control of its eyes, head, and limbs in order to perform nontrivial tasks involving the foveation and visual pursuit of target objects coupled with visually-guided limb-reaching actions to intercept the moving targets, as well as to carry out drawing and writing tasks.

We propose an automatic virtual cinematography method that takes a continuous optimization approach. A suitable camera pose or path is determined automatically by computing the minima of an objective function to obtain some desired parameters, such as those common in live action photography or cinematography. Multiple objective functions can be combined into a single optimizable function, which can be extended to model the smoothness of the optimal camera path using an active contour model. Our virtual cinematography technique can be used to find camera paths in either scripted or unscripted scenes, both with and without smoothing, at a relatively low computational cost.

Exploiting the efficiency and stability of Position-Based Dynamics (PBD), we introduce a novel crowd simulation method that runs at interactive rates for hundreds of thousands of agents. Our method enables the detailed modeling of per-agent behavior in a Lagrangian formulation. We model short-range and long-range collision avoidance to simulate both sparse and dense crowds. On the particles representing agents, we formulate a set of positional constraints that can be readily integrated into a standard PBD solver. We augment the tentative particle motions with planning velocities to determine the preferred velocities of agents, and project the positions onto the constraint manifold to eliminate colliding configurations. The local short-range interaction is represented with collision and frictional contact between agents, as in the discrete simulation of granular materials. We incorporate a cohesion model for modeling collective behaviors and propose a new constraint for dealing with potential future collisions. Our new method is suitable for use in interactive games.

A geometric dissection is a set of pieces which can be assembled in different ways to form distinct shapes. Dissections are used as recreational puzzles because it is striking when a single set of pieces can construct highly different forms. Existing techniques for creating dissections find pieces that reconstruct two input shapes exactly. Unfortunately, these methods only support simple, abstract shapes because an excessive number of pieces may be needed to reconstruct more complex, naturalistic shapes. We introduce a dissection design technique that supports such shapes by requiring that the pieces reconstruct the shapes only approximately. We find that, in most cases, a small number of pieces suffices to tightly approximate the input shapes. We frame the search for a viable dissection as a combinatorial optimization problem, where the goal is to search for the best approximation to the input shapes using a given number of pieces. We find a lower bound on the tightness of the approximation for a partial dissection solution, which allows us to prune the search space and makes the problem tractable. We demonstrate our approach on several challenging examples, showing that it can create dissections between shapes of significantly greater complexity than those supported by previous techniques.

We propose a notion of affordance that takes into account physical quantities generated when the human body interacts with real-world objects, and introduce a learning framework that incorporates the concept of human utilities, which in our opinion provides a deeper and finer-grained account not only of object affordance but also of people's interaction with objects. Rather than defining affordance in terms of the geometric compatibility between body poses and 3D objects, we devise algorithms that employ physicsbased simulation to infer the relevant forces/pressures acting on body parts. By observing the choices people make in videos (particularly in selecting a chair in which to sit) our system learns the comfort intervals of the forces exerted on body parts (while sitting). We account for people's preferences in terms of human utilities, which transcend comfort intervals to account also for meaningful tasks within scenes and spatiotemporal constraints in motion planning, such as for the purposes of robot task planning.

We introduce the Clutterpalette, an interactive tool for detailing indoor scenes with small-scale items. When the user points to a location in the scene, the Clutterpalette suggests detail items for that location. In order to present appropriate suggestions, the Clutterpalette is trained on a dataset of images of real-world scenes, annotated with support relations. Our experiments demonstrate that the adaptive suggestions presented by the Clutterpalette increase modeling speed and enhance the realism of indoor scenes.

Zoomorphic shapes are man-made shapes that possess the form or appearance of an animal. They have desirable aesthetic properties, but are difficult to create using conventional modeling tools. We present a method for creating zoomorphic shapes by merging a man-made shape and an animal shape. To identify a pair of shapes that are suitable for merging, we use an efficient graph kernel based technique. We formulate the merging process as a continuous optimization problem where the two shapes are deformed jointly to minimize an energy function combining several design factors. The modeler can adjust the weighting between these factors to attain high-level control over the final shape produced. A novel technique ensures that the zoomorphic shape does not violate the design restrictions of the man-made shape. We demonstrate the versatility and effectiveness of our approach by generating a wide variety of zoomorphic shapes.

We are developing an ultrasonography training system that promises to accelerate the broader use of ultrasound imaging in healthcare. Aiming at cheaper, more efficient, and more effective ultrasound training, a key feature of our system is the real-time, interactive simulation of a 3D virtual patient that, unlike conventional, purely geometric models of the human body, includes deformable soft tissues. Since soft-tissue deformation is an important factor in the clinical practice of ultrasound imaging, our objective in this paper is to incorporate real-time interactive soft tissue mechanics simulation into our 3D patient model. To this end, we adapt and evaluate two well-known deformable model simulation methods-mass-spring-damper systems and the finite element method-and we apply these methods to the simulation of ultrasound imaging in soft tissues, obtaining promising results on a multicore laptop computer.

We address the challenging problem of controlling a complex biomechanical model of the human body to synthesize realistic swimming animation. Our human model includes all of the relevant articular bones and muscles, including 103 bones (163 articular degrees of freedom) plus a total of 823 muscle actuators embedded in a finite element model of the musculotendinous soft tissues of the body that produces realistic deformations. To coordinate the numerous muscle actuators in order to produce natural swimming movements, we develop a biomimetically motivated motor control system based on Central Pattern Generators (CPGs), which learns to produce activation signals that drive the numerous muscle actuators.

We introduce a framework for outdoor photometric stereo utilizing natural environmental illumination. Our framework extends beyond existing photometric stereo methods intended for laboratory environments to encompass robust outdoor operation in the real world. In this paper, we motivate our framework, describe the components of its processing pipeline, and assess its performance in synthetic experiments as well as in natural experiments including objects in outdoor environments with complex real-world illuminations.

We present an automatic optimization approach to outfit synthesis. Given the hair color, eye color, and skin color of the input body, plus a wardrobe of clothing items, our outfit synthesis system suggests a set of outfits subject to a particular dress code. We introduce a probabilistic framework for modeling and applying dress codes that exploits a Bayesian network trained on example images of real-world outfits. Suitable outfits are then obtained by optimizing a cost function that guides the selection of clothing items to maximize the color compatibility and dress code suitability. We demonstrate our approach on the four most common dress codes:Casual, Sportswear, Business-Casual, andBusiness. A perceptual study validated on multiple resultant outfits demonstrates the efficacy of our framework.

We present a system that automatically synthesizes indoor scenes realistically populated by a variety of furniture objects. Given examples of sensibly furnished indoor scenes, our system extracts, in advance, hierarchical and spatial relationships for various furniture objects, encoding them into priors associated with ergonomic factors, such as visibility and accessibility, which are assembled into a cost function whose optimization yields realistic furniture arrangements. To deal with the prohibitively large search space, the cost function is optimized by simulated annealing using a Metropolis-Hastings state search step. We demonstrate that our system can synthesize multiple realistic furniture arrangements and, through a perceptual study, investigate whether there is a significant difference in the perceived functionality of the automatically synthesized results relative to furniture arrangements produced by human designers.

Summary form only given. The author reviews his research with Faisal Qureshi towards smart camera networks capable of carrying out advanced surveillance tasks with little or no human supervision. A unique centerpiece of our work is the combination of computer vision, computer graphics, and artificial life simulation technologies to develop such networks and experiment with them. Our prototype simulator has enabled us to readily develop and experiment with smart camera networks comprising static and active simulated video surveillance cameras that provide extensive coverage of a large virtual public space, a train station populated by autonomously self animating virtual pedestrians. The simulated networks of smart cameras perform persistent visual surveillance of individual pedestrians with minimal intervention. Our virtual vision simulator has been a potent tool in our quest for innovative camera control strategies that naturally address camera aggregation and handoff, are robust against camera and communication failures, and require no camera calibration, detailed world model, or central controller.

We introduce a comprehensive biomechanical model of the human upper body. Our model confronts the combined challenge of modeling and controlling more or less all of the relevant articular bones and muscles, as well as simulating the physics-based deformations of the soft tissues. Its dynamic skeleton comprises 68 bones with 147 jointed degrees of freedom, including those of each vertebra and most of the ribs. To be properly actuated and controlled, the skeletal submodel requires comparable attention to detail with respect to muscle modeling. We incorporate 814 muscles, each of which is modeled as a piecewise uniaxial Hill-type force actuator. To simulate biomechanically-realistic flesh deformations, we also develop a coupled finite element model with the appropriate constitutive behavior, in which are embedded the detailed 3D anatomical geometries of the hard and soft tissues. Finally, we develop an associated physics-based animation controller that computes the muscle activation signals necessary to drive the elaborate musculoskeletal system in accordance with a sequence of target poses specified by an animator.

We introduce a distributed camera coalition formation scheme for perceptive scene coverage and persistent surveillance by smart camera sensor networks. The proposed model supports task-dependent camera selection and grouping via a "contract net" task allocation protocol augmented with conflict resolution and error recovery mechanisms. Our technique avoids any central controller, and it is robust to node failures and imperfect communication. In the design and empirical evaluation of our camera networks, we exploit a visually and behaviorally realistic virtual environment simulator that is populated by autonomous, lifelike virtual pedestrians.

This paper presents our research towards smart camera networks capable of carrying out advanced surveillance tasks with little or no human supervision. A unique centerpiece of our work is the combination of computer graphics, artificial life, and computer vision simulation technologies to develop such networks and experiment with them. Specifically, we demonstrate a smart camera network comprising static and active simulated video surveillance cameras that provides extensive coverage of a large virtual public space, a train station populated by autonomously self-animating virtual pedestrians. The realistically simulated network of smart cameras performs persistent visual surveillance of individual pedestrians with minimal intervention. Our innovative camera control strategy naturally addresses camera aggregation and handoff, is robust against camera and communication failures, and requires no camera calibration, detailed world model, or central controller.

Spline joints are a novel class of joints that can model general scleronomic constraints for multibody dynamics based on the minimal-coordinates formulation. The main idea is to introduce spline curves and surfaces in the modeling of joints: We model 1-DOF joints using splines on SE(3), and construct multi-DOF joints as the product of exponentials of splines in Euclidean space. We present efficient recursive algorithms to compute the derivatives of the spline joint, as well as geometric algorithms to determine optimal parameters in order to achieve the desired joint motion. Our spline joints can be used to create interesting new simulated mechanisms for computer animation and they can more accurately model complex biomechanical joints such as the knee and shoulder.

This paper advocates a virtual vision paradigm and demonstrates its usefulness in camera sensor network research. Virtual vision prescribes the use of a visually and behaviorally realistic virtual environment simulator in the design and evaluation of surveillance systems. Impediments to deploying and experimenting with appropriately complex camera networks makes virtual vision an attractive alternative for many vision researchers who are motivated to investigate high level multi-camera control issues within such networks. In particular, we present two prototype surveillance systems comprising passive and active pan/tilt/zoom cameras. We deploy these systems in a virtual train station environment populated by autonomous, lifelike virtual pedestrians. The easily reconfigurable virtual cameras situated throughout this environment generate synthetic video feeds that emulate those acquired by real surveillance cameras monitoring extensive public spaces. Our novel multi-camera control strategies enable the cameras to collaborate in persistently observing pedestrians of interest that move across their fields of view and in capturing close-up videos of pedestrians as they travel through designated areas. The sensor networks support task-dependent camera node selection and aggregation through local decision-making and inter-node communication. Our approach to multi-camera control is robust to node failures and message loss.

Numerical multilinear (tensor) algebra is a principled mathematical approach to disentangling and explicitly and parsimoniously representing the essential factors or modes of image formation, among them illumination, scene geometry, and imaging, thereby dramatically improving the performance of appearance-based recognition. Generalizing concepts from linear (matrix) algebra, we define the identity tensor and the pseudo-inverse tensor and we employ them to develop a multilinear projection algorithm, which is natural for performing recognition in the tensor algebraic framework. Our multilinear projection algorithm simultaneously projects an unlabeled test image into multiple constituent mode spaces spanned by learned, mode-specific basis sets in order to infer its mode labels. Multilinear projection is applied to unconstrained facial image recognition, where the mode labels are person identity, viewpoint, illumination, etc.

The virtual vision paradigm features a unique synergy of computer graphics, artificial life, and computer vision technologies. Virtual vision prescribes visually and behaviorally realistic virtual environments as a simulation tool in support of research on large-scale visual sensor networks. Virtual vision has facilitated our research into developing multi-camera control and scheduling algorithms for next-generation smart video surveillance systems.

Unlike the human face, the neck has been largely overlooked in the computer graphics literature, this despite its complex anatomical structure and the important role that it plays in supporting the head in balance while generating the controlled head movements that are essential to so many aspects of human behavior. This paper makes two major contributions. First, we introduce a biomechanical model of the human head-neck system. Emulating the relevant anatomy, our model is characterized by appropriate kinematic redundancy (7 cervical vertebrae coupled by 3-DOF joints) and muscle actuator redundancy (72 neck muscles arranged in 3 muscle layers). This anatomically consistent biomechanical model confronts us with a challenging motor control problem, even for the relatively simple task of balancing the mass of the head in gravity atop the cervical spine. Hence, our second contribution is a novel neuromuscular control model for human head animation that emulates the relevant biological motor control mechanisms. Incorporating low-level reflex and high-level voluntary sub-controllers, our hierarchical controller provides input motor signals to the numerous muscle actuators. In addition to head pose and movement, it controls the tone of mutually opposed neck muscles to regulate the stiffness of the head-neck multibody system. Employing machine learning techniques, the neural networks within our neuromuscular controller are trained offline to efficiently generate the online pose and tone control signals necessary to synthesize a variety of autonomous movements for the behavioral animation of the human head and face.

Expression mapping (also called performance driven animation) has been a popular method for generating facial animations. A shortcoming of this method is that it does not generate expression details such as the wrinkles due to skin deformations. In this paper, we provide a solution to this problem. We have developed a geometry-driven facial expression synthesis system. Given feature point positions (the geometry) of a facial expression, our system automatically synthesizes a corresponding expression image that includes photorealistic and natural looking expression details. Due to the difficulty of point tracking, the number of feature points required by the synthesis system is, in general, more than what is directly available from a performance sequence. We have developed a technique to infer the missing feature point motions from the tracked subset by using an example-based approach. Another application of our system is expression editing where the user drags feature points while the system interactively generates facial expressions with skin deformation details.

Independent components analysis (ICA) maximizes the statistical independence of the representational components of a training image ensemble, but it cannot distinguish between the different factors, or modes, inherent to image formation, including scene structure, illumination, and imaging. We introduce a nonlinear, multifactor model that generalizes ICA. Our multilinear ICA (MICA) model of image ensembles learns the statistically independent components of multiple factors. Whereas ICA employs linear (matrix) algebra, MICA exploits multilinear (tensor) algebra. We furthermore introduce a multilinear projection algorithm which projects an unlabeled test image into the N constituent mode spaces to simultaneously infer its mode labels. In the context of facial image ensembles, where the mode labels are person, viewpoint, illumination, expression, etc., we demonstrate that the statistical regularities learned by MICA capture information that, in conjunction with our multilinear projection algorithm, improves automatic face recognition.

This paper introduces a tensor framework for image-based rendering. In particular, we develop an algorithm called TensorTextures that learns a parsimonious model of the bidirectional texture function (BTF) from observational data. Given an ensemble of images of a textured surface, our nonlinear, generative model explicitly represents the multifactor interaction implicit in the detailed appearance of the surface under varying photometric angles, including local (per-texel) reflectance, complex mesostructural self-occlusion, interreflection and self-shadowing, and other BTF-relevant phenomena. Mathematically, TensorTextures is based on multilinear algebra, the algebra of higher-order tensors, hence its name. It is computed through a decomposition known as the N -mode SVD, an extension to tensors of the conventional matrix singular value decomposition (SVD). We demonstrate the application of TensorTextures to the image-based rendering of natural and synthetic textured surfaces under continuously varying viewpoint and illumination conditions.

Multilinear algebra, the algebra of higher-order tensors, offers a potent mathematical framework for analyzing ensembles of images resulting from the interaction of any number of underlying factors. We present a dimensionality reduction algorithm that enables subspace analysis within the multilinear framework. This N-mode orthogonal iteration algorithm is based on a tensor decomposition known as the N-mode SVD, the natural extension to tensors of the conventional matrix singular value decomposition (SVD). We demonstrate the power of multilinear subspace analysis in the context of facial image ensembles, where the relevant factors include different faces, expressions, viewpoints, and illuminations. In prior work we showed that our multilinear representation, called TensorFaces, yields superior facial recognition rates relative to standard, linear (PCA/eigenfaces) approaches. We demonstrate factor-specific dimensionality reduction of facial image ensembles. For example, we can suppress illumination effects (shadows, highlights) while preserving detailed facial features, yielding a low perceptual error.

We present a framework for composing motor controllers into autonomous composite reactive behaviors for bipedal robots and autonomous, physically-simulated humanoids. A key contribution of our composition framework is an explicit model of the "pre-conditions" under which motor controllers are expected to function properly. Pre-conditions may be determined manually or learned automatically by algorithms based on support vector machine (SVM) learning theory. We demonstrate controller composition and evaluate our composition framework using a family of controllers capable of synthesizing basic actions such a balance, protective stepping when balance is disturbed, protective arm reactions when falling, and multiple ways of regaining an upright stance after a fall.

Article Share on Perceptive agents and systems in virtual reality Author: Demetri Terzopoulos New York University, New York, NY New York University, New York, NYView Profile Authors Info & Claims VRST '03: Proceedings of the ACM symposium on Virtual reality software and technologyOctober 2003Pages 1–3https://doi.org/10.1145/1008653.1008655Published:01 October 2003Publication History 20citation560DownloadsMetricsTotal Citations20Total Downloads560Last 12 Months12Last 6 weeks6 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access

Our goal is to design and build a tool for the creation of expressive character animation. Virtual puppetry, also known as performance animation, is a technique in which the user interactively controls a character's motion. In this paper we introduce local physical models for performance animation and describe how they can augment an existing kinematic method to achieve very effective animation control. These models approximate specific physically-generated aspects of a character's motion. They automate certain behaviours, while still letting the user override such motion via a PD-controller if he so desires. Furthermore, they can be tuned to ignore certain undesirable effects, such as the risk of having a character fall over, by ignoring corresponding components of the force. Although local physical models are a quite simple approximation to real physical behaviour, we show that they are extremely useful for interactive character control, and contribute positively to the expressiveness of the character's motion. In this paper, we develop such models at the knees and ankles of an interactively-animated 3D anthropomorphic character, and demonstrate a resulting animation. This approach can be applied in a straight-forward way to other joints. Categories and Subject Descriptors (according to ACM CCS): I.3.7 [Computer Graphics]: Three-Dimensional Graphics and Realism, Interaction Techniques

The confluence of virtual reality and artificial life, an emerging discipline that spans the computational and biological sciences, has yielded synthetic worlds inhabited by realistic, artificial flora and fauna. Artificial animals are complex synthetic organisms that possess functional biomechanical bodies, sensors, and brains with locomotion, perception, behavior, learning, and cognition centers. Artificial humans and other animals are of interest in computer graphics because they are self-animating characters that dramatically advance the state of the art of production animation and interactive game technologies. More broadly, these biomimetic autonomous agents in their realistic virtual worlds also foster deeper, computationally oriented insights into natural living systems.

We introduce a new paradigm for automatic medical image analysis that adopts concepts from the field of Artificial Life. Our approach prescribes deformable organisms, autonomous agents whose objective is the segmentation and analysis of anatomical structures in medical images. A deformable organism is structured as a ‘muscle’-actuated ‘body’ whose behavior is controlled by a ‘brain’ that is capable of making both reactive and deliberate decisions. This intelligent deformable model possesses an ‘awareness’ of the segmentation process, which emerges from a conflux of perceived sensory data, an internal mental state, memorized knowledge, and a cognitive plan. We develop a class of deformable organisms using a medial representation of body morphology that facilitates a variety of controlled local deformations at multiple spatial scales. Specifically, we demonstrate a deformable ‘worm‘ organism that can overcome noise, incomplete edges, considerable anatomical variation, and occlusion in order to segment and label the corpus callosum in 2D mid-sagittal MR images of the brain. These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

An ambitious goal in the area of physics-based computer animation is the creation of virtual actors that autonomously synthesize realistic human motions and possess a broad repertoire of lifelike motor skills. To this end, the control of dynamic, anthropomorphic figures subject to gravity and contact forces remains a difficult open problem. We propose a framework for composing controllers in order to enhance the motor abilities of such figures. A key contribution of our composition framework is an explicit model of the “pre-conditions” under which motor controllers are expected to function properly. We demonstrate controller composition with pre-conditions determined not only manually, but also automatically based on Support Vector Machine (SVM) learning theory. We evaluate our composition framework using a family of controllers capable of synthesizing basic actions such as balance, protective stepping when balance is disturbed, protective arm reactions when falling, and multiple ways of standing up after a fall. We furthermore demonstrate these basic controllers working in conjunction with more dynamic motor skills within a prototype virtual stunt-person. Our composition framework promises to enable the community of physics-based animation practitioners to easily exchange motor controllers and integrate them into dynamic characters.

Since their debut in 1987, snakes (active contour models) have become a standard image analysis technique with several variants now in common use. We present a portable, reusable software package called "United Snakes". The package unites the most popular snake variants, including finite difference, B-spline, and Hermite polynomial snakes within the mathematical framework of a general finite element formulation with a choice of shape functions. The package furthermore incorporates a recently proposed snake-like technique known as "livewire". We integrate snakes and livewire by introducing an effective method for imposing hard constraints on snakes. Our experiments demonstrate that snakes and livewire have complementary strengths and that their union offers a more powerful tool for interactive image analysis, especially for medical imaging applications. United Snakes is implemented in Java as a JavaBean so that it can easily be integrated in end user application systems.

Deformable surface models are an attractive method for segmenting the three-dimensional shapes of complex anatomic structures in volumetric medical images. Despite the success of this approach, several problems remain. In this paper, we propose a multiscale deformable surface model with non-intersection constraint forces that successfully addresses three significant problems-sensitivity to model initialization, difficulties in dealing with severe object concavities, and model self-intersection. The first two problems are addressed by the multiscale scheme, which progressively resamples the triangulated, deformable surface model both globally and locally, matching its resolution to the levels of a volume image pyramid. We address the third problem by including a non-intersection constraint force among the customary internal and external forces in the physics-based formulation. We apply our new deformable surface model to the challenging task of extracting brain cortical surfaces.

Recent work in behavioral animation has taken impressive steps toward autonomous, self-animating characters for use in production animation and interactive games. It remains difficult, however, to direct autonomous characters to perform specific tasks. This paper addresses the challenge by introducing cognitive modeling. Cognitive models go beyond behavioral models in that they govern what a character knows, how that knowledge is acquired, and how it can be used to plan actions. To help build cognitive models, we develop the cognitive modeling language CML. Using CML, we can imbue a character with domain knowledge, elegantly specified in terms of actions, their preconditions and their effects, and then direct the character’s behavior in terms of goals. Our approach allows behaviors to be specified more naturally and intuitively, more succinctly and at a much higher level of abstraction than would otherwise be possible. With cognitively empowered characters, the animator need only specify a behavior outline or “sketch plan” and, through reasoning, the character will automatically work out a detailed sequence of actions satisfying the specification. We exploit interval methods to integrate sensing into our underlying theoretical framework, thus enabling our autonomous characters to generate action plans even in highly complex, dynamic virtual worlds. We demonstrate cognitive modeling applications in advanced character animation and automated cinematography.

Deformable models, which include deformable contours (the popular snakes) and deformable surfaces, are a powerful model-based medical image analysis technique. We develop a new class of deformable models by formulating deformable surfaces in terms of an affine cell image decomposition (ACID). Our approach significantly extends standard deformable surfaces, while retaining their interactivity and other desirable properties. In particular, the ACID induces an efficient reparameterization mechanism that enables parametric deformable surfaces to evolve into complex geometries, even modifying their topology as necessary. We demonstrate that our new ACID-based deformable surfaces, dubbed T-surfaces, can effectively segment complex anatomic structures from medical volume images.

The numerical simulation of biomechanical models enables the behavioral animation of realistic artificial animals in virtual worlds. Unfortunately, even on high-end graphics workstations, the biomechanical simulation approach is at present computationally too demanding for the animation of numerous animals at interactive frame rates. We tackle this problem by replacing biomechanical animal models with fast kinematic replicas that reproduce the locomotion abilities of the original models with reasonable fidelity. Our technique is based on capturing motion data by systematically simulating the biomechanical models. We refer to it as synthetic motion capture, because of the similarity to natural motion capture applied to real animals. We compile the captured motion data into kinematic action repertoires that are sufficiently rich to support elaborate behavioral animation. Synthetic motion capture in conjunction with level-of-detail geometric modeling and object culling during rendering has enabled us to transform a system designed for the realistic, off-line biomechanical/behavioral animation of artificial fishes into an interactive, stereoscopic, virtual undersea experience.

We develop a vision system for highly mobile autonomous agents that is capable of dynamic obstacle avoidance. We demonstrate the robust performance of the system in artificial animals with directable, foveated eyes, situated in physics-based virtual worlds. Through active perception, each agent controls its eyes and body by continuously analyzing photorealistic binocular retinal image streams. The vision system computes stereo disparity and segments looming targets in the low-resolution visual periphery while controlling eye movements to track an object fixated in the high-resolution fovea. It matches segmented targets against mental models of colored objects of interest in order to decide whether the segmented objects are harmless or represent dangerous obstacles. The latter are localized enabling the artificial animal to exercise the sensorimotor control necessary to avoid collision.

Muscle based face image synthesis is one of the most realistic approaches to realizing life-like agents in a computer. A facial muscle model is composed of facial tissue elements and muscles. In this model, forces are calculated effecting facial tissue elements by contraction of each muscle strength, so the combination of each muscle parameter decides a specific facial expression. Each muscle parameter is decided based on a trial and error procedure comparing the sample photograph and generated image using our Muscle-Editor to generate a specific face image. We propose a strategy of automatic estimation of facial muscle parameters from 2D marker movements using a neural network. We can also carry out 3D motion estimation from 2D point or flow information in a captured image under restriction of a physics based face model.

Animation through the numerical simulation of physics- based graphics models offers unsurpassed realism, but it can be computationally demanding. Likewise, the search for controllers that enable physics-based models to produce desired animations usually entails formidable computational cost. This paper demon- strates the possibility of replacing the numerical simulation and control of dynamic models with a dramatically more efficient al- ternative. In particular, we propose the NeuroAnimator, a novel ap- proach to creating physically realistic animation that exploits neu- ral networks. NeuroAnimators are automatically trained off-line to emulate physical dynamics through the observation of physics- based models in action. Depending on the model, its neural net- work emulator can yield physically realistic animation one or two orders of magnitude faster than conventional numerical simulation. Furthermore, by exploiting the network structure of the NeuroAni- mator, we introduce a fast algorithm for learning controllers that en- ables either physics-based models or their neural network emulators to synthesize motions satisfying prescribed animation goals. We demonstrate NeuroAnimators for a variety of physics-based mod- els.

We develop a simple and very fast method for object tracking based exclusively on color information in digitized video images. Running on a Silicon Graphics R4600 Indy system with an IndyCam, our algorithm is capable of simultaneously tracking objects at full frame size (640/spl times/480 pixels) and video frame rate (30 fps). Robustness with respect to occlusion is achieved via can explicit hypothesis-tree model of the occlusion process. We demonstrate the efficacy of our technique in the challenging task of tracking people, especially tracking human heads and hands.

Panel OverviewFacial animation is now attracting more attention than ever before in its 25 years as an identifiable area of computer graphics.Imaginative applications of animated graphical faces are found in sophisticated human-computer interfaces, interactive games, multimedia titles, VR telepresence experiences, and, as always, in a broad variety of production animations.Graphics technologies underlying facial animation now run the gamut from keyframing to image morphing, video tracking, geometric and physical modeling, and behavioral animation.Supporting technologies include speech synthesis and artificial intelligence.Whether the goal is to synthesize realistic faces or fantastic ones, representing the dynamic facial likeness of humans and other creatures is giving impetus to a diverse and rapidly growing body of cross-disciplinary research.The panel will present a historical perspective, assess the state of the art, and speculate on the exciting future of facial animation.

Free form deformations (FFDs) are a popular tool for modeling and keyframe animation. The paper extends the use of FFDs to a dynamic setting. Our goal is to enable normally inanimate graphics objects, such as teapots and tables, to become animated, and learn to move about in a charming, cartoon like manner. To achieve this goal, we implement a system that can transform a wide class of objects into dynamic characters. Our formulation is based on parameterized hierarchical FFDs augmented with Lagrangian dynamics, and provides an efficient way to animate and control the simulated characters. Objects are assigned mass distributions and elastic deformation properties, which allow them to translate, rotate, and deform according to internal and external forces. In addition, we implement an automated optimization process that searches for suitable control strategies. The primary contributions of the work are threefold. First, we formulate a dynamic generalization of conventional, geometric FFDs. The formulation employs deformation modes which are tailored by the user and are expressed in terms of FFDs. Second, the formulation accommodates a hierarchy of dynamic FFDs that can be used to model local as well as global deformations. Third, the deformation modes can be active, thereby producing locomotion.

Presents dynamic non-uniform rational B-splines (D-NURBS), a physics-based generalization of NURBS. NURBS have become a de facto standard in commercial modeling systems. Traditionally, however, NURBS have been viewed as purely geometric primitives, which require the designer to interactively adjust many degrees of freedom-control points and associated weights-to achieve the desired shapes. The conventional shape modification process can often be clumsy and laborious. D-NURBS are physics-based models that incorporate physical quantities into the NURBS geometric substrate. Their dynamic behavior, resulting from the numerical integration of a set of nonlinear differential equations, produces physically meaningful, and hence intuitive shape variation. Consequently, a modeler can interactively sculpt complex shapes to required specifications not only in the traditional indirect fashion, by adjusting control points and setting weights, but also through direct physical manipulation, by applying simulated forces and local and global shape constraints. We use Lagrangian mechanics to formulate the equations of motion for D-NURBS curves, tensor-product D-NURBS surfaces, swung D-NURBS surfaces and triangular D-NURBS surfaces. We apply finite element analysis to reduce these equations to efficient numerical algorithms computable at interactive rates on common graphics workstations. We implement a prototype modeling environment based on D-NURBS and demonstrate that D-NURBS can be effective tools in a wide range of computer-aided geometric design (CAGD) applications.

The paper presents a typologically adaptable snakes model for image segmentation and object representation. The model is embedded in the framework of domain subdivision using simplicial decomposition. This framework extends the geometric and topological adaptability of snakes while retaining all of the features of traditional snakes, such as user interaction, and overcoming many of the limitations of traditional snakes. By superposing a simplicial grid over the image domain and using this grid to iteratively reparameterize the deforming snakes model, the model is able to flow into complex shapes, even shapes with significant protrusions or branches, and to dynamically change topology as necessitated by the data. Snakes can be created and can split into multiple parts or seamlessly merge into other snakes. The model can also be easily converted to and from the traditional parametric snakes model representation. We apply a 2D model to various synthetic and real images in order to segment objects with complicated shapes and topologies.>

We propose and demonstrate a new paradigm for active vision research that draws upon recent advances in the fields of artificial life and computer graphics. A software alternative to the prevailing hardware vision mindset, animat vision prescribes artificial animals, or animats, situated in physics-based virtual worlds as autonomous virtual robots possessing active perception systems. To be operative in its world, an animat must autonomously control its eyes and muscle-actuated body, applying computer vision algorithms to continuously analyze the retinal image streams acquired by its eyes in order to locomote purposefully through its world. We describe an initial animat vision implementation within lifelike artificial fishes inhabiting a physics-based, virtual marine world. Emulating the appearance, motion, and behavior of real fishes in their natural habitats, these animats are capable of spatially nonuniform retinal imaging, foveation, retinal image stabilization, color object recognition, and perceptually-guided navigation. These capabilities allow them to pursue moving targets such as fellow artificial fishes. Animat vision offers a fertile approach to the development, implementation, and evaluation of computational theories that profess sensorimotor competence for animal or robotic situated agents.>

We present a learning technique that automatically synthesizes realistic locomotion for the animation of physics-based models of animals. The method is especially suitable for animals with highly flexible, many-degree-of-freedom bodies and a considerable number of internal muscle actuators, such as snakes and fish. The multilevel learning process first performs repeated locomotion trials in search of actuator control functions that produce efficient locomotion, presuming virtually nothing about the form of these functions. Applying a short-time Fourier analysis, the learning process then abstracts control functions that produce effective locomotion into a compact representation which makes explicit the natural quasi-periodicities and coordination of the muscle actions. The artificial animals can finally put into practice the compact, efficient controllers that they have learned. Their locomotion learning abilities enable them to accomplish higher-level tasks specified by the animator while guided by sensory perception of their virtual world; e.g., locomotion to a visible target. We demonstrate physics-based animation of learned locomotion in dynamic models of land snakes, fishes, and even marine mammals that have trained themselves to perform "SeaWorld" stunts.

A major unsolved problem in computer graphics is the construction and animation of realistic human facial models. Traditionally, facial models have been built painstakingly by manual digitization and animated by ad hoc parametrically controlled facial mesh deformations or kinematic approximation of muscle actions. Fortunately, animators are now able to digitize facial geometries through the use of scanning range sensors and animate them through the dynamic simulation of facial tissues and muscles. However, these techniques require considerableuser input to construct facial models of individuals suitable for animation. In this paper, we present a methodology for automating this challenging task. Starting with a structured facial mesh, we develop algorithms that automatically construct functional models of the heads of human subjects from laser-scanned range and reflectance data. These algorithms automatically insert contractile muscles at anatomically correct positions within a dynamic skin model and root them in an estimated skull structure with a hinged jaw. They also synthesize functional eyes, eyelids, teeth, and a neck and fit them to the final model. The constructed face may be animated via muscle actuations. In this way, we create the most authentic and functional facial models of individuals available to date and demonstrate their use in facial animation.

This paper describes research on the ARK (Autonomous Mobile Robot in a Known Environment) project. The technical objective of the project is to build a robot that can navigate and carry out survey/inspection tasks in a complex but known industrial environment. Rather than altering the robots environment by adding easily identifiable beacons the robot relies on naturally occurring objects to use as visual landmarks for navigation. The robot is equipped with various sensors that are used to detect unmapped obstacles, landmarks and objects. This paper describes the robot's industrial environment, it's control architecture, and some results in processing the robot's range and vision sensor data for navigation.>

This paper proposes a framework for animation that can achieve the intricacy of motion evident in certain natural ecosystems with minimal input from the animator. The realistic appearance, movement, and behavior of individual animals, as well as the patterns of behavior evident in groups of animals fall within the scope of the framework. Our approach to emulating this level of natural complexity is to model each animal holistically as an autonomous agent situated in its physical world. To demonstrate the approach, we develop a physics-based, virtual marine world. The world is inhabited by artificial fishes that can swim hydrodynamically in simulated water through the motor control of internal muscles that motivates fins. Their repertoire of behaviors relies on their perception of the dynamic environment. As in nature, the detailed motions of artificial fishes in their virtual habitat are not entirely predictable because they are not scripted.

Neuroscientists have studied the relationship between nerve cell morphology and function for over a century. To pursue these studies, they need accurate three-dimensional models of nerve cells that facilitate detailed anatomical measurement and the identification of internal structures. Although serial transmission electron microscopy has been a source of such models since the mid 1960s, model reconstruction and analysis remain very time consuming. The authors have developed a new approach to reconstructing and visualizing 3D nerve cell models from serial microscopy. An interactive system exploits recent computer graphics and computer vision techniques to significantly reduce the time required to build such models. The key ingredients of the system are a digital "blink comparator" for section registration, "snakes," or active deformable contours, for semiautomated cell segmentation, and voxel-based techniques for 3D reconstruction and visualization of complex cell volumes with internal structures.

This article develops a dynamic generalization of the nonuniform rational B-spline (NURBS) model. NURBS have become a defacto standard in commercial modeling systems because of their power to represent free-form shapes as well as common analytic shapes. To date, however, they have been viewed as purely geometric primitives that require the user to manually adjust multiple control points and associated weights in order to design shapes. Dynamic NURBS, or D-NURBS, are physics-based models that incorporate mass distributions, internal deformation energies, and other physical quantities into the popular NURBS geometric substrate. Using D-NURBS, a modeler can interactively sculpt curves and surfaces and design complex shapes to required specifications not only in the traditional indirect fashion, by adjusting control points and weights, but also through direct physical manipulation, by applying simulated forces and local and global shape constraints. D-NURBS move and deform in a physically intuitive manner in response to the user's direct manipulations. Their dynamic behavior results from the numerical integration of a set of nonlinear differential equations that automatically evolve the control points and weights in response to the applied forces and constraints. To derive these equations, we employ Lagrangian mechanics and a finite-element-like discretization. Our approach supports the trimming of D-NURBS surfaces using D-NURBS curves. We demonstrate D-NURBS models and constraints in applications including the rounding of solids, optimal surface fitting to unstructured data, surface design from cross sections, and free-form deformation. We also introduce a new technique for 2D shape metamorphosis using constrained D-NURBS surfaces.

A new approach to surface modeling and reconstruction is developed which overcomes some important limitations of existing surface representations methods. The approach features two components. The first is a dynamic self-organizing oriented particle system which discovers topological and geometric surface structure implicit in visual data. The oriented particles evolve according to Newtonian mechanics and interact through long-range attraction forces, short-range repulsion forces, and coplanarity, conormality, and cocircularity forces. The second component is an efficient triangulation scheme that connects the particles into a continuous global surface model that is consistent with the inferred structure. A flexible surface reconstruction algorithm is developed that can compute complete, detailed, viewpoint-invariant geometric surface descriptions of objects with arbitrary topology. The algorithms are applied to 3-D medical image segmentation and to surface reconstruction from object silhouettes.>

The authors present a physics-based approach for recovering the 3-D shape and tracking the motion of nonrigid objects using a 3-D elastically deformable balloon model. The balloon model is based on a thin-plate under tension spline which deforms to fit visual data according to internal forces stemming from the elastic properties of the surface and external forces which are produced from the data. The finite element method is used to represent the model as a continuous surface. A natural finite element is used whose nodal variables comprise the position of the surface plus its first and second partial derivatives, reflecting each of the partial derivatives that occur in the spline's strain energy functional. Hence, the model directly estimates all the information needed to measure the differential geometric properties of the fitted surface. The balloon model was applied to the reconstruction of 3-D objects with irregular shape features. Its effectiveness is demonstrated in extracting the left ventricular surface and tracking its nonrigid motion in dynamic computerized tomography volume images.>

A physics-based framework for 3-D shape and nonrigid motion estimation for real-time computer vision systems is presented. The framework features dynamic models that incorporate the mechanical principles of rigid and nonrigid bodies into conventional geometric primitives. Through the efficient numerical simulation of Lagrange equations of motion, the models can synthesize physically correct behaviors in response to applied forces and imposed constraints. Applying continuous Kalman filtering theory, a recursive shape and motion estimator that employs the Lagrange equations as a system model is developed. The system model continually synthesizes nonrigid motion in response to generalized forces that arise from the inconsistency between the incoming observations and the estimated model state. The observation forces also account formally for instantaneous uncertainties and incomplete information. A Riccati procedure updates a covariance matrix that transforms the forces in accordance with the system dynamics and prior observation history. Experiments involving model fitting and tracking of articulated and flexible objects from noisy 3-D data are described.>

An approach to the analysis of dynamic facial images for the purposes of estimating and resynthesizing dynamic facial expressions is presented. The approach exploits a sophisticated generative model of the human face originally developed for realistic facial animation. The face model which may be simulated and rendered at interactive rates on a graphics workstation, incorporates a physics-based synthetic facial tissue and a set of anatomically motivated facial muscle actuators. The estimation of dynamical facial muscle contractions from video sequences of expressive human faces is considered. An estimation technique that uses deformable contour models (snakes) to track the nonrigid motions of facial features in video images is developed. The technique estimates muscle actuator controls with sufficient accuracy to permit the face model to resynthesize transient expressions.>

The adaptive mesh model is extended in several ways. Open adaptive meshes and closed adaptive shells based on triangular and rectangular elements are developed. A discontinuity detection and preservation algorithm suitable for the model is proposed. Techniques for adaptive hierarchical subdivision of adaptive meshes and shells are also developed. The extended model is applied to image and 3D surface data.>

This paper develops a systematic approach to deriving dynamic models from pammetrically dejined solid primitives, g!obal geometric deformations and local jiniteelement deformations.Even though their kinematics is styl- ized by the particular solid primitive used, the models behave in a physically correct way with prescribed mass distributions and elasticities.We also propose eficient constraint methods jor connecting these new dynamic primitives together to make articulated models.Our techniques make it possible to build and animate constrained, nonrigid, unibody or multibody objects in simulatedphysical environments at interactive rates.

A physically based approach to the recovery of nonrigid 3-D motion and the tracking of nonrigid objects is presented. The approach makes use of deformable superquadrics, dynamics models that offer global deformation parameters which capture large-scale features and local deformation parameters which capture the details of complex shapes. The equations of motion governing the behavior of the models make them responsive to externally applied forces. The authors extend their prior formulation of these equations to include globally parameterized tapering and bending deformations. They further generalize the formulation to handle physically based point-to-point constraints between models. Such constraints enable one to automatically assemble object models from interconnected deformable superquadric parts. These composite models may be used to track the motions of articulated, flexible objects.>

An approach to visual sampling and reconstruction motivated by concepts from numerical grid generation is presented. Adaptive meshes that can nonuniformly sample and reconstruct intensity and range data are presented. These meshes are dynamic models which are assembled by interconnecting nodal masses with adjustable springs. Acting as mobile sampling sites, the nodes observe properties of the input data, such as intensities, depths, gradients, and curvatures. Based on these nodal observations, the springs automatically adjust their stiffnesses so as to distribute the available degrees of freedom of the reconstructed model in accordance with the local complexity of the input data. The adaptive mesh algorithm runs at interactive rates with continuous 3-D display on a graphics workstation It is applied to the adaptive sampling and reconstruction of images and surfaces.>

Abstract We develop physically‐based graphics models of non‐rigid objects capable of heat conduction, thermoelasticity, melting and fluid‐like behaviour in the molten state. These deformable models feature non‐rigid dynamics governed by Lagrangian equations of motion and conductive heat transfer governed by the heat equation for non‐homogeneous, non‐isotropic media. In its solid state, the discretized model is an assembly of hexahedral finite elements in which thermoelastic units interconnect particles situated in a lattice. The stiffness of a thermoelastic unit decreases as its temperature increases, and the unit fuses when its temperature exceeds the melting point. The molten state of the model involves a molecular dynamics simulation in which ‘fluid’ particles that have broken free from the lattice interact through long‐range attraction forces and short‐range repulsion forces. We present a physically‐based animation of a thermoelastic model in a simulated physical world populated by hot constraint surfaces.

Abstract This paper extends a physically‐based approach to the realistic modelling and animation of human heads using scanned data. A three‐step procedure is proposed which begins with geometry and colour data acquired from a subject's head using a radial laser scanner, continues with the construction of a 3D model of the head with complete colour texture mapping, and culminates in real‐time, interactive facial animation. First, an adaptive meshing technique is described that creates non‐uniform facial meshes from the densely sampled data. These geometrically accurate facial meshes adapt to features of interest in the data, increasing polygon density over articulate face regions and areas of high curvature. Secondly, the mesh is transformed into a physically‐based head model of the subject. This incorporates a physical approximation to facial tissue and a set of anatomically‐motivated facial muscle actuators. Finally, these components are integrated in a facial modelling and animation system.

The authors present a physically based approach to fitting complex three-dimensional shapes using a novel class of dynamic models that can deform both locally and globally. They formulate the deformable superquadrics which incorporate the global shape parameters of a conventional superellipsoid with the local degrees of freedom of a spline. The model's six global deformational degrees of freedom capture gross shape features from visual data and provide salient part descriptors for efficient indexing into a database of stored models. The local deformation parameters reconstruct the details of complex shapes that the global abstraction misses. The equations of motion which govern the behavior of deformable superquadrics make them responsive to externally applied forces. The authors fit models to visual data by transforming the data into forces and simulating the equations of motion through time to adjust the translational, rotational, and deformational degrees of freedom of the models. Model fitting experiments involving 2D monocular image data and 3D range data are presented.>

A physically-based approach is presented to fitting complex 3D shapes using a novel class of dynamic models. These models can deform both locally and globally. The authors formulate deformable superquadrics which incorporate the global shape parameters of a conventional superellipsoid with the local degrees of freedom of a spline. The local/global representational power of a deformable superquadric simultaneously satisfies the conflicting requirements of shape reconstruction and shape recognition. The model's six global deformational degrees of freedom capture gross shape features from visual data and provide salient part descriptors for efficient indexing into a database of stored models. Model fitting experiments involving 2D monocular image data and 3D range data are reported.>

A novel approach is presented to the analysis of dynamic facial images. The approach exploits a realistic model of the human face. The face model, which may be simulated and rendered at interactive rates on a graphics workstation, incorporates a physically-based approximation to facial tissue and a set of anatomically-motivated facial muscle actuators. The authors consider the estimation of dynamic facial muscle contractions from video sequences of expressive human faces. They develop an estimation technique that uses deformable contour models to track the nonrigid motions of facial features in images. The technique computes robust muscle actuator controls, enabling the face model to resynthesize transient expressions accurately.>

Abstract We develop a new 3D hierarchical model of the human face. The model incorporates a physically‐based approximation to facial tissue and a set of anatomically‐motivated facial muscle actuators. Despite its sophistication, the model is efficient enough to produce facial animation at interactive rates on a high‐end graphics workstation. A second contribution of this paper is a technique for estimating muscle contractions from video sequences of human faces performing expressive articulations. These estimates may be input as dynamic control parameters to the face model in order to produce realistic animation. Using an example, we demonstrate that our technique yields sufficiently accurate muscle contraction estimates for the model to reconstruct expressions from dynamic images of faces.

Deterministic splines and stochastic fractals are complementary techniques for generating free-form shapes. Splines are easily constrained and well suited to modeling smooth, man-made objects. Fractals, while difficult to constrain, are suitable for generating various irregular shapes found in nature. This paper develops constrained fractals, a hybrid of splines and fractals which intimately combines their complementary features. This novel shape synthesis technique stems from a formal connection between fractals and generalized energy-minimizing splines which may be derived through Fourier analysis. A physical interpretation of constrained fractal generation is to drive a spline subject to constraints with modulated white noise, letting the spline diffuse the noise into the desired fractal spectrum as it settles into equilibrium. We use constrained fractals to synthesize realistic terrain models from sparse elevation data.

We continue our development of physically-based models for animating nonrigid objects in simulated physical environments. Our prior work treats the special case of objects that undergo perfectly elastic deformations. Real materials, however, exhibit a rich variety of inelastic phenomena. For instance, objects may restore themselves to their natural shapes slowly, or perhaps only partially upon removal of forces that cause deformation. Moreover, the deformation may depend on the history of applied forces. The present paper proposes inelastically deformable models for use in computer graphics animation. These dynamic models tractably simulate three canonical inelastic behaviors---viscoelasticity, plasticity, and fracture. Viscous and plastic processes within the models evolve a reference component, which describes the natural shape, according to yield and creep relationships that depend on applied force and/or instantaneous deformation. Simple fracture mechanics result from internal processes that introduce local discontinuities as a function of the instantaneous deformations measured through the model. We apply our inelastically deformable models to achieve novel computer graphics effects.

A computational theory of visible-surface representations is developed. The visible-surface reconstruction process that computes these quantitative representations unifies formal solutions to the key problems of: (1) integrating multiscale constraints on surface depth and orientation from multiple-visual sources; (2) interpolating dense, piecewise-smooth surfaces from these constraints; (3) detecting surface depth and orientation discontinuities to apply boundary conditions on interpolation; and (4) structuring large-scale, distributed-surface representations to achieve computational efficiency. Visible-surface reconstruction is an inverse problem. A well-posed variational formulation results from the use of a controlled-continuity surface model. Discontinuity detection amounts to the identification of this generic model's distributed parameters from the data. Finite-element shape primitives yield a local discretization of the variational principle. The result is an efficient algorithm for visible-surface reconstruction.>

The theory of elasticity describes deformable materials such as rubber, cloth, paper, and flexible metals. We employ elasticity theory to construct differential equations that model the behavior of non-rigid curves, surfaces, and solids as a function of time. Elastically deformable models are active: they respond in a natural way to applied forces, constraints, ambient media, and impenetrable obstacles. The models are fundamentally dynamic and realistic animation is created by numerically solving their underlying differential equations. Thus, the description of shape and the description of motion are unified.

Image analysis problems, posed mathematically as variational principles or as partial differential equations, are amenable to numerical solution by relaxation algorithms that are local, iterative, and often parallel. Although they are well suited structurally for implementation on massively parallel, locally interconnected computational architectures, such distributed algorithms are seriously handi capped by an inherent inefficiency at propagating constraints between widely separated processing elements. Hence, they converge extremely slowly when confronted by the large representations of early vision. Application of multigrid methods can overcome this drawback, as we showed in previous work on 3-D surface reconstruction. In this paper, we develop multiresolution iterative algorithms for computing lightness, shape-from-shading, and optical flow, and we examine the efficiency of these algorithms using synthetic image inputs. The multigrid methodology that we describe is broadly applicable in early vision. Notably, it is an appealing strategy to use in conjunction with regularization analysis for the efficient solution of a wide range of ill-posed image analysis problems.

Inverse problems, such as the reconstruction problems that arise in early vision, tend to be mathematically ill-posed. Through regularization, they may be reformulated as well-posed variational principles whose solutions are computable. Standard regularization theory employs quadratic stabilizing functionals that impose global smoothness constraints on possible solutions. Discontinuities present serious difficulties to standard regularization, however, since their reconstruction requires a precise spatial control over the smoothing properties of stabilizers. This paper proposes a general class of controlled-continuity stabilizers which provide the necessary control over smoothness. These nonquadratic stabilizing functionals comprise multiple generalized spline kernels combined with (noncontinuous) continuity control functions. In the context of computational vision, they may be thought of as controlled-continuity constraints. These generic constraints are applicable to visual reconstruction problems that involve both continuous regions and discontinuities, for which global smoothness constraints fail.