This paper explores the use of a deformable mesh (also known as the control grid) structure for motion analysis and synthesis in an image sequence. We focus on the synthesis problem, i.e., how to interpolate an image function given nodal positions and values and how to predict a present image frame from a reference one given nodal displacements between the two images. For this purpose, we review the fundamental theory and numerical techniques that have been developed in the finite element method for function approximation and mapping using a mesh structure. Specifically, we focus on (i) the use of shape functions for node-based function interpolation and mapping; and (ii) the use of regular master elements to simplify numerical calculations involved in dealing with irregular mesh structures. In addition to a general introduction that is applicable to an arbitrary mesh structure, we also present specific results for triangular and quadrilateral mesh structures, which are the most useful two-dimensional (2-D) meshes. Finally, we describe how to apply the above results for motion compensated frame prediction and interpolation. It is shown that the concepts of shape functions and master elements are crucial for developing computationally efficient algorithms for both the analysis and synthesis problems.

For pt.I see ibid., vol.6, no.6, p.636-46 (1996). This paper explores the use of the deformable mesh structure for motion/shape analysis and synthesis in an image sequence. We present algorithms for the analysis problem, including scene-adaptive mesh generation and node tracking over successive frames. We also describe a region-based video coder that integrates the analysis and synthesis algorithms presented. The coder describes each region by an ensemble of connected quadrilateral elements embedded in a mesh structure. For each region, its shape and texture are described by the nodal positions and image functions of the elements in this region in an initial frame, while its motion (including shape deformation) is characterized by the nodal trajectories in the following frames, which are in turn specified by a few motion parameters. This coder has been applied to a typical common intermediate format (CIF) resolution, head-and-shoulder type sequence. The visual quality is significantly better than the H.263-TMN4 algorithm at about 50 kb/s (for the luminance component only, 30 Hz).

An active mesh representation of motion has been developed by the authors. It describes the motion in an image sequence by the trajectories of a set of nodes that form a mesh (also known as a control grid). In this paper we investigate how to cluster the nodes, or equivalently the mesh elements, so that the nodes in the same group have similar global motions. We present and compare two approaches. The parallel approach first finds a set of global motion parameters for each element and then partitions these parameters using the K-means clustering algorithm; The sequential (or layered) approach recursively extracts the nodes with the dominant motion from all the nodes, using a modified robust regression algorithm.

One well-known deficiency of the block matching algorithm (BMA) is that it cannot handle nontranslational motion within a block. This paper proposes a nodal-displacement-based deformation model. It assumes that a selected number of control nodes in a block can move freely, and that the displacement of any interior point can be interpolated from nodal displacements. This model includes the translational model as a special case with a single node and can characterize increasingly more complex deformation by the use of more nodes. Of particular importance is the four-node-based rectangle to quadrangle mapping, which covers essentially all possible 2D deformations caused by 3D motions over a reasonably small block. A deformable block matching algorithm (DBMA) is developed for the estimation of nodal displacements. It is a gradient-based search algorithm and each iteration requires the solution of a simple linear equation. A hybrid three-mode method is further proposed, which chooses among the BMA and DBMA, both without error correction, and the BMA with error correction using a DCT coding method (BMA-DCT). The DBMA and the three-mode method have been simulated for the case of the four-node-based mapping. In comparison with the BMA-DCT, the reduction of prediction errors by the DBMA outweighs the increase in required bits for specifying motion. At similar bit rates, the DBMA has yielded better image quality than the BMA-DCT. On the other hand, the three-mode method can significantly reduce the bit rate while maintaining a similar quality, at the expense of a slight increase in the computational cost.

This paper introduces a representation scheme for image sequences using nonuniform samples embedded in a deformable mesh structure. It describes a sequence by nodal positions and colors in a starting frame, followed by nodal displacements in the following frames. The nodal points in the mesh are more densely distributed in regions containing interesting features such as edges and corners; and are dynamically updated to follow the same features in successive frames. They are determined automatically by maximizing feature (e.g., gradient) magnitudes at nodal points, while minimizing interpolation errors within individual elements, and matching errors between corresponding elements. In order to avoid the mesh elements becoming overly deformed, a penalty term is also incorporated, which measures the irregularity of the mesh structure. The notions of shape functions and master elements commonly used in the finite element method have been applied to simplify the numerical calculation of the energy functions and their gradients. The proposed representation is motivated by the active contour or snake model proposed by Kass, Witkin, and Terzopoulos (1988). The current representation retains the salient merit of the original model as a feature tracker based on local and collective information, while facilitating more accurate image interpolation and prediction. Our computer simulations have shown that the proposed scheme can successfully track facial feature movements in head-and-shoulder type of sequences, and more generally, interframe changes that can be modeled as elastic deformation. The treatment for the starting frame also constitutes an efficient representation of arbitrary still images.

This paper introduces a representation scheme for images and video sequences using nonuniform samples embedded in a mesh structure. It describes a video sequence by the nodal positions and colors in a starting frame, followed by the nodal displacements in the following frames. The nodal points are more densely distributed in regions containing interesting features such as edges and corners, and are dynamically updated to follow the same features in successive frames. They are determined automatically by maximizing feature (e.g., gradient) magnitudes at nodal points, while minimizing interpolation errors within individual elements, and matching errors between corresponding elements. In order to avoid the mesh elements becoming overly deformed, a penalty term is also incorporated which measures the irregularity of the mesh structure. The notions of shape functions and master elements commonly used in the finite element method have been employed to simplify the numerical calculation of the energy functions and their gradients. The proposed representation is motivated by the active contour or snake model proposed by Kass, Witkin, and Terzopoulos. The current representation retains the salient merit of the original model as a feature tracker based on local and collective information, while facilitating more accurate image interpolation and prediction.