By adhering to the dictum, “No causation without manipulation (treatment, intervention)”, cause and effect data analysis represents changes in observed data in terms of changes in the causal factors. When causal factors are not amenable for active manipulation in the real world due to current technological limitations or ethical considerations, a counterfactual approach performs an intervention on the model of data formation. In the case of object representation or activity (temporal object) representation, varying object parts is generally unfeasible whether they be spatial and/or temporal. Multilinear algebra, the algebra of higher order tensors, is a suitable and transparent framework for disentangling the causal factors of data formation. Learning a part-based intrinsic causal factor representations in a multilinear framework requires applying a set of interventions on a part-based multilinear model. We propose a unified multilinear model of wholes and parts. We derive a hierarchical block multilinear factorization, the M-mode Block SVD, that computes a disentangled representation of the causal factors by optimizing simultaneously across the entire object hierarchy. Given computational efficiency considerations, we introduce an incremental bottom-up computational alternative, the Incremental M-mode Block SVD, that employs the lower level abstractions, the part representations, to represent the higher level of abstractions, the parent wholes. This incremental computational approach may also be employed to update the causal model parameters when data becomes available incrementally. The resulting object representation is an interpretable combinatorial choice of intrinsic causal factor representations related to an object's recursive hierarchy of wholes and parts that renders object recognition robust to occlusion and reduces training data requirements.

This paper introduces a new multilinear projection algorithm for appearance-based recognition in a tensor framework. The multilinear projection simultaneously maps an unlabeled image from the pixel space into multiple causal factors underlying image formation, including illumination, imaging, and scene structure. For facial recognition, the most relevant aspect of scene structure is the specific person whose face has been imaged. Our new multilinear projection algorithm, which is based on the canonical decomposition of tensors, is superior to our previously proposed multilinear projection algorithm that is based on an M-mode SVD. To develop our new algorithm, we extend and formalize the definition of the mode-m product, the modem identity tensor, and the mode-m pseudo-inverse tensor. We demonstrate our multilinear projection in the context of facial image recognition and compare its results in simultaneously inferring the identity, view, illumination, etc., coefficient vectors of an unlabeled test image against those obtained using multilinear projection based on the M-mode SVD, as well as against the results obtained using a set of multiple linear projections. Finally, we present a strategy for developing a practical biometric system that can enroll an uncooperative subject using a one or more images and then recognize that subject in unconstrained test images.

This paper introduces a multilinear (tensor) framework for the analysis and synthesis of the head-related transfer function (HRTF). The HRTF is the result of the confluence of two factors, sound location and person (anatomy). Our multilinear modeling technique employs a tensor extension of the conventional matrix singular value decomposition (SVD), known as the TV-mode SVD which explicitly represents the HRTF in terms of its constituent factors. Anatomical data is mapped to our multilinear HRTF model using regression. This mapping defines a data-driven model capable of producing different personalized HRTFs from easily obtained anatomical measurements. We show that our approach yields objectively superior results to those of a mapping based on principle components analysis (PCA).

Numerical multilinear (tensor) algebra is a principled mathematical approach to disentangling and explicitly and parsimoniously representing the essential factors or modes of image formation, among them illumination, scene geometry, and imaging, thereby dramatically improving the performance of appearance-based recognition. Generalizing concepts from linear (matrix) algebra, we define the identity tensor and the pseudo-inverse tensor and we employ them to develop a multilinear projection algorithm, which is natural for performing recognition in the tensor algebraic framework. Our multilinear projection algorithm simultaneously projects an unlabeled test image into multiple constituent mode spaces spanned by learned, mode-specific basis sets in order to infer its mode labels. Multilinear projection is applied to unconstrained facial image recognition, where the mode labels are person identity, viewpoint, illumination, etc.

Independent components analysis (ICA) maximizes the statistical independence of the representational components of a training image ensemble, but it cannot distinguish between the different factors, or modes, inherent to image formation, including scene structure, illumination, and imaging. We introduce a nonlinear, multifactor model that generalizes ICA. Our multilinear ICA (MICA) model of image ensembles learns the statistically independent components of multiple factors. Whereas ICA employs linear (matrix) algebra, MICA exploits multilinear (tensor) algebra. We furthermore introduce a multilinear projection algorithm which projects an unlabeled test image into the N constituent mode spaces to simultaneously infer its mode labels. In the context of facial image ensembles, where the mode labels are person, viewpoint, illumination, expression, etc., we demonstrate that the statistical regularities learned by MICA capture information that, in conjunction with our multilinear projection algorithm, improves automatic face recognition.

This paper introduces a tensor framework for image-based rendering. In particular, we develop an algorithm called TensorTextures that learns a parsimonious model of the bidirectional texture function (BTF) from observational data. Given an ensemble of images of a textured surface, our nonlinear, generative model explicitly represents the multifactor interaction implicit in the detailed appearance of the surface under varying photometric angles, including local (per-texel) reflectance, complex mesostructural self-occlusion, interreflection and self-shadowing, and other BTF-relevant phenomena. Mathematically, TensorTextures is based on multilinear algebra, the algebra of higher-order tensors, hence its name. It is computed through a decomposition known as the N -mode SVD, an extension to tensors of the conventional matrix singular value decomposition (SVD). We demonstrate the application of TensorTextures to the image-based rendering of natural and synthetic textured surfaces under continuously varying viewpoint and illumination conditions.

Multilinear algebra, the algebra of higher-order tensors, offers a potent mathematical framework for analyzing ensembles of images resulting from the interaction of any number of underlying factors. We present a dimensionality reduction algorithm that enables subspace analysis within the multilinear framework. This N-mode orthogonal iteration algorithm is based on a tensor decomposition known as the N-mode SVD, the natural extension to tensors of the conventional matrix singular value decomposition (SVD). We demonstrate the power of multilinear subspace analysis in the context of facial image ensembles, where the relevant factors include different faces, expressions, viewpoints, and illuminations. In prior work we showed that our multilinear representation, called TensorFaces, yields superior facial recognition rates relative to standard, linear (PCA/eigenfaces) approaches. We demonstrate factor-specific dimensionality reduction of facial image ensembles. For example, we can suppress illumination effects (shadows, highlights) while preserving detailed facial features, yielding a low perceptual error.

The adaptive mesh model is extended in several ways. Open adaptive meshes and closed adaptive shells based on triangular and rectangular elements are developed. A discontinuity detection and preservation algorithm suitable for the model is proposed. Techniques for adaptive hierarchical subdivision of adaptive meshes and shells are also developed. The extended model is applied to image and 3D surface data.>