Automatic understanding of the product images, particularly the image attributes such as pattern, color, category, material, department, etc., has become a prerequisite for many downstream applications in the e-commerce ecosystem such as visually similar product retrieval and recommendation, publisher website image content monetization. With the recent advancement of deep learning techniques, multi-task deep neural networks (MTDNN) have been widely adopted for multi-class multi-label fashion image classification tasks. In this work, we propose a training technique Multi-Task Generative Adversarial Network (MT-GAN) to improve fashion image classification performance with an additional image translation task. We evaluated the proposed technique on two real-world fashion image datasets and experimental results show that both image classification and image translation benefit from this joint training setting. Specifically the image classifier, even though trained with a smaller amount of data and fewer types of labeled information, manages to outperform existing image classification models such as WTBI and DARN by 54% and 20% respectively for category classification on DeepFashion benchmark. Two main contributors to the superior performance of the image classifier trained as the discriminator in the proposed MT-GAN framework are: 1) the classifier leverages additional amount of data and the regularization effect by taking on the task of a traditional discriminator. 2) The classifier takes not only the RGB image but also the conditional input image shared between the generator and itself as its input, boosting the classification accuracy and recall by 18%. The proposed MT-GAN training technique is GAN-agnostic and can be applied to various state-of-the-art designs of generator and discriminator as well.

In this paper, we propose a RGB-D indoor scene recognition method that has mainly two advantages as compared to existing methods. First, by training object detectors using RGB-D images and recognizing their spatial interrelationships, we not only achieve better object localization accuracy than using RGB images alone, but also obtain details as to how the objects are related to each other in a spatial manner, thus resulting in a more effective high-level feature representation of the scene known as the Objects and Attributes (O&A) representation. Second, we learn class-specific sub-dictionaries that capture the high-order couplings between the objects and attributes. In particular, elastic net regularization and geometric similarity constraint is imposed to increase the discriminative power of the sub-dictionaries. The proposed method is evaluated on two RGB-D datasets, the NYUD dataset and the B3DO dataset. Experiments show that superior scene recognition rate can be obtained using our method.

Face recognition from video has been extensively studied in recent years. Intuitively, video provides more information than a single image. But problems such as variation in pose and occlusion still remain. When a face is partially occluded, handling the occluded part of the face is an especially challenging task. In this paper, we propose a novel method to recognize a face from video based on face patches. First, face patches are cropped from the video frame by frame. Then, face patches are matched to an overall face model and stitched together. By accumulating the patches, a reconstructed face is built which is used in recognition. We test our method in two experiments. In the first experiment, a still face database is used by randomly occluding parts of the face and using the remaining face patches in recognition. The experiments show that our method achieves a comparable recognition rate with the recognition rate from the whole face image. In the second experiment, the method is tested on video sequences. We reach a recognition rate of 81%, while there is still missing data in the reconstructed face.

Face recognition from video has recently received much interest. However, several challenges for such a system exist, such as resolution, occlusion (from objects or self-occlusion), motion blur, and illumination. The aim of this paper is to overcome the problem of self-occlusion by observing a person from multiple cameras with uniquely different views of the person's face and fusing the recognition results in a meaningful way. Each camera may only capture a part of the face, such as the right or left half of the face. We propose a methodology to use cylinder head models (CHMs) to track the face of a subject in multiple cameras. The problem of face recognition from video is then transformed to a still face recognition problem which has been well studied. The recognition results are fused based on the extracted pose of the face. For instance, the recognition result from a frontal face should be weighted higher than the recognition result from a face with a yaw of 30°. Eigenfaces is used for still face recognition along with the average-half-face to reduce the effect of transformation errors. Results of tracking are further aggregated to produce 100% accuracy using video taken from two cameras in our lab.

In this paper, we study the relationship between multi-view active appearance model (AAM) fitting and camera calibration. In the first part of the paper we propose an algorithm to calibrate the relative orientation of a set of N > 1 cameras by fitting an AAM to sets of N images. In essence, we use the human face as a (non-rigid) calibration grid. Our algorithm calibrates a set of 2 /spl times/ 3 weak-perspective camera projection matrices, protections of the world coordinate system origin into the images, depths of the world coordinate system origin, and focal lengths. We demonstrate that the performance of this algorithm is comparable to a standard algorithm using a calibration grid. In the second part of the paper, we show how calibrating the cameras improves tile performance of multi-view AAM fitting.

Non-negative Matrix Factorization (NMF) is a part-based image representation method which adds a non-negativity constraint to matrix factorization. NMF is compatible with the intuitive notion of combining parts to form a whole face. In this paper, we propose a framework of face recognition by adding NMF constraint and classifier constraints to matrix factorization to get both intuitive features and good recognition results. Based on the framework, we present two novel subspace methods: Fisher Non-negative Matrix Factorization (FNMF) and PCA Non-negative Matrix Factorization (PNMF). FNMF adds both the non-negative constraint and the Fisher constraint to matrix factorization. The Fisher constraint maximizes the between-class scatter and minimizes the within-class scatter of face samples. Subsequently, FNMF improves the capability of face recognition. PNMF adds the non-negative constraint and characteristics of PCA, such as maximizing the variance of output coordinates, orthogonal bases, etc. to matrix factorization. Therefore, we can get intuitive features and desirable PCA characteristics. Our experiments show that FNMF and PNMF achieve better face recognition performance than NMF and Local NMF.

Active Appearance Models (AAMs) are a well studied 2D deformable model. One recently proposed extension of AAMs to multiple images is the Coupled-View AAM. Coupled-View AAMs model the 2D shape and appearance of a face in two or more views simultaneously. The major limitation of Coupled-View AAMs, however, is that they are specific to a particular set of cameras, both in geometry and the photometric responses. In this paper, we describe how a single AAM can be fit to multiple images, captured simultaneously by cameras with arbitrary geometry and response functions. Our algorithm retains the major benefits of Coupled-View AAMs: the integration of information from multiple images into a single model, and improved fitting robustness. 1

The active appearance model (AAM) algorithm has proved to be a successful method for face alignment and synthesis. By elegantly combining both shape and texture models, AAM allows fast and robust deformable image matching. However, the method is sensitive to partial occlusions and illumination changes. In such cases, the PCA-based texture model causes the reconstruction error to be globally spread over the image. In this paper, we propose a new method for face alignment called active wavelet networks (AWN), which replaces the AAM texture model by a wavelet network representation. Since we consider spatially localized wavelets for modeling texture, our method shows more robustness against partial occlusions and some illumination changes. 1

We present in this paper an approach to extracting a human parametric 2D model for the purpose of estimating human posture and recognizing human activity. This task is done in two steps. In the first step, a human silhouette is extracted from a complex background under a fixed camera through a statistical method. By this method, we can reconstruct the background dynamically and obtain the moving silhouette. In the second step, a genetic algorithm is used to match the silhouette of the human body to a model in parametric shape space. In order to reduce the searching dimension, a layer method is proposed to take the advantage of the human model. Additionally we apply a structure-oriented Kalman filter to estimate the motion of body parts. Therefore the initial population and value in the GA can be well constrained. Experiments on real video sequences show that our method can extract the human model robustly and accurately.

A region based parametric motion estimation approach is proposed. As a feature based approach, sequential images are firstly represented in a multi-scale region hierarchy, then temporally adjacent region sets are matched along scale-space by symbolic optimization of compound criteria based region similarity, so that temporal region correspondence and the parametric motion representation can be obtained. The motion of regions is represented in affine transformation and estimated in a coarse-to-fine manner so that an intrinsic topological constraint can be applied to enhance the stability of the optimization. Finally the regions are grouped to uniform motion region (UMR) by their motion consistency to realize a compact representation. Compared with classical optical flow methods, our approach shows three advantages. First, motion meaning is easy to interpret at high-level. Second, fundamental problems such as aperture problem, boundary problem, or large motion problem can be some what tackled by applying structure information. Third, computation is more stable and efficient because of the symbolic feature-based methodology and the multi-scale framework.