Reconstructing photorealistic multi-view images from an image with an arbitrary view has a wide range of applications in the field of face generation. However, most current pixel-based generation models cannot generate sufficiently realistic enough images. To address this problem, we propose an edge-conditioned multi-view image generation model called Edge-GAN. Edge-GAN utilizes edge information to guide the image generation based on the perspective of the target view while the details of the input image are used to influence the target image. Edge-GAN combines the input image with the target pose information to generate a coarse image with an approximate target outline which is then refined to a better quality using adversanal training. Experiments conducted show that our Edge-GAN is able to generate high-quality images of people with convincing details.

Recent advances in Generative Adversarial Networks (GANs) have made it possible to conduct multi-domain image-to-image translation using a single generative network. While recent methods such as Ganimation and SaGAN are able to conduct translations on attribute-relevant regions using attention, they do not perform well when the number of attributes increases as the training of attention masks mostly rely on classification losses. To address this and other limitations, we introduce Attribute Manipulation Generative Adversarial Networks (AMGAN) for fashion images. While AMGAN's generator network uses class activation maps (CAMs) to empower its attention mechanism, it also exploits perceptual losses by assigning reference (target) images based on attribute similarities. AMGAN incorporates an additional discriminator network that focuses on attribute-relevant regions to detect unrealistic translations. Additionally, AMGAN can be controlled to perform attribute manipulations on specific regions such as the sleeve or torso regions. Experiments show that AMGAN outperforms state-of-the-art methods using traditional evaluation metrics as well as an alternative one that is based on image retrieval.

In this paper, we investigate ways of conducting a detailed fashion search using query images and attributes. A credible fashion search platform should be able to (1) find images that share the same attributes as the query image, (2) allow users to manipulate certain attributes, e.g. replace collar attribute from round to v-neck, and (3) handle region-specific attribute manipulations, e.g. replacing the color attribute of the sleeve region without changing the color attribute of other regions. A key challenge to be addressed is that fashion products have multiple attributes and it is important for each of these attributes to have representative features. To address these challenges, we propose the FashionSearchNet which uses a weakly supervised localization method to extract regions of attributes. By doing so, unrelated features can be ignored thus improving the similarity learning. Also, FashionSearchNet incorporates a new procedure that enables region awareness to be able to handle region-specific requests. FashionSearchNet outperforms the most recent fashion search techniques and is shown to be able to carry out different search scenarios using the dynamic queries.

In this paper, we propose an attribute-based query & retrieval system designed for fashion products. Our system addresses the problem of carrying out fashion searches by the query image and attribute manipulation, e.g. replacing long sleeve attribute of a dress to sleeveless. We present the attributes in two groups: (1) general attributes (category, gender etc.) and (2) special attributes (sleeve length, collar etc.). The special attributes are more suitable for the attribute manipulation and thus conducting searches. In order to solve the mentioned fashion search problem, it is crucial for the deep neural networks to understand attribute similarities. To facilitate more specific similarity learning, clothing items are represented by their structural subcomponents or "parts". The parts are estimated using an unsupervised segmentation method and used inside the proposed Convolutional Neural Network (CNN) as an attention mechanism. Meaning, different parts are connected to the special attributes, e.g. sleeve part is connected with sleeve length attribute. With this mechanism, part-based triplet ranking constraint is applied to learn similarity of each special attribute independently from one another in a single network. In the end, the well-defined features are used to conduct the fashion search. Additionally, an adaptive relevance feedback module is used to personalize the fashion search process with the feature descriptions. For our experiments, a new dataset is constructed containing 101,021 images which consist of pure clothing items. Besides achieving decent retrieval results in our dataset, the experiments show that proposed technique outperforms different baselines and is able to adapt towards user's requests.

Despite remarkable progress of face analysis techniques, detecting landmarks on large-pose faces is still difficult due to self-occlusion, subtle landmark difference and incomplete information. To address these challenging issues, we introduce a novel recurrent 3D-2D dual learning model that alternatively performs 2D-based 3D face model refinement and 3D-to-2D projection based 2D landmark refinement to reliably reason about self-occluded landmarks, precisely capture the subtle landmark displacement and accurately detect landmarks even in presence of extremely large poses. The proposed model presents the first loop-closed learning framework that effectively exploits the informative feedback from the 3D-2D learning and its dual 2D-3D refinement tasks in a recurrent manner. Benefiting from these two mutual-boosting steps, our proposed model demonstrates appealing robustness to large poses (up to profile pose) and outstanding ability to capture fine-scale landmark displacement compared with existing 3D models. It achieves new state-of-the-art on the challenging AFLW benchmark. Moreover, our proposed model introduces a new architectural design that economically utilizes intermediate features and achieves 4× faster speed than its deep learning based counterparts.

We investigate the feature design and classification architectures in temporal action localization. This application focuses on detecting and labeling actions in untrimmed videos, which brings more challenge than classifying presegmented videos. The major difficulty for action localization is the uncertainty of action occurrence and utilization of information from different scales. Two innovations are proposed to address this issue. First, we propose a Pyramid of Score Distribution Feature (PSDF) to capture the motion information at multiple resolutions centered at each detection window. This novel feature mitigates the influence of unknown action position and duration, and shows significant performance gain over previous detection approaches. Second, inter-frame consistency is further explored by incorporating PSDF into the state-of-the-art Recurrent Neural Networks, which gives additional performance gain in detecting actions in temporally untrimmed videos. We tested our action localization framework on the THUMOS'15 and MPII Cooking Activities Dataset, both of which show a large performance improvement over previous attempts.

We propose a novel computationally efficient hierarchical dictionary learning (HDL) approach for data-driven unmixing and functional connectivity analysis of functional magnetic resonance imaging (fMRI) data. It is shown that by simultaneously exploiting the sparsity of the spatial brain maps and the incoherence among their evolution in time or task functions, one can achieve better performance while overcoming the drawbacks of existing approaches. The task functions constituting the dictionary, are learned using a hierarchical subset selection approach. Here, to enforce incoherence among atoms, any new atom is selected from suitable training candidates if it does not lie in the column span of past selected atoms. Also, since the sparsity of spatial maps is generally unknown and affected due to acquisition artifacts, HDL doesn't make use of an implicit sparse coding stage while dictionary update. This makes HDL a very fast and efficient data-driven approach for fMRI analysis. Experimental results on synthetic and real fMRI datasets provide compelling evidences that HDL performs better than existing state-of-the-art methods.

In this paper, we introduce a multi-dimensional approach to the problem of reconstruction of MR image sequences that are highly undersampled in k-space. By formulating the reconstruction as a high-order low-rank plus sparse tensor decomposition problem, we propose an efficient numerical algorithm based on the alternating direction method of multipliers (ADMM) to solve the optimization. Through extensive experimental results we show that our proposed method achieves superior reconstruction quality, compared to the state-of-the-art reconstruction methods.

In this technical demonstration, we propose a face swapping framework, which is able to interactively change the appearance of a face in the wild to a different person/creature's face in real time on a mobile device. To realize this objective, we develop a deep learning-based face detector which is able to accurately detect faces in the wild. Our face feature points tracking system based on progressive initialization ensures accurate and robust localization of facial landmarks under extreme poses and expressions in real time. Relying on the advances of our face detector and face feature points tracker, we construct the Face Swapper which can smoothly replace the face appearance of a user in real time.

Identifying the presence of Anti-Nuclear Antibody (ANA) in Human Epithelial Type-2 (HEp-2) cells via Indirect Immunofluoresence (IIF) images is commonly used to detect various diseases in clinical pathology tests. The main task at hand is the classification of cells into different categories by observing their staining patterns. However, when performed manually, this method is time and labour intensive. Also, as IIF analysis is still subjective, medical doctors have not been able to get satisfactory accuracy rates on the classification task. Pattern recognition techniques have been introduced, but performance of current systems in literature are not satisfactory as they use a common set of descriptors to describe images from all classes and perform classification in one go. Here, we propose a system based on hierarchical classification that takes into account discriminative features for classification purpose. A new texture descriptor based on change of curvature of the parametric from of image intensity is proposed. We have also introduced a model adaptation technique using non-linear transformation functions to make the classifiers more robust.

It has been estimated that autoimmune diseases are among the top ten leading causes of death among women in all age groups up to 65 years. However, the detection of it by indirect immunofluorescence (IIF) image analysis depends heavily on the experience of the physicians. An accurate and automatic Computer Aided Diagnosis (CAD) system will help greatly for the classification of the Human Epithelial type 2 (HEp-2) cell images with little human intervention. In this paper we present an automatic HEp-2 cell image classification technique that exploits different spatial scaled image representation and sparse coding of SIFT features. Additionally, spatial max pooling of sparse coding at different scales is used to boost the classification performance. The proposed method is tested on the ICPR 2012 contest dataset and experiments show that it clearly outperforms state-of-the-art techniques in cell and image level as well as two intensity level images.

We propose a sparse sampling model as a feature selection tool for 3D face recognition, and compare its performance with the traditional dense subspace methods. The sparse LDA algorithm is applied to find the most discriminative features on range and texture images, meanwhile achieving the region selection purpose. The selected regions from both shape and texture are demonstrated. The classification remains accurate even at a high level of sparsity. To generalize the model, a probability density function is then estimated according to the selected region, and new samples are drawn accordingly to form new sparse features for classification. We also use the local coordinate system to make the sampling process more efficient, and insensitive to geometric transforms.

It has been shown recently that incorporating priori knowledge into the basic compressive sensing results in significant improvement of its performance. This has motivated us to extend the incorporation of partial known support into the problem of Robust Principal Component Analysis (RPCA) from compressive measurements. Our proposed algorithm utilizes the known part of the support to recover a matrix as the sum of a low-rank matrix and a sparse component and is tested on the problem of surveillance video reconstruction from compressive measurements. Our experimental results show that the incorporation of partial known support, can significantly improve the reconstruction performance of video sequences.

In this paper, we propose a novel method for spatial context modeling toward boosting visual discriminating power. We are particularly interested in how to model high-order local spatial contexts instead of the intensively studied second-order spatial contexts, i.e., co-occurrence relations. Motivated by the recent success of random forest in learning discriminative visual codebook, we present a spatialized random forest (SRF) approach, which can encode an unlimited length of high-order local spatial contexts. By spatially random neighbor selection and random histogram-bin partition during the tree construction, the SRF can explore much more complicated and informative local spatial patterns in a randomized manner. Owing to the discriminative capability test for the random partition in each tree node's split process, a set of informative high-order local spatial patterns are derived, and new images are then encoded by counting the occurrences of such discriminative local spatial patterns. Extensive comparison experiments on face recognition and object/scene classification clearly demonstrate the superiority of the proposed spatial context modeling method over other state-of-the-art approaches for this purpose.

Attention is an integral part of the human visual system and has been widely studied in the visual attention literature. The human eyes fixate at important locations in the scene, and every fixation point lies inside a particular region of arbitrary shape and size, which can either be an entire object or a part of it. Using that fixation point as an identification marker on the object, we propose a method to segment the object of interest by finding the "optimal" closed contour around the fixation point in the polar space, avoiding the perennial problem of scale in the Cartesian space. The proposed segmentation process is carried out in two separate steps: First, all visual cues are combined to generate the probabilistic boundary edge map of the scene; second, in this edge map, the "optimal" closed contour around a given fixation point is found. Having two separate steps also makes it possible to establish a simple feedback between the mid-level cue (regions) and the low-level visual cues (edges). In fact, we propose a segmentation refinement process based on such a feedback process. Finally, our experiments show the promise of the proposed method as an automatic segmentation framework for a general purpose visual system.

In this paper, we present a novel framework, called learning by propagability, for two essential data mining tasks, i.e., classification and regression. The whole learning process is driven by the philosophy that the data labels and the optimal feature representation jointly constitute a harmonic system, where the data labels are invariant with respect to the propagation on the similarity graph constructed based on the optimal feature representation. Based on this philosophy, a unified framework of learning by propagability is proposed for the purposes of both classification and regression. Specifically, this framework has three characteristics: 1) the formulation unifies the label propagation and optimal feature representation pursuing, and thus the label propagation process is enhanced by benefiting from the refined similarity graph constructed with the derived optimal feature representation instead of the original representation; 2) it unifies the formulations for supervised and semisupervised learning in both classification and regression tasks; and 3) it can directly deal with the multiclass classification problems. Extensive experiments for the classification task on UCI toy data sets, handwritten digit recognition, face recognition, and microarray recognition as well as for the regression task of human age estimation on the FG-NET aging database, all validate the effectiveness of our proposed learning framework, compared with the state-of-the-art counterparts.

Concerning medical images, which are known to have sparsity in either the spatial (or its derivative), DFT, DCT or curvelet domain, we propose a new approach for reconstruction from sparse samples, based on Simultaneous Perturbation Stochastic Optimization (SPSA) to minimize a nonconvex ℓp-norm for 00-norm as is computationally feasible. This approach is distinct from the homotopy-theoretic and hard-thresholding techniques of recent literature for ℓ0- and ℓp-norm minimization. For lack of space, our illustrations are limited to only one each of synthetic and real images.

We propose a novel strategy, based on resampling of 3D meshes, to recognize facial expressions. This entails conversion of the existing irregular 3D mesh structure in the database to a uniformly sampled 3D matrix structure. An important consequence of this operation is that the classical correspondence problem can be dispensed with. In the present paper, in order to demonstrate the feasibility of the proposed strategy, we employ only spectral flow matrices as features to recognize facial expressions. Experimental results are presented, along with suggestions for possible refinements to the strategy to improve classification accuracy.

The aim of this paper is to address the problem of recognizing human group activities in surveillance videos. This task has great potentials in practice, however was rarely studied due to the lack of benchmark database and the difficulties caused by large intra-class variations. Our contributions are two-fold. Firstly, we propose to encode the group-activities with three types of localized causalities, namely self-causality, pair-causality, and group-causality, which characterize the local interaction/reasoning relations within, between, and among motion trajectories of different humans respectively. Each type of causality is expressed as a specific digital filter, whose frequency responses then constitute the feature representation space. Finally, each video clip of certain group activity is encoded as a bag of localized causalities/filters. We also collect a human group-activity video database, which involves six popular group activity categories with about 80 video clips for each in average, captured in five different sessions with varying numbers of participants. Extensive experiments on this database based on our proposed features and different classifiers show the promising results on this challenging task.

In this paper, we investigate how to incorporate spatial and/or temporal contextual information into classical histogram features with the aim of boosting visual classification performance. Firstly, we show that the stationary distribution derived from the normalized histogram-bin co-occurrence matrix characterizes the row sums of the original histogram-bin co-occurrence matrix. This underlying rationale of the histogram-bin co-occurrence features then motivates us to propose the concept of general contextualizing histogram process, which encodes the spatial and/or temporal contexts as local homogeneity distributions and produces the so called contextualized histograms by convoluting these local homogeneity distributions with the histogram-bin index images/videos. Finally, the third and even higher order contextualized histograms are instantiated for encoding more complicated and informative spatial and/or temporal contextual information into histograms. We evaluate these proposed methods on face recognition and group activity classification problems, and the results demonstrate that the contextualized histograms significantly boost the visual classification performance.

We investigate how to effectively incorporate spatial structure information into histogram features for boosting visual classification performance motivated by recently proposed Markov stationary features (MSF). First, we show that due to the symmetric property of the image occurrence modeling procedure, the stationary distribution derived from the normalized co-occurrence matrix has a trivial informative solution which only approximates the original histogram representation, i.e., does not encode proper spatial structure information. To eliminate this ambiguity, we propose in this work the so called directed Markov stationary features (DMSF) to encode spatial information into histogram features, and the asymmetric essence of the co-occurrence matrices in DMSF avoids the trivial informative solutions in MSF. Extensive experiments on face recognition show the significant performance improvement brought by our proposed DMSF.

A novel segmentation-based image approximation and coding technique is proposed. A hybrid quad-binary (QB) tree structure is utilized to efficiently model and code geometrical information within images. Compared to other tree-based representation such as wedgelets, the proposed QB-tree based method is more efficient for a wide range of contour features such as junctions, corners and ridges, especially at low bit rates.

This paper presents a new method for segmenting medical images by modeling interaction between neighboring structures. Compared to previously reported methods, the proposed approach enables simultaneous segmentation of multiple neighboring structures for improved robustness. During the segmentation process, the object contour evolution and shape prior estimates are influenced by the interactions between neighboring shapes consisting of attraction, repulsion, and competition. Instead of estimating the a priori shape of each structure independently, an interactive maximum a posteriori shape estimation method is used for estimating the shape priors by considering shape prior distribution, neighboring shapes, and image features. Energy functionals are then formulated to model the interaction and segmentation. With the proposed method, neighboring structures with similar intensities and/or textures, and blurred boundaries can be extracted simultaneously. Experimental results obtained on both synthetic data and medical images demonstrate that the introduced interaction between neighboring structures improves segmentation performance compared with other existing approaches.

In this paper, we present a novel feature extraction framework, called learning by propagability. The whole learning process is driven by the philosophy that the data labels and optimal feature representation can constitute a harmonic system, namely, the data labels are invariant with respect to the propagation on the similarity-graph constructed by the optimal feature representation. Based on this philosophy, a unified formulation for learning by propagability is proposed for both supervised and semi-supervised configurations. Specifically, this formulation offers the semi-supervised learning two characteristics: 1) unlike conventional semi-supervised learning algorithms which mostly include at least two parameters, this formulation is parameter-free; and 2) the formulation unifies the label propagation and optimal representation pursuing, and thus the label propagation is enhanced by benefiting from the graph constructed with the derived optimal representation instead of the original representation. Extensive experiments on UCI toy data, handwritten digit recognition, and face recognition all validate the effectiveness of our proposed learning framework compared with the state-of-the-art methods for feature extraction and semi-supervised learning.

Edges provide critical information which enables viewers to better discern objects in images. Although transform-based image compression schemes have been successful, they are unable to efficiently represent 2D edges. Wedgelets capture geometrical structures in images by explicitly defining an edge. In this paper, we introduce high order wedgelets in a more generalized form of quad-tree partitioning to realize improved compression performance compared to existing compression methods.

In this paper, we present a hybrid framework for articulated 3-D human motion tracking from multiple synchronized cameras with potential uses in surveillance systems. Although the recovery of 3-D motion provides richer information for event understanding, existing methods based on either deterministic search or stochastic sampling lack robustness or efficiency. We therefore propose a hybrid sample-and-refine framework that combines both stochastic sampling and deterministic optimization to achieve a good compromise between efficiency and robustness. Similar motion patterns are used to learn a compact low-dimensional representation of the motion statistics. Sampling in a low-dimensional space is implemented during tracking, which reduces the number of particles drastically. We also incorporate a local optimization method based on simulated physical force/moment into our framework, which further improves the optimality of the tracking. Experimental results on several real human motion sequences show the accuracy and robustness of our method, which also has a higher sampling efficiency than most particle filtering-based methods.

We apply the probability hypothesis density (PHD) filter to track a random number of pedestrians in image sequences. The PHD filter is implemented using particle filter. How to design importance functions of the particle PHD filter remains a challenge, especially when targets can appear, disappear, merge, or split at any time. To meet this challenge, we have modeled the targets into two categories: survival objects and spontaneous birth objects. Based on the model, we have derived the data-driven importance function for a particle PHD filter and applied to pedestrians tracking where people or groups appear, merge, split, and disappear in the field of view of a camera. The experimental results have demonstrated the effectiveness of the particle PHD filter using the proposed importance function in tracking random number of pedestrians and deriving their locations.

This paper presents our work which involves the application of a recursive Bayesian filter, the Gaussian mixture probability hypothesis density (GMPHD) filter, to a visual tracking problem. Foreground objects are detected using statistical background modeling to obtain measurements which are input into the filter. The GMPHD filter explicitly models the birth, survival and death of objects by managing the number of Gaussian components and jointly estimates the time-varying number of objects and their states. A scene-driven method is proposed to initialize the GMPHD filter and model the birth of new objects. The results shows when a person or a group appeared, merged, split, and disappeared in the field of view, the GMPHD filter can track the number and positions at the most time. The scene-driven GMPHD filter can track the birth of new objects faster than the particle PHD filter.

This paper presents a novel scheme for automatic initialization for all types of deformable models. Our method is able to automatically generate a close-to-boundary initialization which is independent of the subsequent segmentation process. Therefore, our method enables different types of deformable models achieve more accurate and robust results. Topographic independent component analysis (TICA) based feature extraction technique is presented for learning a representation from a set of un-labeled image patches. During learning, a topographic map of basis components emerge. An intelligent contour generation procedure is also proposed. Experimental results on abdominal CT images demonstrate the potential of our approach.

In signal approximation, classical wavelet synthesis are known to produce Gibbs-like phenomenon around discontinuities when wavelet coefficients in the cone of influence of the discontinuities are quantized. By analyzing a function in a piecewise manner, filtering across discontinuities can be avoided. Using this principle, the interval wavelet transform can generate sparser representations in the vicinity of discontinuities than classical wavelet transforms. This work introduces two new constructions of interval wavelets and shows how they can be used for image compression and upscaling.

We present a method based on the morphological 3D h-maxima transform to segment the kidneys in image volumes obtained by magnetic resonance imaging after injection of a contrast agent. Manual cropping techniques had been heavily relied upon to sever connections to other organs (arteries, liver, spleen, intestines, etc.) when routine methods produced oversegmented results. We show that simply generated rectangular masks and edge information may be incorporated to guide the segmentation. This approach is particularly useful when severe pathology or congenital abnormalities are present as no prior information about location or appearance is assumed. Results and validation from experiments on a number of selected MR datasets are demonstrated

Text superimposed on the video frames provides supplemental but important information for video indexing and retrieval. The detection and recognition of text from video is thus an important issue in automated content-based indexing of visual information in video archives. Text of interest is not limited to static text. They could be scrolling in a linear motion where only part of the text information is available during different frames of the video. The problem is further complicated if the video is corrupted with noise. An algorithm is proposed to detect, classify and segment both static and simple linear moving text in complex noisy background. The extracted texts are further processed using averaging to attain a quality suitable for text recognition by commercial optical character recognition (OCR) software

We propose a novel approach to the detection and classification of human facial expressions using a morphable 3D model. We acquire the various expressions of an individual using a face scanner that produces textured 3D meshes using stereoscopic reconstruction. A morphable expression model (MEM), that incorporates emotion-dependent face variations in terms of morphing parameters, is then computed by establishing correspondence among the emotive faces. These morphing parameters are used for emotion recognition and classification. We demonstrate that the different facial expressions correspond to distinct clusters in the expression space.

Just-noticeable distortion (JND), which refers to the maximum distortion that the human visual system (HVS) cannot perceive, plays an important role in perceptual image and video processing. In comparison with JND estimation for images, estimation of the JND profile for video needs to take into account the temporal HVS properties in addition to the spatial properties. In this paper, we develop a spatio-temporal model estimating JND in the discrete cosine tranform domain. The proposed model incorporates the spatio-temporal contrast sensitivity function, the influence of eye movements, luminance adaptation, and contrast masking to be more consistent with human perception. It is capable of yielding JNDs for both still images and video with significant motion. The experiments conducted in this study have demonstrated that the JND values estimated for video sequences with moving objects by the model are in line with the HVS perception. The accurate JND estimation of the video towards the actual visibility bounds can be translated into resource savings (e.g., for bandwidth/storage or computation) and performance improvement in video coding and other visual processing tasks (such as perceptual quality evaluation, visual signal restoration/enhancement, watermarking, authentication, and error protection).

This paper presents a new method for segmentation of medical images by extracting organ contours, using minimal path deformable models incorporated with statistical shape priors. In our approach, boundaries of structures are considered as minimal paths, i.e., paths associated with the minimal energy, on weighted graphs. Starting from the theory of minimal path deformable models, an intelligent "worm" algorithm is proposed for segmentation, which is used to evaluate the paths and finally find the minimal path. Prior shape knowledge is incorporated into the segmentation process to achieve more robust segmentation. The shape priors are implicitly represented and the estimated shapes of the structures can be conveniently obtained. The worm evolves under the joint influence of the image features, its internal energy, and the shape priors. The contour of the structure is then extracted as the worm trail. The proposed segmentation framework overcomes the short-comings of existing deformable models and has been successfully applied to segmenting various medical images.

A novel content adaptive rate-distortion optimization scheme has been proposed. The scheme can effectively distinguish texture regions, edge regions and flat regions using a directional field technique. Since the human visual system (HVS) perceives distortions more easily near edges and in flat regions, distortion reduction is more important in those regions than the bits it consumes to code the motion information. Adaptive rate-distortion optimization is carried out by adjusting the Lagrangian multiplier so that small values are assigned to edge and flat regions and large values to the random texture region. The proposed scheme has been tested in the scalable video coding (SVC) reference codec by Microsoft Research Asia (MSRA) (Xu, J. et al., ISO/EEC JTC/WG11 M10569, S05, 2004). Experimental results show that the accuracy of motion alignment in visually important regions is greatly improved in the temporal transform step of 3D wavelet coding and the scheme effectively preserves details in the most perceptually prominent regions for all bitstream layers, with no loss in PSNR.

This paper has proposed a content adaptive rate-distortion optimization scheme for scalable video coding. To adaptively determine the tradeoff between texture and motion coding, the scheme first makes use of spatial and temporal complexities to adjust the Lagrangian multiplier for all the macroblocks in a frame. After that, the Lagrangian multiplier for every macroblock of the frame is further refined using local perceptual cues. To ensure optimal local selection of the Lagrangian multiplier, several characteristics of human visual system have been taken into account to guide the adjustment. Experiments have shown that the proposed scheme could effectively adapt the adjustment of Lagrangian multiplier to video contents and the subjective quality of the reconstructed frames could be improved without compromising PSNR.

The processing time needed for motion estimation usually accounts for a significant part of the overall processing time of the video encoder. To improve the video encoding speed, reducing the execution time for motion estimation process is essential. Parallel implementation of video encoding systems using either the software or the hardware approach has attracted much attention in the area of real time video coding. In this paper, we attempt to implement a video encoder on a bus network. Usually, for such a parallel system, the key concern is associated with partitioning and balancing of the computational load among the processors such that the overall processing time of the video encoder is minimized. With the use of the divisible load theory (DLT) paradigm, a strip-wise load partitioning/balancing scheme, a load distribution strategy, two implementation strategies are developed to exploit the data parallelism inherent in the video encoding process. The striking feature of our design is that,both the granularity of the load partitions and all the associated overheads caused during parallel video encoding process can be explicitly considered. This significantly contributes to the minimization of the overall processing time of the video encoder. Extensive experimental studies are carried out to test the effectiveness of the proposed strategies. The performance of the parallel video encoder is quantified using the metrics speedup and performance gain, respectively. The experimental results show that our strategies are effective for exploiting the available parallelism inherent in the video encoding process and provide a theoretical insight on how to analytically quantify and minimize the overall processing time of a parallel system. The proposed strategies can be easily extended and applied to improve other existing parallel systems.

This paper proposes a method for progressive lossy-to-lossless compression of four-dimensional (4-D) medical images (sequences of volumetric images over time) by using a combination of three-dimensional (3-D) integer wavelet transform (IWT) and 3-D motion compensation. A 3-D extension of the set-partitioning in hierarchical trees (SPIHT) algorithm is employed for coding the wavelet coefficients. To effectively exploit the redundancy between consecutive 3-D images, the concepts of key and residual frames from video coding is used. A fast 3-D cube matching algorithm is employed to do motion estimation. The key and the residual volumes are then coded using 3-D IWT and the modified 3-D SPIHT. The experimental results presented in this paper show that our proposed compression scheme achieves better lossy and lossless compression performance on 4-D medical images when compared with JPEG-2000 and volumetric compression based on 3-D SPIHT.

This paper comes up with a linear self-calibration approach for camera focal length estimation. Compared with other linear techniques, the advantage of this method is that no priori information of motion is needed. From the degenerated equations (one quadratic and two linear), the essential constraints are generated on calibration of camera intrinsic parameters. Thus the focal length of camera is conveniently obtained in closed form on the reliable assumption that only the focal length is unknown but constant, and the skew factor is zero. Synthetic object/image, real images of indoor and outdoor generic scenes and 3D model reconstruction are involved in our experiments to analyze the accuracy of the estimated focal length. The experimental results indicate that our proposed method is both robust and efficient.

In this paper, a linear self-calibration approach for camera focal length estimation is proposed. No priori information of camera motion is needed for the method. The unknown focal length of camera is conveniently obtained in closed form from degenerated Kruppa's equations. These degenerated equations (one quadratic and two linear) provide the essential constraints on calibration of camera intrinsic parameters. Real images of indoor and outdoor generic scenes and 3D metric reconstruction are used to analyze the accuracy of the estimated focal length. Experiments results show that the method is both robust and efficient.

Generation and transmission of complex animation sequences can benefit substantially from the availability of tools for handling large amounts of data associated with dynamic three-dimensional (3-D) models. Previous works in 3-D dynamic compression consider only the simplest situation where the connectivity changes do not occur with time. In this paper, we present an approach for compressing 3-D dynamic models in which both the vertex data and the connectivity data can change with time. Using our framework, 3-D animation sequences generated using commercial graphics tools or dynamic range data captured using range scanners can be compressed significantly. We use 3-D registration to identify the changes in the vertex data and the connectivity of the 3-D geometry between successive frames. Next, the interframe motion is encoded using affine motion parameters and the differential pulse coded modulation (DPCM) predictor. Our work is the first to exploit the temporal coherence in the connectivity data between frames and presents a detailed encoding scheme for 3-D dynamic data. We also discuss the issue of inserting I-frames in the compressed data for better performance. We show that our algorithm has a far superior performance when compared with existing techniques for both vertex compression and connectivity compression of 3-D dynamic datasets.

This paper describes a variation of the set partitioning in hierarchical trees (SPIHT) scheme for color image coding. By using partially linked spatial orientation tree structures across different spectral planes, the new color-SPIHT scheme is able to embed both chrominance and luminance data in the coded bit stream. The performance is comparable to that of a SPIHT-based coding scheme but with significantly lower computational complexity.

As the tool wears during a machining operation, the texture of the machined surface varies dramatically. There is a strong relationship between the degree of wear of the cutting tool and the geometry imparted by the tool on to the workpiece surface. It is more practical and suitable to analyze the machined surface than to measure the wear of the cutting tool directly. This paper discusses our work, which involves fractal analysis of texture of workpiece surfaces that have been subjected to end milling operations. Two characteristics of the texture, high directionality and self-affinity, are dealt with by computing the fractal features of texture images. The hidden Markov model is used to differentiate the various states of tool wear. Our result shows that fractal features can be effectively used to monitor the tool wear.

In this paper, we present a object modeling technique based on volume interaction using silhouettes of objects placed far away from the camera. The method is flexible and easy to use. By using a generic calibration matrix, we surpass the stringent requirements of accurate camera calibration inherent in volume intersection algorithms. This widens the scope of utility of this technique even to the laymen without significant back ground in computer vision. In addition, we propose a modified intersection test to determine inclusion/ exclusion of projected cubes which is fast, accurate and does not require any additional sophisticated hardware. We successfully conducted experimental test on synthetic objects and real images taken through video input. The technique is highly suitable for modeling objects for virtual environments, Internet shopping, e-commerce, etc. and requires easily available, Ubiquitous hardware.

In this paper, we present an embedded approach for coding image regions with arbitrary shapes. Our scheme takes a different approach by separating the objects in the transform domain instead of the image domain so that only one transform for the entire image is required. We define a new shape-adaptive embedded zerotree wavelet coding (SA-EZW) technique for encoding the coefficients corresponding to specific objects in gray-scale and color-image segments by implicitly representing their shapes, thereby forgoing the need for separately coding the region boundary. At our decoder, the shape information can be recovered without separate and explicit shape coding. The implicit shape coding enables the bit stream for the object to be fully rate scalable, since no explicit bit allocation is needed for the object shape. This makes it particularly suitable when content-based functionalities are desired in situations where the user bit rate is constrained and enables precise bit-rate control while avoiding the problem of contour coding. We show that our algorithm sufficiently addresses the issue of content-based scalability and improved coding efficiency when compared with the "chroma keying" technique, an implicit shape-coding technique which is adopted by the current MPEG-4 standard.

We introduce a highly scalable video compression system for very low bit-rate videoconferencing and telephony applications around 10-30 kbits/s. The video codec first performs a motion-compensated three-dimensional (3-D) wavelet (packet) decomposition of a group of video frames, and then encodes the important wavelet coefficients using a new data structure called tri-zerotrees (TRI-ZTR). Together, the proposed video coding framework forms an extension of the original zero tree idea of Shapiro (1992) for still image compression. In addition, we also incorporate a high degree of video scalability into the codec by combining the layered/progressive coding strategy with the concept of embedded resolution block coding. With scalable algorithms, only one original compressed video bit stream is generated. Different subsets of the bit stream can then be selected at the decoder to support a multitude of display specifications such as bit rate, quality level, spatial resolution, frame rate, decoding hardware complexity, and end-to-end coding delay. The proposed video codec also allows precise bit rate control at both the encoder and decoder, and this can be achieved independently of the other video scaling parameters. Such a scheme is very useful for both constant and variable bit rate transmission over mobile communication channels, as well as video distribution over heterogeneous multicast networks. Finally, our simulations demonstrated comparable objective and subjective performance when compared to the ITU-T H.263 video coding standard, while providing both multirate and multiresolution video scalability.

The widespread use of block-based interframe motion estimation for video sequence compression in both MPEG and H.263 standards is due to its effectiveness and simplicity of implementation. Nevertheless, the high computational complexity of the full-search algorithm has motivated a host of suboptimal but faster search strategies. A popular example is the three-step search (TSS) algorithm. However, its uniformly spaced search pattern is not well matched to most real-world video sequences in which the motion vector distribution is nonuniformly biased toward the zero vector. Such an observation inspired the new three-step search (NTSS) which has a center-biased search pattern and supports a halfway-stop technique. It is faster on average, and gives better motion estimation as compared to the well-known TSS. Later, the four-step search (4SS) algorithm was introduced to reduce the average case from 21 to 19 search points, while maintaining a performance similar to NTSS in terms of motion compensation errors. We propose a novel unrestricted center-biased diamond search (UCBDS) algorithm which is more efficient, effective, and robust than the previous techniques. It has a best case scenario of only 13 search points and an average of 15.5 block matches. This makes UCBDS consistently faster than the other suboptimal block-matching techniques. This paper also compares the above methods in which both the processing speed and the accuracy of motion compensation are tested over a wide range of test video sequences.